AWS

AWS ECS & EKS Power Up with Remote MCP Servers

Leave a comment

The Model Context Protocol (MCP) has rapidly become the standard for connecting AI models to your data and tools. However, most initial implementations are strictly local—relying on stdio to pipe data between a local process and your AI client (like Claude Desktop or Cursor). While this works for personal scripts, it doesn’t scale for teams.

To truly unlock the potential of AI agents in the enterprise, you need to decouple the “Brain” (the AI client) from the “Hands” (the tools). This means moving your MCP servers from localhost to robust cloud infrastructure.

This guide details the architectural shift required to run AWS ECS EKS MCP workloads. We will cover how to deploy remote MCP servers using Server-Sent Events (SSE), how to host them on Fargate and Kubernetes, and—most importantly—how to secure them so you aren’t exposing your internal database tools to the open internet.

The Architecture Shift: From Stdio to Remote SSE

In a local setup, the MCP client spawns the server process and communicates via standard input/output. This is secure by default because it’s isolated to your machine. To move this to AWS, we must switch the transport layer.

The MCP specification supports SSE (Server-Sent Events) for remote connections. This changes the communication flow:

  • Server-to-Client: Uses a persistent SSE connection to push events (like tool outputs or log messages).
  • Client-to-Server: Uses standard HTTP POST requests to send commands (like “call tool X”).

Pro-Tip: Unlike WebSockets, SSE is unidirectional (Server -> Client). This is why the protocol also requires an HTTP POST endpoint for the client to talk back. When deploying to AWS, your Load Balancer must support long-lived HTTP connections for the SSE channel.

Option A: Serverless Simplicity with AWS ECS (Fargate)

For most standalone MCP servers—such as a tool that queries a specific RDS database or interacts with an internal API—AWS ECS Fargate is the ideal host. It removes the overhead of managing EC2 instances while providing native integration with AWS VPCs for security.

1. The Container Image

You need an MCP server that listens on a port (usually via a web framework like FastAPI or Starlette) rather than just running a script. Here is a conceptual Dockerfile for a Python-based remote MCP server:

FROM python:3.11-slim

WORKDIR /app

# Install MCP SDK and a web server (e.g., Starlette/Uvicorn)
RUN pip install mcp[cli] uvicorn starlette

COPY . .

# Expose the port for SSE and HTTP POST
EXPOSE 8080

# Run the server using the SSE transport adapter
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8080"]

2. The Task Definition & ALB

When defining your ECS Service, you must place an Application Load Balancer (ALB) in front of your tasks. The critical configuration here is the Idle Timeout.

  • Health Checks: Ensure your container exposes a simple /health endpoint, or the ALB will kill the task during long AI-generation cycles.
  • Timeout: Increase the ALB idle timeout to at least 300 seconds. AI models can take time to “think” or process large tool outputs, and you don’t want the SSE connection to drop prematurely.

Option B: Scalable Orchestration with Amazon EKS

If your organization already operates on Kubernetes, deploying AWS ECS EKS MCP servers as standard deployments allows for advanced traffic management. This is particularly useful if you are running a “Mesh” of MCP servers.

The Ingress Challenge

The biggest hurdle on EKS is the Ingress Controller. If you use NGINX Ingress, it defaults to buffering responses, which breaks SSE (the client waits for the buffer to fill before receiving the first event).

You must apply specific annotations to your Ingress resource to disable buffering for the SSE path:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mcp-server-ingress
  annotations:
    # Critical for SSE to work properly
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  rules:
    - host: mcp.internal.yourcompany.com
      http:
        paths:
          - path: /sse
            pathType: Prefix
            backend:
              service:
                name: mcp-service
                port:
                  number: 80

Warning: Never expose an MCP server Service as LoadBalancer (public) without strict Security Groups or authentication. An exposed MCP server gives an AI direct execution access to whatever tools you’ve enabled (e.g., “Drop Database”).

Security: The “MCP Proxy” & Auth Patterns

This is the section that separates a “toy” project from a production deployment. How do you let an AI client (running on a developer’s laptop) access a private ECS/EKS service securely?

1. The VPN / Tailscale Approach

The simplest method is network isolation. Keep the MCP server in a private subnet. Developers must be on the corporate VPN or use a mesh overlay like Tailscale to reach the `http://internal-mcp:8080/sse` endpoint. This requires zero code changes to the MCP server.

2. The AWS SigV4 / Auth Proxy Approach

For a more cloud-native approach, AWS recently introduced the concept of an MCP Proxy. This involves:

  1. Placing your MCP Server behind an ALB with AWS IAM Authentication or Cognito.
  2. Running a small local proxy on the client machine (the developer’s laptop).
  3. The developer configures their AI client to talk to localhost:proxy-port.
  4. The local proxy signs requests with the developer’s AWS credentials (SigV4) and forwards them to the remote ECS/EKS endpoint.

This ensures that only users with the correct IAM Policy (e.g., AllowInvokeMcpServer) can access your tools.

Frequently Asked Questions (FAQ)

Can I use the official Amazon EKS MCP Server remotely?

Yes, but it’s important to distinguish between hosting a server and using a tool. AWS provides an open-source Amazon EKS MCP Server. This is a tool you run (locally or remotely) that gives your AI the ability to run kubectl commands and inspect your cluster. You can host this inside your cluster to give an AI agent “SRE superpowers” over that specific environment.

Why does my remote MCP connection drop after 60 seconds?

This is almost always a Load Balancer or Reverse Proxy timeout. SSE requires a persistent connection. Check your AWS ALB “Idle Timeout” settings or your Nginx proxy_read_timeout. Ensure they are set to a value higher than your longest expected idle time (e.g., 5-10 minutes).

Should I use ECS or Lambda for MCP?

While Lambda is cheaper for sporadic use, MCP is a stateful protocol (via SSE). Running SSE on Lambda requires using Function URLs with response streaming, which has a 15-minute hard limit and can be tricky to debug. ECS Fargate is generally preferred for the stability of the long-lived connection required by the protocol.

Conclusion

Moving your Model Context Protocol infrastructure from local scripts to AWS ECS and EKS is a pivotal step in maturing your AI operations. By leveraging Fargate for simplicity or EKS for mesh-scale orchestration, you provide your AI agents with a stable, high-performance environment to operate in.

Remember, “Powering Up” isn’t just about connectivity; it’s about security. Whether you choose a VPN-based approach or the robust AWS SigV4 proxy pattern, ensuring your AI tools are authenticated is non-negotiable in a production environment.

Next Step: Audit your current local MCP tools. Identify one “heavy” tool (like a database inspector or a large-context retriever) and containerize it using the Dockerfile pattern above to deploy your first remote MCP service on Fargate. Thank you for reading the DevopsRoles page!

AWS

Agentic AI is Revolutionizing AWS Security Incident Response

Leave a comment

For years, the gold standard in cloud security has been defined by deterministic automation. We detect an anomaly in Amazon GuardDuty, trigger a CloudWatch Event (now EventBridge), and fire a Lambda function to execute a hard-coded remediation script. While effective for known threats, this approach is brittle. It lacks context, reasoning, and adaptability.

Enter Agentic AI. By integrating Large Language Models (LLMs) via services like Amazon Bedrock into your security stack, we are moving from static “Runbooks” to dynamic “Reasoning Engines.” AWS Security Incident Response is no longer just about automation; it is about autonomy. This guide explores how to architect Agentic workflows that can analyze forensics, reason through containment strategies, and execute remediation with human-level nuance at machine speed.

The Evolution: From SOAR to Agentic Security

Traditional Security Orchestration, Automation, and Response (SOAR) platforms rely on linear logic: If X, then Y. This works for blocking an IP address, but it fails when the threat requires investigation. For example, if an IAM role is exfiltrating data, a standard script might revoke keys immediately—potentially breaking production applications—whereas a human analyst would first check if the activity aligns with a scheduled maintenance window.

Agentic AI introduces the ReAct (Reasoning + Acting) pattern to AWS Security Incident Response. Instead of blindly firing scripts, the AI Agent:

  1. Observes the finding (e.g., “S3 Bucket Public Access Enabled”).
  2. Reasons about the context (Queries CloudTrail: “Who did this? Was it authorized?”).
  3. Acts using defined tools (Calls boto3 functions to correct the policy).
  4. Evaluates the result (Verifies the bucket is private).

GigaCode Pro-Tip:
Don’t confuse “Generative AI” with “Agentic AI.” Generative AI writes a report about the hack. Agentic AI logs into the console (via API) and fixes the hack. The differentiator is the Action Group.

Architecture: Building a Bedrock Security Agent

To modernize your AWS Security Incident Response, we leverage Amazon Bedrock Agents. This managed service orchestrates the interaction between the LLM (reasoning), the knowledge base (RAG for company policies), and the action groups (Lambda functions).

1. The Foundation: Knowledge Bases

Your agent needs context. Using Retrieval-Augmented Generation (RAG), you can index your internal Wiki, incident response playbooks, and architecture diagrams into an Amazon OpenSearch Serverless vector store connected to Bedrock. When a finding occurs, the agent first queries this base: “What is the protocol for a compromised EC2 instance in the Production VPC?”

2. Action Groups (The Hands)

Action groups map OpenAPI schemas to AWS Lambda functions. This allows the LLM to “call” Python code. Below is an example of a remediation tool that an agent might decide to use during an active incident.

Code Implementation: The Isolation Tool

This Lambda function serves as a “tool” that the Bedrock Agent can invoke when it decides an instance must be quarantined. 

import boto3
import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

ec2 = boto3.client('ec2')

def lambda_handler(event, context):
    """
    Tool for Bedrock Agent: Isolates an EC2 instance by attaching a forensic SG.
    Input: {'instance_id': 'i-xxxx', 'vpc_id': 'vpc-xxxx'}
    """
    agent_params = event.get('parameters', [])
    instance_id = next((p['value'] for p in agent_params if p['name'] == 'instance_id'), None)
    
    if not instance_id:
        return {"response": "Error: Instance ID is required for isolation."}

    try:
        # Logic to find or create a 'Forensic-No-Ingress' Security Group
        logger.info(f"Agent requested isolation for {instance_id}")
        
        # 1. Get current SG for rollback context (Forensics)
        current_attr = ec2.describe_instance_attribute(
            InstanceId=instance_id, Attribute='groupSet'
        )
        
        # 2. Attach Isolation SG (Assuming sg-isolation-id is pre-provisioned)
        isolation_sg = "sg-0123456789abcdef0" 
        
        ec2.modify_instance_attribute(
            InstanceId=instance_id,
            Groups=[isolation_sg]
        )
        
        return {
            "response": f"SUCCESS: Instance {instance_id} has been isolated. Previous SGs logged for analysis."
        }
        
    except Exception as e:
        logger.error(f"Failed to isolate: {str(e)}")
        return {"response": f"FAILED: Could not isolate instance. Reason: {str(e)}"}

Implementing the Workflow

Deploying this requires an Event-Driven Architecture. Here is the lifecycle of an Agentic AWS Security Incident Response:

  • Detection: GuardDuty detects UnauthorizedAccess:EC2/TorIPCaller.
  • Ingestion: EventBridge captures the finding and pushes it to an SQS queue (for throttling/buffering).
  • Invocation: A Lambda “Controller” picks up the finding and invokes the Bedrock Agent Alias using the invoke_agent API.
  • Reasoning Loop:
    • The Agent receives the finding details.
    • It checks the “Knowledge Base” and sees that Tor connections are strictly prohibited.
    • It decides to call the GetInstanceDetails tool to check tags.
    • It sees the tag Environment: Production.
    • It decides to call the IsolateInstance tool (code above).
  • Resolution: The Agent updates AWS Security Hub with the workflow status, marks the finding as RESOLVED, and emails the SOC team a summary of its actions.

Human-in-the-Loop (HITL) and Guardrails

For expert practitioners, the fear of “hallucinating” agents deleting production databases is real. To mitigate this in AWS Security Incident Response, we implement Guardrails for Amazon Bedrock.

Guardrails allow you to define denied topics and content filters. Furthermore, for high-impact actions (like terminating instances), you should design the Agent to request approval rather than execute immediately. The Agent can send an SNS notification with a standard “Approve/Deny” link. The Agent pauses execution until the approval signal is received via a callback webhook.

Pro-Tip: Use CloudTrail Lake to audit your Agents. Every API call made by the Agent (via the assumed IAM role) is logged. Create a QuickSight dashboard to visualize “Agent Remediation Success Rates” vs. “Human Intervention Required.”

Frequently Asked Questions (FAQ)

How does Agentic AI differ from AWS Lambda automation?

Lambda automation is deterministic (scripted steps). Agentic AI is probabilistic and reasoning-based. It can handle ambiguity, such as deciding not to act if a threat looks like a false positive based on cross-referencing logs, whereas a script would execute blindly.

Is it safe to let AI modify security groups automatically?

It is safe if scoped correctly using IAM Roles. The Agent’s role should adhere to the Principle of Least Privilege. Start with “Read-Only” agents that only perform forensics and suggest remediation, then graduate to “Active” agents for low-risk environments.

Which AWS services are required for this architecture?

At a minimum: Amazon Bedrock (Agents & Knowledge Bases), AWS Lambda (Action Groups), Amazon EventBridge (Triggers), Amazon GuardDuty (Detection), and AWS Security Hub (Centralized Management).

Conclusion

The landscape of AWS Security Incident Response is shifting. By adopting Agentic AI, organizations can reduce Mean Time to Respond (MTTR) from hours to seconds. However, this is not a “set and forget” solution. It requires rigorous engineering of prompts, action schemas, and IAM boundaries.

Start small: Build an agent that purely performs automated forensics—gathering logs, querying configurations, and summarizing the blast radius—before letting it touch your infrastructure. The future of cloud security is autonomous, and the architects who master these agents today will define the standards of tomorrow.

For deeper reading on configuring Bedrock Agents, consult the official AWS Bedrock User Guide or review the AWS Security Incident Response Guide.

Kubernetes

Kubernetes DRA: Optimize GPU Workloads with Dynamic Resource Allocation

Leave a comment

For years, Kubernetes Platform Engineers and SREs have operated under a rigid constraint: the Device Plugin API. While it served the initial wave of containerization well, its integer-based resource counting (e.g., nvidia.com/gpu: 1) is fundamentally insufficient for modern, high-performance AI/ML workloads. It lacks the nuance to handle topology awareness, arbitrary constraints, or flexible device sharing at the scheduler level.

Enter Kubernetes DRA (Dynamic Resource Allocation). This is not just a patch; it is a paradigm shift in how Kubernetes requests and manages hardware accelerators. By moving resource allocation logic out of the Kubelet and into the control plane (via the Scheduler and Resource Drivers), DRA allows for complex claim lifecycles, structured parameters, and significantly improved cluster utilization.

The Latency of Legacy: Why Device Plugins Are Insufficient

To understand the value of Kubernetes DRA, we must first acknowledge the limitations of the standard Device Plugin framework. In the “classic” model, the Scheduler is essentially blind. It sees nodes as bags of counters (Capacity/Allocatable). It does not know which specific GPU it is assigning, nor its topology (PCIe switch locality, NVLink capabilities) relative to other requested devices.

Pro-Tip: In the classic model, the actual device assignment happens at the Kubelet level, long after scheduling. If a Pod lands on a node that has free GPUs but lacks the specific topology required for efficient distributed training, you incur a silent performance penalty or a runtime failure.

The Core Limitations

  • Opaque Integers: You cannot request “A GPU with 24GB VRAM.” You can only request “1 Unit” of a device, requiring complex node labeling schemes to separate hardware tiers.
  • Late Binding: Allocation happens at container creation time (StartContainer), making it impossible for the scheduler to make globally optimal decisions based on device attributes.
  • No Cross-Pod Sharing: Device Plugins generally assume exclusive access or rigid time-slicing, lacking native API support for dynamic sharing of a specific device instance across Pods.

Architectural Deep Dive: How Kubernetes DRA Works

Kubernetes DRA decouples the resource definition from the Pod spec. It introduces a new API group, resource.k8s.io, and a set of Custom Resource Definitions (CRDs) that treat hardware requests similarly to Persistent Volume Claims (PVCs).

1. The Shift to Control Plane Allocation

Unlike Device Plugins, DRA involves the Scheduler directly. When utilizing the new Structured Parameters model (promoted in K8s 1.30+), the scheduler can make decisions based on the actual attributes of the devices without needing to call out to an external driver for every Pod decision, dramatically reducing scheduling latency compared to early alpha DRA implementations.

2. Core API Objects

If you are familiar with PVCs and StorageClasses, the DRA mental model will feel intuitive.

API Object Role Analogy
ResourceClass Defines the driver and common parameters for a type of hardware. StorageClass
ResourceClaim A request for a specific device instance satisfying certain constraints. PVC (Persistent Volume Claim)
ResourceSlice Published by the driver; advertises available resources and their attributes to the cluster. PV (but dynamic and granular)
DeviceClass (New in Structured Parameters) Defines a set of configuration presets or hardware selectors. Hardware Profile

Implementing DRA: A Practical Workflow

Let’s look at how to implement Kubernetes DRA for a GPU workload. We assume a cluster running Kubernetes 1.30+ with the DynamicResourceAllocation feature gate enabled.

Step 1: The ResourceClass

First, the administrator defines a class that points to the specific DRA driver (e.g., the NVIDIA DRA driver). 

apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClass
metadata:
  name: nvidia-gpu
driverName: dra.nvidia.com
structuredParameters: true  # Enabling the high-performance scheduler path

Step 2: The ResourceClaimTemplate

Instead of embedding requests in the Pod spec, we create a template. This allows the Pod to generate a unique ResourceClaim upon creation. Notice how we can now specify arbitrary selectors, not just counts. 

apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaimTemplate
metadata:
  name: gpu-claim-template
spec:
  metadata:
    labels:
      app: deep-learning
  spec:
    resourceClassName: nvidia-gpu
    parametersRef:
      kind: GpuConfig
      name: v100-high-mem
      apiGroup: dra.nvidia.com

Step 3: The Pod Specification

The Pod references the claim template. The Kubelet ensures the container is not started until the claim is “Allocated” and “Reserved.” 

apiVersion: v1
kind: Pod
metadata:
  name: model-training-pod
spec:
  containers:
  - name: trainer
    image: nvidia/cuda:12.0-base
    command: ["/bin/sh", "-c", "nvidia-smi; sleep 3600"]
    resources:
      claims:
      - name: gpu-access
  resourceClaims:
  - name: gpu-access
    source:
      resourceClaimTemplateName: gpu-claim-template

Advanced Concept: Unlike PVCs, ResourceClaims have a allocationMode. Setting this to WaitForFirstConsumer (similar to storage) ensures that the GPU is not locked to a node until the Pod is actually scheduled, preventing resource fragmentation.

Structured Parameters: The “Game Changer” for Scheduler Performance

Early iterations of DRA had a major flaw: the Scheduler had to communicate with a sidecar controller via gRPC for every pod to check if a claim could be satisfied. This was too slow for large clusters.

Structured Parameters (introduced in KEP-3063) solves this.

  • How it works: The Driver publishes ResourceSlice objects containing the device inventory and opaque parameters. However, the constraints are defined in a standardized format that the Scheduler understands natively.
  • The Result: The generic Kubernetes Scheduler can calculate which node satisfies a ResourceClaim entirely in-memory, without network round-trips to external drivers. It only calls the driver for the final “Allocation” confirmation.

Best Practices for Production DRA

As you migrate from Device Plugins to DRA, keep these architectural constraints in mind:

  1. Namespace Isolation: Unlike device plugins which are node-global, ResourceClaims are namespaced. This provides better multi-tenancy security but requires stricter RBAC management for the resource.k8s.io API group.
  2. CDI Integration: DRA relies heavily on the Container Device Interface (CDI) for the actual injection of device nodes into containers. Ensure your container runtime (containerd/CRI-O) is updated to a version that supports CDI injection fully.
  3. Monitoring: The old metric kubelet_device_plugin_allocations will no longer tell the full story. You must monitor `ResourceClaim` statuses. A claim stuck in Pending often indicates that no `ResourceSlice` satisfies the topology constraints.

Frequently Asked Questions (FAQ)

Is Kubernetes DRA ready for production?

As of Kubernetes 1.30, DRA is in Beta. While the API is stabilizing, the ecosystem of drivers (Intel, NVIDIA, AMD) is still maturing. For critical, high-uptime production clusters, a hybrid approach is recommended: keep critical workloads on Device Plugins and experiment with DRA for batch AI jobs.

Can I use DRA and Device Plugins simultaneously?

Yes. You can run the NVIDIA Device Plugin and the NVIDIA DRA Driver on the same node. However, you must ensure they do not manage the same physical devices to avoid conflicts. Typically, this is done by using node labels to segregate “Legacy Nodes” from “DRA Nodes.”

Does DRA support GPU sharing (MIG/Time-Slicing)?

Yes, and arguably better than before. DRA allows drivers to expose “Shared” claims where multiple Pods reference the same `ResourceClaim` object, or where the driver creates multiple slices representing fractions of a physical GPU (e.g., MIG instances) with distinct attributes.

Conclusion

Kubernetes DRA represents the maturation of Kubernetes as a platform for high-performance computing. By treating devices as first-class schedulable resources rather than opaque counters, we unlock the ability to manage complex topologies, improve cluster density, and standardize how we consume hardware.

While the migration requires learning new API objects like ResourceClaim and ResourceSlice, the control it offers over GPU workloads makes it an essential upgrade for any serious AI/ML platform team. Thank you for reading the DevopsRoles page!

Docker

Developing Secure Software: Docker & Sonatype at Scale

Leave a comment

In the era of Log4Shell and SolarWinds, the mandate for engineering leaders is clear: security cannot be a gatekeeper at the end of the release cycle; it must be the pavement on which the pipeline runs. Developing secure software at an enterprise scale requires more than just scanning code—it demands a comprehensive orchestration of the software supply chain.

For organizations leveraging the Docker ecosystem, the challenge is twofold: ensuring the base images are immutable and trusted, and ensuring the application artifacts injected into those images are free from malicious dependencies. This is where the synergy between Docker’s containerization standards and Sonatype’s Nexus platform (Lifecycle and Repository) becomes critical.

This guide moves beyond basic setup instructions. We will explore architectural strategies for integrating Sonatype Nexus IQ with Docker registries, implementing policy-as-code in CI/CD, and managing the noise of vulnerability reporting to maintain high-velocity deployments.

The Supply Chain Paradigm: Beyond Simple Scanning

To succeed in developing secure software, we must acknowledge that modern applications are 80-90% open-source components. The “code” your developers write is often just glue logic binding third-party libraries together. Therefore, the security posture of your Docker container is directly inherited from the upstream supply chain.

Enterprise strategies must align with frameworks like the NIST Secure Software Development Framework (SSDF) and SLSA (Supply-chain Levels for Software Artifacts). The goal is not just to find bugs, but to establish provenance and governance.

Pro-Tip for Architects: Don’t just scan build artifacts. Implement a “Nexus Firewall” at the proxy level. If a developer requests a library with a CVSS score of 9.8, the proxy should block the download entirely, preventing the vulnerability from ever entering your ecosystem. This is “Shift Left” in its purest form.

Architecture: Integrating Nexus IQ with Docker Registries

At scale, you cannot rely on developers manually running CLI scans. Integration must be seamless. A robust architecture typically involves three layers of defense using Sonatype Nexus and Docker.

1. The Proxy Layer (Ingestion)

Configure Nexus Repository Manager (NXRM) as a proxy for Docker Hub. All `docker pull` requests should go through NXRM. This allows you to cache images (improving build speeds) and, more importantly, inspect them.

2. The Build Layer (CI Integration)

This is where the Nexus IQ Server comes into play. During the build, the CI server (Jenkins, GitLab CI, GitHub Actions) generates an SBOM (Software Bill of Materials) of the application and sends it to Nexus IQ for policy evaluation.

3. The Registry Layer (Continuous Monitoring)

Even if an image is safe today, it might be vulnerable tomorrow (Zero-Day). Nexus Lifecycle offers “Continuous Monitoring” for artifacts stored in the repository, alerting you to new CVEs in old images without requiring a rebuild.

Policy-as-Code: Enforcement in CI/CD

Developing secure software effectively means automating decision-making. Policies should be defined in Nexus IQ (e.g., “No Critical CVEs in Production App”) and enforced by the pipeline.

Below is a production-grade Jenkinsfile snippet demonstrating how to enforce a blocking policy using the Nexus Platform Plugin. Note the use of failBuildOnNetworkError to ensure fail-safe behavior.

pipeline {
    agent any
    stages {
        stage('Build & Package') {
            steps {
                sh 'mvn clean package -DskipTests' // Create the artifact
                sh 'docker build -t my-app:latest .' // Build the container
            }
        }
        stage('Sonatype Policy Evaluation') {
            steps {
                script {
                    // Evaluate the application JARs and the Docker Image
                    nexusPolicyEvaluation failBuildOnNetworkError: true,
                                          iqApplication: 'payment-service-v2',
                                          iqStage: 'build',
                                          iqScanPatterns: [[pattern: 'target/*.jar'], [pattern: 'Dockerfile']]
                }
            }
        }
        stage('Push to Registry') {
            steps {
                // Only executes if Policy Evaluation passes
                sh 'docker push private-repo.corp.com/my-app:latest'
            }
        }
    }
}

By scanning the Dockerfile and the application binaries simultaneously, you catch OS-level vulnerabilities (e.g., glibc issues in the base image) and Application-level vulnerabilities (e.g., log4j in the Java classpath).

Optimizing Docker Builds for Security

While Sonatype handles the governance, the way you construct your Docker images fundamentally impacts your risk profile. Expert teams minimize the attack surface using Multi-Stage Builds and Distroless images.

This approach removes build tools (Maven, GCC, Gradle) and shells from the final runtime image, making it significantly harder for attackers to achieve persistence or lateral movement.

Secure Dockerfile Pattern

# Stage 1: The Build Environment
FROM maven:3.8.6-eclipse-temurin-17 AS builder
WORKDIR /app
COPY pom.xml .
COPY src ./src
RUN mvn package -DskipTests

# Stage 2: The Runtime Environment
# Using Google's Distroless image for Java 17
# No shell, no package manager, minimal CVE footprint
FROM gcr.io/distroless/java17-debian11
COPY --from=builder /app/target/my-app.jar /app/my-app.jar
WORKDIR /app
CMD ["my-app.jar"]

Pro-Tip: When scanning distroless images or stripped binaries, standard scanners often fail because they rely on package managers (like apt or apk) to list installed software. Sonatype’s “Advanced Binary Fingerprinting” is superior here as it identifies components based on hash signatures rather than package manifests.

Scaling Operations: Automated Waivers & API Magic

The biggest friction point in developing secure software is the “False Positive” or the “Unfixable Vulnerability.” If you block builds for a vulnerability that has no patch available, developers will revolt.

To handle this at scale, you must utilize the Nexus IQ Server API. You can script logic that automatically grants temporary waivers for vulnerabilities that meet specific criteria (e.g., “Vendor status: Will Not Fix” AND “CVSS < 7.0”).

Here is a conceptual example of how to interact with the API to manage waivers programmatically:

# Pseudo-code for automating waivers via Nexus IQ API
import requests

IQ_SERVER = "https://iq.corp.local"
APP_ID = "payment-service-v2"
AUTH = ('admin', 'password123')

def apply_waiver(violation_id, reason):
    endpoint = f"{IQ_SERVER}/api/v2/policyViolations/{violation_id}/waiver"
    payload = {
        "comment": reason,
        "expiryTime": "2025-12-31T23:59:59.999Z" # Waiver expires in future
    }
    response = requests.post(endpoint, json=payload, auth=AUTH)
    if response.status_code == 200:
        print(f"Waiver applied for {violation_id}")

# Logic: If vulnerability is effectively 'noise', auto-waive it
# This prevents the pipeline from breaking on non-actionable items

Frequently Asked Questions (FAQ)

How does Sonatype IQ differ from ‘docker scan’?

docker scan (often powered by Snyk) is excellent for ad-hoc developer checks. Sonatype IQ is an enterprise governance platform. It provides centralized policy management, legal compliance (license checking), and deep binary fingerprinting that persists across the entire SDLC, not just the local machine.

What is the performance impact of scanning in CI/CD?

A full binary scan can take time. To optimize, ensure your Nexus IQ Server is co-located (network-wise) with your CI runners. Additionally, utilize the “Proprietary Code” settings in Nexus to exclude your internal JARs/DLLs from being fingerprinted against the public Central Repository, which speeds up analysis significantly.

How do we handle “InnerSource” components?

Large enterprises often reuse internal libraries. You should publish these to a hosted repository in Nexus. By configuring your policies correctly, you can ensure that consuming applications verify the version age and quality of these internal components, applying the same rigor to internal code as you do to open source.

Conclusion

Developing secure software using Docker and Sonatype at scale is not an endpoint; it is a continuous operational practice. It requires shifting from a reactive “patching” mindset to a proactive “supply chain management” mindset.

By integrating Nexus Firewall to block bad components at the door, enforcing Policy-as-Code in your CI/CD pipelines, and utilizing minimal Docker base images, you create a defense-in-depth strategy. This allows your organization to innovate at the speed of Docker, with the assurance and governance required by the enterprise.

Next Step: Audit your current CI pipeline. If you are running scans but not blocking builds on critical policy violations, you are gathering data, not securing software. Switch your Nexus action from “Warn” to “Fail” for CVSS 9+ vulnerabilities today. Thank you for reading the DevopsRoles page!

Kubernetes

Kubernetes Migration: Strategies & Best Practices

Leave a comment

For the modern enterprise, the question is no longer if you will adopt cloud-native orchestration, but how you will manage the transition. Kubernetes migration is rarely a linear process; it is a complex architectural shift that demands a rigorous understanding of distributed systems, state persistence, and networking primitives. Whether you are moving legacy monoliths from bare metal to K8s, or orchestrating a multi-cloud cluster-to-cluster shift, the margin for error is nonexistent.

This guide is designed for Senior DevOps Engineers and SREs. We will bypass the introductory concepts and dive straight into the strategic patterns, technical hurdles of stateful workloads, and zero-downtime cutover techniques required for a successful production migration.

The Architectural Landscape of Migration

A successful Kubernetes migration is 20% infrastructure provisioning and 80% application refactoring and data gravity management. Before a single YAML manifest is applied, the migration path must be categorized based on the source and destination architectures.

Types of Migration Contexts

  • V2C (VM to Container): The classic modernization path. Requires containerization (Dockerfiles), defining resource limits, and decoupling configuration from code (12-Factor App adherence).
  • C2C (Cluster to Cluster): Moving from on-prem OpenShift to EKS, or GKE to EKS. This involves handling API version discrepancies, CNI (Container Network Interface) translation, and Ingress controller mapping.
  • Hybrid/Multi-Cloud: Spanning workloads across clusters. Complexity lies in service mesh implementation (Istio/Linkerd) and consistent security policies.

GigaCode Pro-Tip: In C2C migrations, strictly audit your API versions using tools like kubent (Kube No Trouble) before migration. Deprecated APIs in the source cluster (e.g., v1beta1 Ingress) will cause immediate deployment failures in a newer destination cluster version.

Strategic Patterns: The 6 Rs in a K8s Context

While the “6 Rs” of cloud migration are standard, their application in a Kubernetes migration is distinct.

1. Rehost (Lift and Shift)

Wrapping a legacy binary in a container without code changes. While fast, this often results in “fat containers” that behave like VMs (using SupervisorD, lacking liveness probes, local logging).

Best for: Low-criticality internal apps or immediate datacenter exits.

2. Replatform (Tweak and Shift)

Moving to containers while replacing backend services with cloud-native equivalents. For example, migrating a local MySQL instance inside a VM to Amazon RDS or Google Cloud SQL, while the application moves to Kubernetes.

3. Refactor (Re-architect)

Breaking a monolith into microservices to fully leverage Kubernetes primitives like scaling, self-healing, and distinct release cycles.

Technical Deep Dive: Migrating Stateful Workloads

Stateless apps are trivial to migrate. The true challenge in any Kubernetes migration is Data Gravity. Handling StatefulSets and PersistentVolumeClaims (PVCs) requires ensuring data integrity and minimizing Return to Operation (RTO) time.

CSI and Volume Snapshots

Modern migrations rely heavily on the Container Storage Interface (CSI). If you are migrating between clusters (C2C), you cannot simply “move” a PV. You must replicate the data.

Migration Strategy: Velero with Restic/Kopia

Velero is the industry standard for backing up and restoring Kubernetes cluster resources and persistent volumes. For storage backends that do not support native snapshots across different providers, Velero integrates with Restic (or Kopia in newer versions) to perform file-level backups of PVC data.

# Example: Creating a backup including PVCs using Velero
velero backup create migration-backup \
  --include-namespaces production-app \
  --default-volumes-to-fs-backup \
  --wait

Upon restoration in the target cluster, Velero reconstructs the Kubernetes objects (Deployments, Services, PVCs) and hydrates the data into the new StorageClass defined in the destination.

Database Migration Patterns

For high-throughput databases, file-level backup/restore is often too slow (high downtime). Instead, utilize replication:

  1. Setup a Replica: Configure a read-replica in the destination Kubernetes cluster (or managed DB service) pointing to the source master.
  2. Sync: Allow replication lag to drop to near zero.
  3. Promote: During the maintenance window, stop writes to the source, wait for the final sync, and promote the destination replica to master.

Zero-Downtime Cutover Strategies

Once the workload is running in the destination environment, switching traffic is the highest-risk phase. A “Big Bang” DNS switch is rarely advisable for high-traffic systems.

1. DNS Weighted Routing (Canary Cutover)

Utilize DNS providers (like AWS Route53 or Cloudflare) to shift traffic gradually. Start with a 5% weight to the new cluster’s Ingress IP.

2. Ingress Shadowing (Dark Traffic)

Before the actual cutover, mirror production traffic to the new cluster to validate performance without affecting real users. This can be achieved using Service Mesh capabilities (like Istio) or Nginx ingress annotations.

# Example: Nginx Ingress Mirroring Annotation
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: production-ingress
  annotations:
    nginx.ingress.kubernetes.io/mirror-target-service: "new-cluster-endpoint"
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: legacy-service
            port:
              number: 80

CI/CD and GitOps Adaptation

A Kubernetes migration is the perfect opportunity to enforce GitOps. Migrating pipeline logic (Jenkins, GitLab CI) directly to Kubernetes manifests managed by ArgoCD or Flux ensures that the “source of truth” for your infrastructure is version controlled.

When migrating pipelines:

  • Abstraction: Replace complex imperative deployment scripts (kubectl apply -f ...) with Helm Charts or Kustomize overlays.
  • Secret Management: Move away from environment variables stored in CI tools. Adopt Secrets Store CSI Driver (Vault/AWS Secrets Manager) or Sealed Secrets.

Frequently Asked Questions (FAQ)

How do I handle disparate Ingress Controllers during migration?

If moving from AWS ALB Ingress to Nginx Ingress, the annotations will differ significantly. Use a “Translation Layer” approach: Use Helm to template your Ingress resources. Define values files for the source (ALB) and destination (Nginx) that render the correct annotations dynamically, allowing you to deploy to both environments from the same codebase during the transition.

What is the biggest risk in Kubernetes migration?

Network connectivity and latency. Often, migrated services in the new cluster need to communicate with legacy services left behind on-prem or in a different VPC. Ensure you have established robust peering, VPNs, or Transit Gateways before moving applications to prevent timeouts.

Should I migrate stateful workloads to Kubernetes at all?

This is a contentious topic. For experts, the answer is: “Yes, if you have the operational maturity.” Operators (like the Prometheus Operator or Postgres Operator) make managing stateful apps easier, but if your team lacks deep K8s storage knowledge, offloading state to managed services (RDS, Cloud SQL) lowers the migration risk profile significantly.

Conclusion

Kubernetes migration is a multifaceted engineering challenge that extends far beyond simple containerization. It requires a holistic strategy encompassing data persistence, traffic shaping, and observability.

By leveraging tools like Velero for state transfer, adopting GitOps for configuration consistency, and utilizing weighted DNS for traffic cutovers, you can execute a migration that not only modernizes your stack but does so with minimal risk to the business. The goal is not just to be on Kubernetes, but to operate a platform that is resilient, scalable, and easier to manage than the legacy system it replaces. Thank you for reading the DevopsRoles page!

Terraform

Unlock Reusable VPC Modules: Terraform for Dev/Stage/Prod Environments

Leave a comment

If you are managing infrastructure at scale, you have likely felt the pain of the “copy-paste” sprawl. You define a VPC for Development, then copy the code for Staging, and again for Production, perhaps changing a CIDR block or an instance count manually. This breaks the fundamental DevOps principle of DRY (Don’t Repeat Yourself) and introduces drift risk.

For Senior DevOps Engineers and SREs, the goal isn’t just to write code that works; it’s to architect abstractions that scale. Reusable VPC Modules are the cornerstone of a mature Infrastructure as Code (IaC) strategy. They allow you to define the “Gold Standard” for networking once and instantiate it infinitely across environments with predictable results.

In this guide, we will move beyond basic syntax. We will construct a production-grade, agnostic VPC module capable of dynamic subnet calculation, conditional resource creation (like NAT Gateways), and strict variable validation suitable for high-compliance Dev, Stage, and Prod environments.

Why Reusable VPC Modules Matter (Beyond DRY)

While reducing code duplication is the obvious benefit, the strategic value of modularizing your VPC architecture runs deeper.

  • Governance & Compliance: By centralizing your network logic, you enforce security standards (e.g., “Flow Logs must always be enabled” or “Private subnets must not have public IP assignment”) in a single location.
  • Testing & Versioning: You can version your module (e.g., v1.2.0). Production can remain pinned to a stable version while you iterate on features in Development, effectively applying software engineering lifecycles to your network.
  • Abstraction Complexity: A consumer of your module (perhaps a developer spinning up an ephemeral environment) shouldn’t need to understand Route Tables or NACLs. They should only need to provide a CIDR block and an Environment name.

Pro-Tip: Avoid the “God Module” anti-pattern. While it’s tempting to bundle the VPC, EKS, and RDS into one giant module, this leads to dependency hell. Keep your Reusable VPC Modules strictly focused on networking primitives: VPC, Subnets, Route Tables, Gateways, and ACLs.

Anatomy of a Production-Grade Module

Let’s build a module that calculates subnets dynamically based on Availability Zones (AZs) and handles environment-specific logic (like high availability in Prod vs. cost savings in Dev).

1. Input Strategy & Validation

Modern Terraform (v1.0+) allows for powerful variable validation. We want to ensure that downstream users don’t accidentally pass invalid CIDR blocks. 

# modules/vpc/variables.tf

variable "environment" {
  description = "Deployment environment (dev, stage, prod)"
  type        = string
  validation {
    condition     = contains(["dev", "stage", "prod"], var.environment)
    error_message = "Environment must be one of: dev, stage, prod."
  }
}

variable "vpc_cidr" {
  description = "CIDR block for the VPC"
  type        = string
  validation {
    condition     = can(cidrhost(var.vpc_cidr, 0))
    error_message = "Must be a valid IPv4 CIDR block."
  }
}

variable "az_count" {
  description = "Number of AZs to utilize"
  type        = number
  default     = 2
}

2. Dynamic Subnetting with cidrsubnet

Hardcoding subnet CIDRs (e.g., 10.0.1.0/24) is brittle. Instead, use the cidrsubnet function to mathematically carve up the VPC CIDR. This ensures no overlap and automatic scalability if you change the base CIDR size. 

# modules/vpc/main.tf

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.environment}-vpc"
  }
}

# Public Subnets
resource "aws_subnet" "public" {
  count                   = var.az_count
  vpc_id                  = aws_vpc.main.id
  # Example: 10.0.0.0/16 -> 10.0.1.0/24, 10.0.2.0/24, etc.
  cidr_block              = cidrsubnet(var.vpc_cidr, 8, count.index)
  availability_zone       = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.environment}-public-${data.aws_availability_zones.available.names[count.index]}"
    Tier = "Public"
  }
}

# Private Subnets
resource "aws_subnet" "private" {
  count             = var.az_count
  vpc_id            = aws_vpc.main.id
  # Offset the CIDR calculation by 'az_count' to avoid overlap with public subnets
  cidr_block        = cidrsubnet(var.vpc_cidr, 8, count.index + var.az_count)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name = "${var.environment}-private-${data.aws_availability_zones.available.names[count.index]}"
    Tier = "Private"
  }
}

3. Conditional NAT Gateways (Cost Optimization)

NAT Gateways are expensive. In a Dev environment, you might only need one shared NAT Gateway (or none if you use instances with public IPs for testing), whereas Prod requires High Availability (one NAT per AZ). 

# modules/vpc/main.tf

locals {
  # If Prod, create NAT per AZ. If Dev/Stage, create only 1 NAT total to save costs.
  nat_gateway_count = var.environment == "prod" ? var.az_count : 1
}

resource "aws_eip" "nat" {
  count = local.nat_gateway_count
  domain = "vpc"
}

resource "aws_nat_gateway" "main" {
  count         = local.nat_gateway_count
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id

  tags = {
    Name = "${var.environment}-nat-${count.index}"
  }
}

Implementing Across Environments

Once your Reusable VPC Module is polished, utilizing it across environments becomes a trivial exercise in configuration management. I recommend a directory-based structure over Terraform Workspaces for clearer isolation of state files and variable definitions.

Directory Structure

infrastructure/
├── modules/
│   └── vpc/ (The code we wrote above)
├── environments/
│   ├── dev/
│   │   └── main.tf
│   ├── stage/
│   │   └── main.tf
│   └── prod/
│       └── main.tf

The Implementation (DRY at work)

In environments/prod/main.tf, your code is now incredibly concise: 

module "vpc" {
  source      = "../../modules/vpc"
  
  environment = "prod"
  vpc_cidr    = "10.0.0.0/16"
  az_count    = 3 # High Availability
}

Contrast this with environments/dev/main.tf: 

module "vpc" {
  source      = "../../modules/vpc"
  
  environment = "dev"
  vpc_cidr    = "10.10.0.0/16" # Different CIDR
  az_count    = 2 # Lower cost
}

Advanced Patterns & Considerations

Tagging Standards

Effective tagging is non-negotiable for cost allocation and resource tracking. Use the default_tags feature in the AWS provider configuration to apply global tags, but ensure your module accepts a tags map variable to merge specific metadata.

Outputting Values for Dependency Injection

Your VPC module is likely the foundation for other modules (like EKS or RDS). Ensure you output the IDs required by these dependent resources. 

# modules/vpc/outputs.tf

output "vpc_id" {
  value = aws_vpc.main.id
}

output "private_subnet_ids" {
  value = aws_subnet.private[*].id
}

output "public_subnet_ids" {
  value = aws_subnet.public[*].id
}

Frequently Asked Questions (FAQ)

Should I use the official terraform-aws-modules/vpc/aws or build my own?

For beginners or rapid prototyping, the community module is excellent. However, for Expert SRE teams, building your own Reusable VPC Module is often preferred. It reduces “bloat” (unused features from the community module) and allows strict adherence to internal naming conventions and security compliance logic that a generic module cannot provide.

How do I handle VPC Peering between these environments?

Generally, you should avoid peering Dev and Prod. However, if you need shared services (like a tooling VPC), create a separate vpc-peering module. Do not bake peering logic into the core VPC module, as it creates circular dependencies and makes the module difficult to destroy.

What about VPC Flow Logs?

Flow Logs should be a standard part of your reusable module. I recommend adding a variable enable_flow_logs (defaulting to true) and storing logs in S3 or CloudWatch Logs. This ensures that every environment spun up with your module has auditing enabled by default.

Conclusion

Transitioning to Reusable VPC Modules transforms your infrastructure from a collection of static scripts into a dynamic, versioned product. By abstracting the complexity of subnet math and resource allocation, you empower your team to deploy Dev, Stage, and Prod environments that are consistent, compliant, and cost-optimized.

Start refactoring your hardcoded network configurations today. Isolate your logic into a module, version it, and watch your drift disappear. Thank you for reading the DevopsRoles page!

Docker

Monitor Docker: Efficient Container Monitoring Across All Servers with Beszel

Leave a comment

In the world of Docker container monitoring, we often pay a heavy “Observability Tax.” We deploy complex stacks—Prometheus, Grafana, Node Exporter, cAdvisor—just to check if a container is OOM (Out of Memory). For large Kubernetes clusters, that complexity is justified. For a fleet of Docker servers, home labs, or edge devices, it’s overkill.

Enter Beszel. It is a lightweight monitoring hub that fundamentally changes the ROI of observability. It gives you historical CPU, RAM, and Disk I/O data, plus specific Docker stats for every running container, all while consuming less than 10MB of RAM.

This guide is for the expert SysAdmin or DevOps engineer who wants robust metrics without the bloat. We will deploy the Beszel Hub, configure Agents with hardened security settings, and set up alerting.

Why Beszel for Docker Environments?

Unlike push-based models that require heavy scrappers, or agentless models that lack granularity, Beszel uses a Hub-and-Agent architecture designed for efficiency.

  • Low Overhead: The agent is a single binary (packaged in a container) that typically uses negligible CPU and <15MB RAM.
  • Docker Socket Integration: By mounting the Docker socket, the agent automatically discovers running containers and pulls stats (CPU/MEM %) directly from the daemon.
  • Automatic Alerts: No complex PromQL queries. You get out-of-the-box alerting for disk pressure, memory spikes, and offline status.

Pro-Tip: Beszel is distinct from “Uptime Monitors” (like Uptime Kuma) because it tracks resource usage trends inside the container, not just HTTP 200 OK statuses.

Step 1: Deploying the Beszel Hub (Control Plane)

The Hub is the central dashboard. It ingests metrics from all your agents. We will use Docker Compose to define it.

Hub Configuration

services:
  beszel:
    image: 'henrygd/beszel:latest'
    container_name: 'beszel'
    restart: unless-stopped
    ports:
      - '8090:8090'
    volumes:
      - ./beszel_data:/beszel_data

Deployment:

Run docker compose up -d. Navigate to http://your-server-ip:8090 and create your admin account.

Step 2: Deploying the Agent (Data Plane)

This is where the magic happens. The agent sits on your Docker hosts, collects metrics, and pushes them to the Hub.

Prerequisite: In the Hub UI, click “Add System”. Enter the IP of the node you want to monitor. The Hub will generate a Public Key. You need this key for the agent configuration.

The Hardened Agent Compose File

We use network_mode: host to allow the agent to accurately report network interface statistics for the host machine. We also mount the Docker socket in read-only mode to adhere to the Principle of Least Privilege. 

services:
  beszel-agent:
    image: 'henrygd/beszel-agent:latest'
    container_name: 'beszel-agent'
    restart: unless-stopped
    network_mode: host
    volumes:
      # Critical: Mount socket RO (Read-Only) for security
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # Optional: Mount extra partitions if you want to monitor specific disks
      # - /mnt/storage:/extra-filesystems/sdb1:ro
    environment:
      - PORT=45876
      - KEY=YOUR_PUBLIC_KEY_FROM_HUB
      # - FILESYSTEM=/dev/sda1 # Optional: Override default root disk monitoring

Technical Breakdown

  • /var/run/docker.sock:ro: This is the critical line for Docker Container Monitoring. It allows the Beszel agent to query the Docker Daemon API to fetch real-time stats (CPU shares, memory usage) for other containers running on the host. The :ro flag ensures the agent cannot modify or stop your containers.
  • network_mode: host: Without this, the agent would only report network traffic for its own container, which is useless for host monitoring.

Step 3: Advanced Alerting & Notification

Beszel simplifies alerting. Instead of writing alert rules in YAML files, you configure them in the GUI.

Go to Settings > Notifications. You can configure:

  • Webhooks: Standard JSON payloads for integration with custom dashboards or n8n workflows.
  • Discord/Slack: Paste your channel webhook URL.
  • Email (SMTP): For traditional alerts.

Expert Strategy: Configure a “System Offline” alert with a 2-minute threshold. Since Beszel agents push data, the Hub immediately knows when a heartbeat is missed, providing faster “Server Down” alerts than external ping checks that might be blocked by firewalls.

Comparison: Beszel vs. Prometheus Stack

For experts deciding between the two, here is the resource reality:

FeatureBeszelPrometheus + Grafana + Exporters
RAM Usage (Agent)~10-15 MB100MB+ (Node Exporter + cAdvisor)
Setup Time< 5 MinutesHours (Configuring targets, dashboards)
Data RetentionSQLite (Auto-pruning)TSDB (Requires management for long-term)
Ideal Use CaseVPS Fleets, Home Labs, Docker HostsKubernetes Clusters, Microservices Tracing

Frequently Asked Questions (FAQ)

Is it safe to expose the Docker socket?

Mounting docker.sock always carries risk. However, by mounting it as read-only (:ro), you mitigate the risk of the agent (or an attacker inside the agent) modifying your container states. The agent only reads metrics; it does not issue commands.

Can I monitor remote servers behind a NAT/Firewall?

Yes. Because the Agent connects to the Hub (or the Hub can connect to the agent, but the standard Docker setup usually relies on the Agent knowing the Hub’s location if using the binary, but in the Docker agent setup, the Hub scrapes the agent).

Correction for Docker Agent: The Hub actually polls the agent. Therefore, if your Agent is behind a NAT, you have two options:
1. Use a VPN (like Tailscale) to mesh the networks.
2. Use a reverse proxy (like Caddy or Nginx) on the Agent side to expose the port securely with SSL.

Does Beszel support GPU monitoring?

As of the latest versions, GPU monitoring (NVIDIA/AMD) is supported but may require passing specific hardware devices to the container or running the binary directly on the host for full driver access.

Conclusion

For Docker container monitoring, Beszel represents a shift towards “Just Enough Administration.” It removes the friction of maintaining the monitoring stack itself, allowing you to focus on the services you are actually hosting.

Your Next Step: Spin up the Beszel Hub on a low-priority VPS today. Add your most critical Docker host as a system using the :ro socket mount technique above. You will have full visibility into your container resource usage in under 10 minutes. Thank you for reading the DevopsRoles page!

Kubernetes

Boost Kubernetes: Fast & Secure with AKS Automatic

Leave a comment

For years, the “Promise of Kubernetes” has been somewhat at odds with the “Reality of Kubernetes.” While K8s offers unparalleled orchestration capabilities, the operational overhead for Platform Engineering teams is immense. You are constantly balancing node pool sizing, OS patching, upgrade cadences, and security baselining. Enter Kubernetes AKS Automatic.

This is not just another SKU; it is Microsoft’s answer to the “NoOps” paradigm, structurally similar to GKE Autopilot but deeply integrated into the Azure ecosystem. For expert practitioners, AKS Automatic represents a shift from managing infrastructure to managing workload definitions.

In this guide, we will dissect the architecture of Kubernetes AKS Automatic, evaluate the trade-offs regarding control vs. convenience, and provide Terraform implementation strategies for production-grade environments.

The Architectural Shift: Why AKS Automatic Matters

In a Standard AKS deployment, the responsibility model is split. Microsoft manages the Control Plane, but you own the Data Plane (Worker Nodes). If a node runs out of memory, or if an OS patch fails, that is your pager going off.

Kubernetes AKS Automatic changes this ownership model. It applies an opinionated configuration that enforces best practices by default.

1. Node Autoprovisioning (NAP)

Forget about calculating the perfect VM size for your node pools. AKS Automatic utilizes Node Autoprovisioning. Instead of static Virtual Machine Scale Sets (VMSS) that you define, NAP analyzes the pending pods in the scheduler. It looks at CPU/Memory requests, taints, and tolerations, and then spins up the exact compute resources required to fit those pods.

Pro-Tip: Under the Hood
NAP functions similarly to the open-source project Karpenter. It bypasses the traditional Cluster Autoscaler’s logic of scaling existing groups and instead provisions just-in-time compute capacity directly against the Azure Compute API.

2. Guardrails and Policies

AKS Automatic comes with Azure Policy enabled and configured in “Deny” mode for critical security baselines. This includes:

  • Disallowing Privileged Containers: Unless explicitly exempted.
  • Enforcing Resource Quotas: Pods without resource requests may be mutated or rejected to ensure the scheduler can make accurate placement decisions.
  • Network Security: Strict network policies are applied by default.

Deep Dive: Technical Specifications

For the Senior SRE, understanding the boundaries of the platform is critical. Here is what the stack looks like:

FeatureSpecification in AKS Automatic
CNI PluginAzure CNI Overlay (Powered by Cilium)
IngressManaged NGINX (via Application Routing add-on)
Service MeshIstio (Managed add-on available and recommended)
OS UpdatesFully Automated (Node image upgrades handled by Azure)
SLAProduction SLA (Uptime SLA) enabled by default

Implementation: Deploying AKS Automatic via Terraform

As of the latest Azure providers, deploying an Automatic cluster requires specific configuration flags. Below is a production-ready snippet using the azurerm provider.

Note: Ensure you are using an azurerm provider version > 3.100 or the 4.x series. 

resource "azurerm_kubernetes_cluster" "aks_automatic" {
  name                = "aks-prod-automatic-01"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "aks-prod-auto"

  # The key differentiator for Automatic SKU
  sku_tier = "Standard" # Automatic features are enabled via run_command or specific profile flags in current GA
  
  # Automatic typically requires Managed Identity
  identity {
    type = "SystemAssigned"
  }

  # Enable the Automatic feature profile
  # Note: Syntax may vary slightly based on Preview/GA status updates
  auto_scaler_profile {
    balance_similar_node_groups = true
  }

  # Network Profile defaults for Automatic
  network_profile {
    network_plugin      = "azure"
    network_plugin_mode = "overlay"
    network_policy      = "cilium"
    load_balancer_sku   = "standard"
  }

  # Enabling the addons associated with Automatic behavior
  maintenance_window {
    allowed {
        day   = "Saturday"
        hours = [21, 23]
    }
  }
  
  tags = {
    Environment = "Production"
    ManagedBy   = "Terraform"
  }
}

Note on IaC: Microsoft is rapidly iterating on the Terraform provider support for the specific sku_tier = "Automatic" alias. Always check the official Terraform AzureRM documentation for the breaking changes in the latest provider release.

The Trade-offs: What Experts Need to Know

Moving to Kubernetes AKS Automatic is not a silver bullet. You are trading control for operational velocity. Here are the friction points you must evaluate:

1. No SSH Access

You generally cannot SSH into the worker nodes. The nodes are treated as ephemeral resources.

The Fix: Use kubectl debug node/<node-name> -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0 to launch a privileged ephemeral container for debugging.

2. DaemonSet Complexity

Since you don’t control the node pools, running DaemonSets (like heavy security agents or custom logging forwarders) can be trickier. While supported, you must ensure your DaemonSets tolerate the taints applied by the Node Autoprovisioning logic.

3. Cost Implications

While you save on “slack” capacity (because you don’t have over-provisioned static node pools waiting for traffic), the unit cost of compute in managed modes can sometimes be higher than Spot instances managed manually. However, for 90% of enterprises, the reduction in engineering hours spent on upgrades outweighs the raw compute premium.

Frequently Asked Questions (FAQ)

Is AKS Automatic suitable for stateful workloads?

Yes. AKS Automatic supports Azure Disk and Azure Files CSI drivers. However, because nodes can be recycled more aggressively by the autoprovisioner, ensure your applications handle `SIGTERM` gracefully and that your Persistent Volume Claims (PVCs) utilize Retain policies where appropriate to prevent accidental data loss during rapid scaling events.

Can I use Spot Instances with AKS Automatic?

Yes, AKS Automatic supports Spot VMs. You define this intent in your workload manifest (PodSpec) using nodeSelector or tolerations specifically targeting spot capability, and the provisioner will attempt to fulfill the request with Spot capacity.

How does this differ from GKE Autopilot?

Conceptually, they are identical. The main difference lies in the ecosystem integration. AKS Automatic is deeply coupled with Azure Monitor, Azure Policy, and the specific versions of Azure CNI. If you are a multi-cloud shop, the developer experience (DX) is converging, but the underlying network implementation (Overlay vs VPC-native) differs.

Conclusion

Kubernetes AKS Automatic is the maturity of the cloud-native ecosystem manifesting in a product. It acknowledges that for most organizations, the value is in the application, not in curating the OS version of the worker nodes.

For the expert SRE, AKS Automatic allows you to refocus your efforts on higher-order problems: Service Mesh configurations, progressive delivery strategies (Canary/Blue-Green), and application resilience, rather than nursing a Node Pool upgrade at 2 AM.

Next Step: If you are running a Standard AKS cluster today, try creating a secondary node pool with Node Autoprovisioning enabled (preview features permitting) or spin up a sandbox AKS Automatic cluster to test your Helm charts against the stricter security policies. Thank you for reading the DevopsRoles page!

Kubernetes

Building the Largest Kubernetes Cluster: 130k Nodes & Beyond

Leave a comment

The official upstream documentation states that a single Kubernetes Cluster supports up to 5,000 nodes. For the average enterprise, this is overkill. For hyperscalers and platform engineers designing the next generation of cloud infrastructure, it’s merely a starting point.

When we talk about managing a fleet of 130,000 nodes, we enter a realm where standard defaults fail catastrophically. We are no longer just configuring software; we are battling the laws of physics regarding network latency, etcd storage quotas, and Go routine scheduling. This article dissects the architectural patterns, kernel tuning, and control plane sharding required to push a Kubernetes Cluster (or a unified fleet of clusters) to these extreme limits.

The “Singularity” vs. The Fleet: Defining the 130k Boundary

Before diving into the sysctl flags, let’s clarify the architecture. Running 130k nodes in a single control plane is currently theoretically impossible with vanilla upstream Kubernetes due to the etcd hard storage limit (8GB recommended max) and the sheer volume of watch events.

Achieving this scale requires one of two approaches:

  1. The “Super-Cluster” (Heavily Modified): Utilizing sharded API servers and segmented etcd clusters (splitting events from resources) to push a single cluster ID towards 10k–15k nodes.
  2. The Federated Fleet: Managing 130k nodes across multiple clusters via a unified control plane (like Karmada or custom controllers) that abstracts the “cluster” concept away from the user.

We will focus on optimizing the unit—the Kubernetes Cluster—to its absolute maximum, as these optimizations are prerequisites for any large-scale fleet.

Phase 1: Surgical Etcd Tuning

At scale, etcd is almost always the first bottleneck. In a default Kubernetes Cluster, etcd stores both cluster state (Pods, Services) and high-frequency events. At 10,000+ nodes, the write IOPS from Kubelet heartbeats and event recording will bring the cluster to its knees.

1. Vertical Sharding of Etcd

You must split your etcd topology. Never run events in the same etcd instance as your cluster configuration. 

# Example API Server flags to split storage
--etcd-servers="https://etcd-main-0:2379,https://etcd-main-1:2379,..."
--etcd-servers-overrides="/events#https://etcd-events-0:2379,https://etcd-events-1:2379,..."

2. Compression and Quotas

The default 2GB quota is insufficient. Increase the backend quota to 8GB (the practical safety limit). Furthermore, enable compression in the API server to reduce the payload size hitting etcd.

Pro-Tip: Monitor the etcd_mvcc_db_total_size_in_bytes metric religiously. If this hits the quota, your cluster enters a read-only state. Implement aggressive defragmentation schedules (e.g., every hour) for the events cluster, as high churn creates massive fragmentation.

Phase 2: The API Server & Control Plane

The kube-apiserver is the CPU-hungry brain. In a massive Kubernetes Cluster, the cost of serialization and deserialization (encoding/decoding JSON/Protobuf) dominates CPU cycles.

Priority and Fairness (APF)

Introduced to prevent controller loops from dDoSing the API server, APF is critical at scale. You must define custom FlowSchemas and PriorityLevelConfigurations. The default “catch-all” buckets will fill up instantly with 10k nodes, causing legitimate administrative calls (`kubectl get pods`) to time out. 

apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: system-critical-high
spec:
  type: Limited
  limited:
    assuredConcurrencyShares: 50
    limitResponse:
      type: Queue

Disable Unnecessary API Watches

Every node runs a kube-proxy and a kubelet. If you have 130k nodes, that is 130k watchers. If a significantly scoped change happens (like an EndpointSlice update), the API server must serialize that update 130k times.

  • Optimization: Use EndpointSlices instead of Endpoints.
  • Optimization: Set --watch-cache-sizes manually for high-churn resources to prevent cache misses which force expensive calls to etcd.

Phase 3: The Scheduler Throughput Challenge

The default Kubernetes scheduler evaluates every feasible node to find the “best” fit. With 130k nodes (or even 5k), scanning every node is O(N) complexity that results in massive scheduling latency.

You must tune the percentageOfNodesToScore parameter. 

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
percentageOfNodesToScore: 5  # Only look at 5% of nodes before making a decision

By lowering this to 5% (or even less in hyperscale environments), you trade a theoretical “perfect” placement for the ability to actually schedule pods in a reasonable timeframe.

Phase 4: Networking (CNI) at Scale

In a massive Kubernetes Cluster, iptables is the enemy. It relies on linear list traversal for rule updates. At 5,000 services, iptables becomes a noticeable CPU drag. At larger scales, it renders the network unusable.

IPVS vs. eBPF

While IPVS (IP Virtual Server) uses hash tables and offers O(1) complexity, modern high-scale clusters are moving entirely to eBPF (Extended Berkeley Packet Filter) solutions like Cilium.

  • Why: eBPF bypasses the host networking stack for pod-to-pod communication, significantly reducing latency and CPU overhead.
  • Identity Management: At 130k nodes, storing IP-to-Pod mappings is expensive. eBPF-based CNIs can use identity-based security policies rather than IP-based, which scales better in high-churn environments.

Phase 5: The Node (Kubelet) Perspective

Often overlooked, the Kubelet itself can dDoS the control plane.

  • Heartbeats: Adjust --node-status-update-frequency. In a 130k node environment (likely federated), you do not need 10-second heartbeats. Increasing this to 1 minute drastically reduces API server load.
  • Image Pulls: Serialize image pulls (`–serialize-image-pulls=false`) carefully. While parallel pulling is faster, it can spike disk I/O and network bandwidth, causing the node to go NotReady under load.

Frequently Asked Questions (FAQ)

What is the hard limit for a single Kubernetes Cluster?

As of Kubernetes v1.29+, the official scalability thresholds are 5,000 nodes, 150,000 total pods, and 300,000 total containers. Exceeding this requires significant customization of the control plane, specifically around etcd storage and API server caching mechanisms.

How do Alibaba and Google run larger clusters?

Tech giants often run customized versions of Kubernetes. They utilize techniques like “Cell” architectures (sharding the cluster into smaller failure domains), custom etcd storage drivers, and highly optimized networking stacks that replace standard Kube-Proxy implementations.

Should I use Federation or one giant cluster?

For 99% of use cases, Federation (multi-cluster) is superior. It provides better isolation, simpler upgrades, and drastically reduces the blast radius of a failure. Managing a single Kubernetes Cluster of 10k+ nodes is a high-risk operational endeavor.

Conclusion

Building a Kubernetes Cluster that scales toward the 130k node horizon is less about installing software and more about systems engineering. It requires a deep understanding of the interaction between the etcd key-value store, the Go runtime scheduler, and the Linux kernel networking stack.

While the allure of a single massive cluster is strong, the industry best practice for reaching this scale involves a sophisticated fleet management strategy. However, applying the optimizations discussed here-etcd sharding, APF tuning, and eBPF networking-will make your clusters, regardless of size, more resilient and performant. Thank you for reading the DevopsRoles page!

devops

Automate Software Delivery & Deployment with DevOps

Leave a comment

At the Senior Staff level, we know that DevOps automation is no longer just about writing bash scripts to replace manual server commands. It is about architecting self-sustaining platforms that treat infrastructure, security, and compliance as first-class software artifacts. In an era of microservices sprawl and multi-cloud complexity, the goal is to decouple deployment complexity from developer velocity.

This guide moves beyond the basics of CI/CD. We will explore how to implement rigorous DevOps automation strategies using GitOps patterns, Policy as Code (PaC), and ephemeral environments to achieve the elite performance metrics defined by DORA (DevOps Research and Assessment).

The Shift: From Scripting to Platform Engineering

Historically, automation was imperative: “Run this script to install Nginx.” Today, expert automation is declarative and convergent. We define the desired state, and autonomous controllers ensure the actual state matches it. This shift is crucial for scaling.

When we talk about automating software delivery in 2025, we are orchestrating a complex interaction between:

  • Infrastructure Provisioning: Dynamic, immutable resources.
  • Application Delivery: Progressive delivery (Canary/Blue-Green).
  • Governance: Automated guardrails that prevent bad configurations from ever reaching production.

Pro-Tip: Don’t just automate the “Happy Path.” True DevOps automation resilience comes from automating the failure domains—automatic rollbacks based on Prometheus metrics, self-healing infrastructure nodes, and automated certificate rotation.

Core Pillars of Advanced DevOps Automation

1. GitOps: The Single Source of Truth

GitOps elevates DevOps automation by using Git repositories as the source of truth for both infrastructure and application code. Tools like ArgoCD or Flux do not just “deploy”; they continuously reconcile the cluster state with the Git state.

This creates an audit trail for every change and eliminates “configuration drift”—the silent killer of reliability. If a human manually changes a Kubernetes deployment, the GitOps controller detects the drift and reverts it immediately.

2. Policy as Code (PaC)

In a high-velocity environment, manual security reviews are a bottleneck. PaC automates compliance. By using the Open Policy Agent (OPA), you can write policies that reject deployments if they don’t meet security standards (e.g., running as root, missing resource limits).

Here is a practical example of a Rego policy ensuring no container runs as root:

package kubernetes.admission

deny[msg] {
    input.request.kind.kind == "Pod"
    input.request.operation == "CREATE"
    container := input.request.object.spec.containers[_]
    container.securityContext.runAsNonRoot != true
    msg := sprintf("Container '%v' must set securityContext.runAsNonRoot to true", [container.name])
}

Integrating this into your pipeline or admission controller ensures that DevOps automation acts as a security gatekeeper, not just a delivery mechanism.

3. Ephemeral Environments

Static staging environments are often broken or outdated. A mature automation strategy involves spinning up full-stack ephemeral environments for every Pull Request. This allows QA and Product teams to test changes in isolation before merging.

Using tools like Crossplane or Terraform within your CI pipeline, you can provision a namespace, database, and ingress route dynamically, run integration tests, and tear it down automatically to save costs.

Orchestrating the Pipeline: A Modern Approach

To achieve true DevOps automation, your pipeline should resemble an assembly line with distinct stages of verification. It is not enough to simply build a Docker image; you must sign it, scan it, and attest its provenance.

Example: Secure Supply Chain Pipeline

Below is a conceptual high-level workflow for a secure, automated delivery pipeline:

  1. Code Commit: Triggers CI.
  2. Lint & Unit Test: Fast feedback loops.
  3. SAST/SCA Scan: Check for vulnerabilities in code and dependencies.
  4. Build & Sign: Build the artifact and sign it (e.g., Sigstore/Cosign).
  5. Deploy to Ephemeral: Dynamic environment creation.
  6. Integration Tests: E2E testing against the ephemeral env.
  7. GitOps Promotion: CI opens a PR to the infrastructure repo to update the version tag for production.

Advanced Concept: Implement “Progressive Delivery” using a Service Mesh (like Istio or Linkerd). Automate the traffic shift so that a new version receives only 1% of traffic initially. If error rates spike (measured by Prometheus), the automation automatically halts the rollout and reverts traffic to the stable version without human intervention.

Frequently Asked Questions (FAQ)

What is the difference between CI/CD and DevOps Automation?

CI/CD (Continuous Integration/Continuous Delivery) is a subset of DevOps Automation. CI/CD focuses specifically on the software release lifecycle. DevOps automation is broader, encompassing infrastructure provisioning, security policy enforcement, log management, database maintenance, and self-healing operational tasks.

How do I measure the ROI of DevOps Automation?

Focus on the DORA metrics: Deployment Frequency, Lead Time for Changes, Time to Restore Service, and Change Failure Rate. Automation should directly correlate with an increase in frequency and a decrease in lead time and failure rates.

Can you automate too much?

Yes. “Automating the mess” just makes the mess faster. Before applying automation, ensure your processes are optimized. Additionally, avoid automating tasks that require complex human judgment or are done so rarely that the engineering effort to automate exceeds the time saved (xkcd theory of automation).

Conclusion

Mastering DevOps automation requires a mindset shift from “maintaining servers” to “engineering platforms.” By leveraging GitOps for consistency, Policy as Code for security, and ephemeral environments for testing velocity, you build a system that is resilient, scalable, and efficient.

The ultimate goal of automation is to make the right way of doing things the easiest way. As you refine your pipelines, focus on observability and feedback loops—because an automated system that fails silently is worse than a manual one. Thank you for reading the DevopsRoles page!

Devops Tutorial

Exit mobile version