Joseph Velliah

Building a Rust gRPC AI Security Gateway for LLM Traffic

Sun, 29 Mar 2026 00:00:00 +0000

I wanted a small, honest implementation of the GenAI governance shape in code: a component on every LLM call that applies policy first, optionally scrubs prompts and responses, and emits metrics—without pretending to be enterprise inline inspection. This repo is a Rust gRPC MVP with keyword and rate limits, regex redaction, Prometheus counters, and pluggable providers (OpenAI, Anthropic, mock).

Industry collateral often uses the same vocabulary—visibility, inline policy, sensitive data in prompts and answers—for example Zscaler on securing generative AI and AI Guardrails. No affiliation with Zscaler; not an endorsement or a capability comparison.

Problem

When many clients talk straight to a provider API, you get recurring failure modes: no single choke point for policy, accidental or careless PII in prompts or model output, abuse and cost spikes, and weak signals for operators who need to know what was allowed, blocked, or altered.

A gateway in front of the provider gives you that choke point: enforce rules before the model runs, redact or block on the way in and out, and emit metrics so you are not flying blind.

This repo implements that as an MVP in Rust (Tokio, gRPC/tonic, pluggable backends): keyword blocklist, fixed-window rate limit, regex redaction—not a full DLP catalog or ML classifiers, but the same narrative line: inspect and govern the path, then call the model.

Why Rust and gRPC for this kind of gateway

The gateway sits inline: if that hop jitters, people stop trusting “govern every call.” Rust buys predictable latency in the enforcement path—no GC pauses while you scan and rewrite text—and memory safety while doing it. gRPC with Protobuf gives a versioned request/response contract (SecureCompletionRequest / SecureCompletionResponse), compact wire encoding, and generated server stubs so callers share one schema instead of ad hoc JSON that drifts quietly as fields change. The same surface extends cleanly to server streaming when you want token-by-token replies without a bespoke HTTP contract per client.

Architecture

The one-liner version:

Client → gRPC Gateway (Rust) → Policy Pipeline → Pluggable LLM Provider → Response + Metrics

The diagram below is inspired by the “at a glance” story common in GenAI security collateral (visibility, inline control, data-in-motion)—for example Zscaler’s Gen AI Security at-a-glance PDF—but redrawn for this open-source MVP only. It is not a depiction of Zscaler’s product or deployment model.

The three stacked stages echo the access · data · visibility framing used in GenAI security “at a glance” sheets: one inline choke point, with metrics as the separate HTTP scrape surface (port 8080), not an extra hop on the gRPC path.

Components:

Gateway: Tokio async server. gRPC on port 50051 for SecureCompletion, HTTP on 8080 for Prometheus /metrics only.
Policy before the model: Keyword blocklist and per-user rate limit run inside PolicyEngine before any LLM call.
Redaction: Regex-based scrubbing on the prompt and/or response in the gRPC handler when enabled (not part of the allow/block decision).
Providers: ChatCompletionProvider trait. Swap via LLM_PROVIDER env: OpenAI, Anthropic, or in-process mock (no HTTP; see README for ghz benchmarks).
Observability: Prometheus metrics (gateway_total_requests, gateway_blocked_requests, gateway_allowed_requests, gateway_provider_errors_total, gateway_request_latency_seconds, gateway_tokens_used_total).

Implementation

Async and gRPC

Tokio for async runtime. gRPC server uses tonic; HTTP uses axum.
Protobuf defines SecureCompletionRequest (user_id, prompt) and SecureCompletionResponse with SecureCompletionDecision enum (ALLOWED / BLOCKED), plus response text and reason. tonic-build compiles proto to Rust at build time.
Dual servers: gRPC and HTTP run on separate ports. HTTP serves Prometheus scrape only; chat flows through gRPC only.

Policies

Keywords: BANNED_KEYWORDS env (comma-separated). Case-insensitive match; blocks before LLM call.
Rate limit: In-memory fixed window per user_id (counter resets after the window elapses). Configurable via RATE_LIMIT_REQUESTS and RATE_LIMIT_WINDOW_SECS; RATE_LIMIT_MAX_TRACKED_USERS caps how many distinct IDs are tracked (eviction when full).
Redaction: Regex-based. Built-in patterns for email, API keys, SSN, credit cards, private IPs. Custom patterns via REDACT_CUSTOM_PATTERNS (JSON file path); each custom rule’s id must also appear in REDACT_PATTERNS to run. Runs on prompt (before LLM) and response (before client).

Pluggable Providers

Each provider implements ChatCompletionProvider. from_config() reads LLM_PROVIDER and instantiates OpenAI, Anthropic, or an in-process mock that returns a fixed string with no HTTP—useful for ghz runs that isolate gRPC, policy, and redaction from real LLM latency (see README Gateway-only benchmark).

Running It

export OPENAI_API_KEY=sk-...
cargo run

grpcurl -plaintext -import-path proto -proto ai_security_gateway.proto \
  -d '{"user_id":"user-1","prompt":"Say hello"}' \
  localhost:50051 ai_security.AiSecurityGateway/SecureCompletion

Docker (Dockerfile, docker-compose.yml) and Kubernetes manifests (k8s/) support local images and kind/minikube-style deploys. For cluster runs, the README covers loading the image, creating the API key Secret before pods start when using OpenAI or Anthropic (otherwise the container exits on OPENAI_API_KEY must be set), rollout restart after ConfigMap or Secret changes, and port-forward smoke tests with grpcurl and /metrics.

Gateway-only load check

With LLM_PROVIDER=mock and the ghz commands in the README, you can stress gRPC + policy + redaction without spending tokens. Latency and RPS depend on your machine and concurrency; turn keyword checks and redaction back on when you want those paths included. For long runs with a single user_id, raise RATE_LIMIT_REQUESTS and clear BANNED_KEYWORDS as the README describes so you are measuring the stack, not the default rate limit.

What worked

Separate ports for gRPC and HTTP: Prometheus /metrics on HTTP; chat only on gRPC. No gRPC-Web or transcoding in this MVP.
Keywords and rate limits before the LLM call. Redaction on allowed traffic mirrors the “sensitive data in prompts and answers” theme—bidirectional scrub, separate from the allow/block decision.
Trait-based providers: new backend = new type + from_config() branch.
In-memory rate limit is enough for one replica; multiple replicas need a shared store (e.g. Redis).

Next steps

Streaming: gRPC server-streaming for token-by-token responses.
Distributed rate limiting: Redis-backed for horizontal scaling.
More backends: Vertex AI, Azure OpenAI, Ollama.
Jailbreak / prompt-injection classifiers: Closer to the guardrails pages’ “inspect before harm” story than a static keyword list (still out of scope for this MVP).
Response caching: Cache by prompt hash to reduce LLM calls.

Repository: github.com/sprider/rust-grpc-ai-security-gateway

Claude Code Security: The Smart Way to Integrate AI

Sat, 21 Feb 2026 00:00:00 +0000

Anthropic just dropped Claude Code Security, and if you’re anywhere near AppSec or DevSecOps, you’ve probably already seen the debate lighting up on LinkedIn and Hacker News. The tool promises to scan entire repositories, reason about code the way a human researcher would, and even suggest patches your team can review before merging.

Here’s my take on how to actually use it—without throwing away everything you’ve already built.

Key Takeaways

Don’t replace your existing tools. Deterministic rules stay as the hard gate; AI sits on top as an advisory layer.
Use AI for triage, not truth. Claude excels at sorting findings by exploitability and risk—not at being the final word on what ships.
Deploy a “two-net” architecture. Your SAST and linters catch known-bad patterns; Claude catches the subtle stuff they miss.
Lock down variability. Pin model versions, log everything, and never let AI merge to protected branches on its own.

The Four-Pillar Framework

What is Claude Code Security?

If you haven’t seen the announcement yet, Claude Code Security is a new capability baked into Claude Code. It’s currently in limited research preview for Enterprise and Team customers, with broader availability expected later this year.

The short version: it scans your codebase for vulnerabilities and suggests fixes—but unlike traditional static analysis, it actually reasons about your code. It traces data flows across files, understands business logic, and catches issues that pattern-matching tools routinely miss. Anthropic claims Claude Opus 4.6 found over 500 vulnerabilities in production open-source projects that had gone undetected for years.

What caught my attention is the multi-stage verification. Every finding goes through an adversarial self-review before it reaches your dashboard, which (in theory) should cut down on the false positive noise that makes most SAST tools unbearable at scale.

But here’s the thing: reasoning-based detection is powerful, and it’s also non-deterministic. That’s not a flaw you can toggle off. It’s baked into how large language models work. So the question isn’t whether Claude Code Security is useful—it clearly is. The question is how you integrate it without losing the guarantees your compliance and governance teams depend on.

How Claude Differs from Traditional SAST

Keep Rules as the Baseline Gate

Most mature teams already run a stack of deterministic controls in CI/CD: linters, SAST scanners, secret detection, dependency checks, policy-as-code gates. These tools aren’t glamorous, but they give you something AI fundamentally cannot: predictable coverage.

Every rule executes on every build, in exactly the same way. You can reason about that behavior when you write policy. You can audit it. You can explain it to regulators.

And look, I know the pain points. A 2023 Ponemon study found that developers consider nearly half of all security alerts to be false positives, with the average engineer burning six hours a week just chasing down noise. Some SAST configurations hit false positive rates above 60-70%, depending on the language and ruleset. That’s brutal.

But turning off your deterministic tools in favor of an LLM doesn’t fix that problem—it trades one kind of uncertainty for another. The issue in most organizations isn’t “we can’t find vulnerabilities.” It’s “we can’t keep up with the ones we already find” and “we don’t know which ones actually matter.”

So the first principle here is non-negotiable: your existing static tools remain the hard gate in the pipeline. If a critical or high-severity rule fires, the build fails. Full stop. Claude can add signal, flag additional risks, even open blocking findings of its own—but it should never be able to override a deterministic rule that’s already failing. That’s how you preserve the guarantees your governance story is built on.

Use Claude Primarily for Triage

Where Claude Code Security genuinely moves the needle isn’t raw detection—it’s triage and explanation.

Anyone who’s run static analysis at scale knows exactly what I’m talking about. You roll out a new scanner, it dumps three thousand findings on your backlog, and within two weeks your developers have learned to ignore it entirely. Not because they don’t care about security. Because the signal-to-noise ratio is terrible and nothing in that wall of warnings tells them which issues are actually exploitable.

This is precisely the kind of problem large language models are good at.

Claude can look at a finding and answer questions like:

Is this actually exploitable given how data flows through this specific code path?
What’s the realistic blast radius if an attacker hits this?
How would I fix it in a way that fits this repository’s patterns and conventions?

Instead of handing your team a flat list sorted by severity label, you can pipe your SAST results into Claude and ask it to rank findings by real-world risk. The output isn’t just another label—it’s a narrative explanation and a patch suggestion that makes sense in context.

The key is that triage is advisory, not authoritative. You’re still enforcing your rules. But now you’re giving engineers a prioritized, annotated backlog instead of an undifferentiated wall of warnings. That cuts alert fatigue, shortens time-to-remediation, and honestly makes your legacy tools feel a lot less “legacy” because they’re plugging into a smarter workflow.

Use Claude as a “Second Net” for Coverage

Once your baseline and triage story are solid, you can start thinking about Claude as a second net—an additional layer that catches what your rules miss.

Traditional static tools are excellent at the patterns they were explicitly built to find: SQL injection sinks, missing output encoding, direct use of dangerous APIs, weak cryptographic primitives. They’re much less effective at anything that requires understanding business logic, tracing data across multiple files, or reasoning about authorization invariants. That’s where a model that can read and summarize code like a human starts to earn its keep.

Claude Code Security builds an internal model of how your application works—where data enters, how it transforms, what the code is trying to accomplish. In practice, that means it can surface vulnerabilities that never trip a regex or AST pattern. Things like:

An authorization check applied in most controllers but quietly bypassed in one edge-case endpoint
A multi-step workflow where an assumption about state can be violated if services execute out of order
A data path that’s harmless in default configuration but dangerous when a specific feature flag is enabled

Here’s how I’d wire this into a pipeline:

The asymmetry here is intentional. If Claude misses something your rules caught, the build still fails. If Claude finds something your rules missed, you’ve just upgraded your coverage. AI can only help you win more—it can’t redefine what “safe enough to ship” means on its own.

A reasonable policy might look like:

Finding Source	Severity	Action
Deterministic tool	Critical/High	Auto-block PR
Deterministic tool	Medium/Low	Create ticket, don’t gate
Claude only	Critical/High	Block after human confirms
Claude only	Medium/Low	Comment on PR, create ticket

That gives you a practical balance. You’re not ignoring AI insights, but you’re not handing over the keys either.

How Claude Compares to Other Tools

It’s worth understanding where Claude Code Security sits relative to the other options you’re probably already evaluating.

Capability	Claude Code Security	Snyk Code	Semgrep	GitHub Advanced Security
Detection approach	LLM reasoning + self-verification	AI + rules (DeepCode)	Pattern-based YAML rules	Semantic analysis (CodeQL)
Cross-file data flow	Strong	Moderate	Limited	Strong
Business logic flaws	Yes	Limited	No	Limited
False positive handling	Adversarial self-review	ML-based filtering	Rule tuning	Manual triage
Custom rules	Natural language prompts	Limited (Enterprise)	YAML (minutes to write)	QL queries (hours to learn)
Scan speed	Minutes (depends on repo size)	Fast	Very fast (~10 sec)	Slow (minutes to 30+ min)
Pricing*	Enterprise (custom)	$25/month per product (Team)	$40/month per contributor	$30/month per committer

*Pricing as of February 2026. Snyk Team plan requires minimum 5 developers; Enterprise is custom. GitHub unbundled GHAS in April 2025 into Code Security ($30) and Secret Protection ($19) per committer. See vendor sites for current pricing.

The honest assessment: Claude isn’t trying to replace your SAST tooling. It’s trying to do something those tools can’t—reason about code semantically and explain its findings in plain language. The tradeoff is non-determinism, which is why the two-net architecture makes sense. Use Semgrep or CodeQL for the predictable baseline, and use Claude for the intelligent layer on top.

Lock Down Variability Where It Matters

Everything I’ve described only works if you’re honest about how large language models behave. Even with temperature cranked down and prompts held constant, you won’t get identical output every time. That’s not a bug. It’s the nature of the technology.

So instead of pretending otherwise, deliberately lock down where that variability can affect outcomes.

At the configuration level:

Pin model versions and prompt templates where the platform allows—you want behavior to stay stable across builds
Define exactly which branches and events trigger AI scans (every PR for smaller services, nightly for monoliths)
Log all requests and responses so you can audit what the system did when it influenced a decision

At the process level:

AI can propose patches, open PRs, annotate findings, and request human review
AI cannot merge to protected branches or override mandatory controls
AI-suggested changes go through the same code review standards as any human commit

AI Permissions Boundary

And finally, treat this like any other production system. Threat-model its inputs—yes, including prompt injection risks from code comments and config files. Monitor its behavior over time. Build feedback loops for when it gets things wrong.

We’ve all seen examples of models confidently suggesting insecure patterns or ignoring instructions under the right (wrong?) conditions. Those stories aren’t reasons to avoid AI entirely. But they’re strong arguments for never putting it in sole control of your deployment gates.

Final Thoughts

The question in 2026 isn’t “should we use AI in application security?” The marginal cost of additional signal is low, and the upside for developer experience is significant. The real question is how we integrate it.

If you keep deterministic rules as your baseline gate, use Claude primarily for triage, deploy it as a second net for additional coverage, and deliberately constrain where its variability can influence outcomes—you get the best of both worlds. You keep the guarantees and auditability that security and compliance teams require, while giving your engineers a much more usable experience on top of the tools they already know.

That’s not about replacing “legacy” tooling. It’s about surrounding those tools with enough intelligence that they finally deliver on their original promise.

Ready to try it? Claude Code Security is currently in limited research preview for Anthropic Enterprise and Team customers. Access it through the Claude Code web interface, where you can scan repositories, review findings in the dashboard, and approve suggested patches—all within the tools you already use. Open-source maintainers can also apply for free, expedited access.

Have questions or want to share how you’re approaching AI in your security stack? Drop me a note—I’m always interested in hearing what’s working (and what isn’t) in production environments.

How I Built a Semantic Cache Using Only AWS Services

Sun, 25 Jan 2026 00:00:00 +0000

LLM calls are expensive and slow, but here’s the thing - users ask the same questions in different ways all the time. “What’s your refund policy?” and “How do I get my money back?” are different strings but the same question. Without semantic caching, you’re paying full price to answer identical questions over and over again.

I spent a weekend building a semantic cache that matches queries by meaning, not exact text - using only AWS-native services. S3 Vectors for similarity search, Bedrock for embeddings and LLM, Lambda for compute. Fully serverless, no external dependencies. The result? Cache hits that are 10x faster and cost nothing compared to calling the LLM.

The Problem

Every call to Amazon Bedrock costs money and takes 1-3 seconds. But here’s the thing - a huge chunk of queries are semantically identical to ones you’ve already answered. You’re literally paying to answer the same question over and over.

The Solution

Instead of matching exact strings, I used vector embeddings to match meaning. When a new query comes in:

Convert the query into a vector embedding (using Titan V2)
Search for similar queries in the cache (using S3 Vectors)
If similarity is above 85%, return the cached response
Otherwise, call the LLM and cache the result for next time

Simple concept. The trick was making it work with AWS-native services only.

The Tech Stack

Component	Service
Vector Storage	Amazon S3 Vectors
Embeddings	Bedrock Titan V2
LLM	Bedrock Claude Haiku 4.5
Compute	Lambda
API	API Gateway HTTP API

The best part? It’s fully serverless. No baseline costs. Pay only for what you use.

Results

After running some tests:

Cache hits are ~10x faster than calling the LLM
Semantic matching works - “capital of France” matches “France’s capital city”
Graceful degradation - if the cache fails, it falls back to the LLM

What I Learned

S3 Vectors is underrated - native similarity search without managing infrastructure
Serverless means fast startup - requests start processing in ~300ms
Similarity threshold matters - 0.85 worked well to avoid false matches while still catching rephrased questions

Try It Yourself

The complete code is available on GitHub. One-click deploy, one-click cleanup.

GitHub: github.com/sprider/semantic-cache-demo

The repo includes:

Full infrastructure as code (ready to deploy)
71 unit tests
One-click deploy and cleanup scripts
Architecture diagrams

Fair warning: it creates AWS resources that cost money. But the scripts make cleanup easy, and a few hours of testing costs less than a dollar.

What’s Next?

This is a demo, not production-ready code. For real use, you’d want:

API authentication
Cache invalidation strategy
Multi-region deployment
Better observability

Design alternatives

This demo uses S3 Vectors only for the cache layer. S3 Vectors has its own trade-offs (e.g. no built-in TTL, 40 KB metadata limit per vector). Combining S3 Vectors with DynamoDB—for example, storing vectors in S3 Vectors for similarity search and payloads or TTL in DynamoDB—lets you design differently for larger payloads, expiry, or exact-key lookups without changing the core flow shown here.

But as a proof of concept? It works. And it’s a pattern worth knowing.

Questions? Found a bug? Open an issue on the repo. Happy to chat about semantic caching, AWS architecture, or why vector databases are the future.

How to Build Better AI Agent Tools: Cut Costs by 70% (MCP Server Case Study)

Sat, 24 Jan 2026 00:00:00 +0000

Building tools for AI agents isn’t the same as building regular APIs. This guide shows you how to design tools that reduce token costs by 60-70% while improving accuracy. Whether you’re building Model Context Protocol (MCP) servers, LangChain tools, or custom agent functions—these principles apply.

Quick Take

I reduced my AI tool count from 30 to 8 (73% reduction) and cut token usage by 60-70% per response. This guide shows you how to:

Consolidate tools using action parameters
Optimize response formats to reduce costs
Write tool descriptions that AI agents understand
Avoid common pitfalls in AI tool design

The Problem: Why Multiple AI Tools Are Costing You Money

I thought I was being smart when I built 30 separate tools for my AI agent. Each tool did exactly one thing. Clean. Organized. Professional.

Then I got the token bill.

And watched my AI agent call the wrong tool 38% of test queries—calling three tools when it only needed one, requesting detailed responses when summaries would work, and burning through my budget.

Here’s what happened: I was building a Model Context Protocol (MCP) server for SharePoint integration, and I did what seemed logical: create one tool for every API endpoint. Need to get site info? That’s a tool. Need to list subsites? Another tool. Need to search? Yet another tool.

I ended up with 30 tools. It seemed organized on paper.

But when I tested it, the reality hit hard. The AI agent kept making mistakes. It would call the wrong tool, or call three tools when it only needed one. And the token costs? They were way higher than expected.

The Solution: Consolidating AI Tools for Better Performance

I took a step back and asked: “What are people actually trying to do?”

The key insight: Think about tasks, not API endpoints.

Instead of wrapping each API call in its own tool, I focused on what users were trying to accomplish. This single mindset shift changed everything.

I combined 30 tools into 8. That’s a 73% reduction. Here’s what it looked like:

Visual: The Transformation

Before

❌ get_site_info
❌ get_site_lists
❌ get_site_libraries
❌ get_site_pages
❌ search_sites
… (25 more tools)

After

✅ sharepoint_site (actions: get_info, list_subsites, search)
✅ sharepoint_list (actions: get_lists, get_items, create_item)
✅ sharepoint_files (actions: search, get_metadata, download)

Two Small Changes That Made a Big Difference

1. Action parameter - One tool can do multiple things

action: Literal["get_info", "list_subsites", "search"]

2. Response format parameter - Control how much detail you get back

response_format: Literal["concise", "detailed"] = "concise"

How Token Costs Impact AI Development Budget

Every API call your AI agent makes costs money. When your agent calls the wrong tool or requests more data than needed, those costs add up fast.

Tokens Are Expensive

Here’s a real example from my SharePoint server that made me rethink everything. When you ask for a site’s information, you can get back a lot of detail:

Detailed response (~280 tokens):

{
    "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#sites/$entity",
    "@microsoft.graph.tips": "Use $select to choose only the properties your app needs, as this can lead to performance improvements. For example: GET sites('<key>')/microsoft.graph.getByPath(path=<key>)?$select=displayName,error",
    "createdDateTime": "2025-04-12T16:40:22.963Z",
    "description": "A centralized repository for accessing country-specific HR policies and procedures across ACME Corporation's global operations.",
    "id": "spridermvp.sharepoint.com,506b7692-04ba-4be9-afc6-df146925948b,c7f4ceb0-f301-4280-8cc4-a8dba8560b64",
    "lastModifiedDateTime": "2026-01-24T13:42:56Z",
    "name": "acme-global-hr-policies-portal",
    "webUrl": "https://spridermvp.sharepoint.com/sites/acme-global-hr-policies-portal",
    "displayName": "ACME Global HR Policies Portal",
    "root": {},
    "siteCollection": {
        "hostname": "spridermvp.sharepoint.com"
    }
}

Concise response (~88 tokens):

{
    "description": "A centralized repository for accessing country-specific HR policies and procedures across ACME Corporation's global operations.",
    "lastModifiedDateTime": "2026-01-24T13:42:56Z",
    "name": "acme-global-hr-policies-portal",
    "webUrl": "https://spridermvp.sharepoint.com/sites/acme-global-hr-policies-portal"
}

Most of the time, you just need the name and URL. You don’t need all those IDs and timestamps. So I made concise the default. If the agent needs the technical details for a follow-up call, it can ask for detailed.

This approach can reduce token usage by 60-70% per response.

How Does the AI Know Which Action and Format to Use?

You might be wondering: “How does the AI agent pick the right action and response format?”

The Tool Description Pattern

The answer is in your tool description. The AI reads it like instructions. Here’s an example:

@mcp.tool()
def sharepoint_site(
    action: Literal["get_info", "list_subsites", "search"],
    site_url: str = None,
    query: str = None,
    response_format: Literal["concise", "detailed"] = "concise"
) -> str:
    """
    Work with SharePoint sites.
    
    Actions:
    - get_info: Get details about a specific site (requires site_url)
    - list_subsites: List all subsites under a parent site (requires site_url)
    - search: Find sites matching a query (requires query)
    
    Response formats:
    - concise: Returns only essential information (names, titles, URLs)
    - detailed: Returns full metadata including IDs for follow-up operations
    
    Use 'detailed' only when you need technical IDs for subsequent tool calls.
    """

Example Walkthrough: Finding a Marketing Site

When a user asks “Find the marketing site”, the AI:

Reads the tool description
Sees that search action requires a query
Picks action="search" and query="marketing"
Uses default response_format="concise" since it just needs to show results

Example Walkthrough: Fetching Documents

If the user then says “Get all the documents from that site”, the AI:

Remembers it needs the site ID for the next call
Goes back and calls the same tool with response_format="detailed"
Gets the technical IDs it needs
Uses those IDs in the next tool call

The Key Principle

💡 Key Insight: The AI isn’t magic—it’s following your instructions. The better you explain what each action does and when to use each format, the better it performs.

The Tradeoffs

Nothing is perfect. Here are the downsides I ran into:

1. More Complex Tool Descriptions

Before, each tool was simple: “Get site info.” Done.

Now, I have to explain multiple actions in one description. The tool description got longer. If you have 5-6 actions in one tool, it can get messy and the AI might get confused.

My rule: Keep it to 3-4 actions max per tool. If you need more, split it into two tools.

2. Harder to Debug

When something goes wrong, it’s trickier to figure out what happened. With 30 separate tools, if get_site_info failed, I knew exactly where to look.

Now, if sharepoint_site fails, I have to check: Which action was called? What parameters were passed? Was it a problem with the action logic or the parameter validation?

My solution: Add detailed logging for each action within the tool. Log the action name, parameters, and response format every time.

3. The AI Can Still Pick Wrong

Even with clear descriptions, the AI sometimes picks the wrong action or forgets to use detailed when it needs IDs for the next call.

This happens maybe 5-10% of the time. It’s better than the 38% error rate I had with 30 tools (where test queries resulted in wrong tool selection), but it’s not zero.

What helps:

Add examples in your tool description
Test with real user queries
Use clear parameter names (site_url not just url)

4. Not Every Tool Should Be Consolidated

Some tools are better left separate. If two operations are completely different and rarely used together, don’t force them into one tool just to reduce the count.

For example, I kept user_profile and user_search as separate tools. They serve different purposes and combining them would make the description confusing.

The test: Ask yourself: “Would a person naturally think these actions belong together?” If not, keep them separate.

Reflection point: Which of these tradeoffs concerns you most for your use case? The debugging complexity or the risk of AI confusion?

When This Approach Works Best

This works great when:

You have multiple tools that operate on the same resource (sites, files, users)
The actions are related and often used in sequence
You’re dealing with high token costs
Your users do varied tasks (not just one specific workflow)

This might not work if:

You have very specialized, single-purpose tools
Each tool has completely different parameters
You need extremely precise error handling for each operation
Your users only do one or two specific tasks

What I Learned: Key Principles for AI Tool Design

1. Think about tasks, not API endpoints (Most Important!)

Don’t just wrap your API. Think about what people are trying to accomplish. This is the most important principle that drives everything else.

❌ Three separate tools: list_users, list_events, create_event
✅ One tool: schedule_event (finds availability and creates the event)

2. Return information people can actually read

AI agents do better with names than with cryptic IDs.

❌ user_uuid: "e1b2c3d4-e5f6-7890"
✅ user_name: "Sarah Chen, Engineering Manager"

3. Use smart defaults

Start with concise responses
Add pagination (I limit responses to 25,000 tokens)
Let agents filter results to get exactly what they need

4. Write tool descriptions like you’re explaining to a coworker

The AI reads your tool description. Make it clear and helpful.

❌ “Searches SharePoint”
✅ “Search across SharePoint sites, documents, and lists. Use filters to narrow results. Returns top 10 matches by default.”

The Results

Based on the consolidation and MCP best practices:

Metric	Impact
Total Tools	73% reduction (30 → 8)
Token Efficiency	~70% fewer tokens per response
Agent Performance	Faster tool selection, fewer errors
Monthly Cost Savings	50-80% reduction (varies by query complexity)*

Note: These are projected savings based on tool consolidation and response format optimization. Actual results will vary depending on your specific use cases and query patterns.

How to Do This Yourself

Here’s a basic template you can use:

from enum import Enum
from typing import Literal

class ResponseFormat(Enum):
    DETAILED = "detailed"
    CONCISE = "concise"

@mcp.tool()
def my_action_tool(
    action: Literal["search", "get", "list"],
    query: str,
    response_format: ResponseFormat = ResponseFormat.CONCISE
) -> str:
    """
    Multi-purpose tool for [resource].
    
    Actions:
    - search: Find items matching query
    - get: Retrieve specific item details
    - list: Show all available items
    
    Use 'concise' for human-readable summaries.
    Use 'detailed' when you need IDs for follow-up calls.
    """
    
    result = perform_action(action, query)
    
    if response_format == ResponseFormat.CONCISE:
        return format_concise(result)
    else:
        return format_detailed(result)

Quick Summary

Before you dive in, here’s the roadmap:

Combine related tools using action parameters → reduces tool count and confusion
Add a response_format option (concise vs detailed) → cuts token usage by 60-70%
Default to concise to save tokens → agents request detailed only when needed
Return human-readable information, not just IDs → improves agent decision-making
Write clear tool descriptions → think of them as instructions for a coworker
Test with real tasks and measure results → validate your optimizations

Other Token Reduction Techniques

Beyond tool design, consider these approaches:

TOON Format – A JSON alternative designed for LLMs, reducing tokens by 30-60%
Prompt Caching – Cache repeated context for 75% cheaper tokens
Model Cascading – Use cheaper models for simple tasks, up to 90% savings
RAG – Retrieve only relevant context instead of full documents

Want to Learn More?

The official MCP documentation has a great guide on this topic: Writing Effective Tools for Agents

Take Action

Ready to optimize your AI tools?

Next Steps:

Audit your current tools - how many could be combined?
Identify which tools could benefit from response format options
Start with your highest-traffic tools for maximum impact
Measure token usage before and after

Questions or feedback? I’d love to hear about your optimization results or challenges you’re facing. What’s your tool count, and which optimization would help your use case most?

The Bottom Line

The key insight: Think about tasks, not API endpoints. This single principle drives everything else in AI tool design.

Building tools for AI isn’t the same as building regular APIs. I cut my tool count by 73% and this approach can reduce token usage by 60-70% per response, depending on the data complexity. The agent worked better, costs went down, and maintenance became simpler.

Sometimes less really is more.

Building a DevSecOps Pipeline on AWS (And You Can Too)

Sun, 04 Jan 2026 00:00:00 +0000

I have been working with CI/CD pipelines for a while now, and honestly, most of them just focus on getting code deployed fast. But what about security? That is usually an afterthought. So I decided to build something different—a platform where security checks happen automatically at every step.

Why I Did This

Look, pushing code fast is great until you realize you just deployed a vulnerability to production. I needed something that could:

Scan for security issues before deployment
Block builds that do not meet security standards
Keep an audit trail (because compliance audits are fun, right?)
Run without me babysitting it

What is Inside

I built this on AWS using EKS on Fargate. No EC2 instances to patch, which is nice. The whole thing runs on a custom VPC with multi-AZ setup for redundancy.

Here is how it works:

Every time code gets pushed, CodePipeline kicks off. The build stage runs security scans—SBOM generation (Syft), container vulnerability scanning (Trivy/Grype), SAST checks (Semgrep), secrets detection (detect-secrets), and OPA policy validation. If anything fails, the pipeline stops. No exceptions.

I intentionally picked open-source tools for the security gates. This keeps costs down and makes the whole setup reproducible without vendor lock-in. You can swap them out for commercial alternatives if you want, but these work great.

For access, I’m using Cognito for auth and WAF sits in front of the ALB to block sketchy traffic. CloudWatch alarms watch for anything weird—security events, performance drops, unexpected costs.

What I Learned

The automated scans actually caught stuff I missed. SBOM generation showed me I had some old dependencies with known CVEs that I did not even know were there.

Running on Fargate removed a lot of headaches. No patching EC2 instances, no worrying about the control plane. I just focus on securing my containers.

OPA policies are great once you write them. They enforce the same rules on every deployment without me having to remember anything.

Terraform makes this whole thing reproducible. I can destroy everything and rebuild it in 30 minutes flat. No clicking around in the console.

One thing to note: some verification steps need manual commands (like checking EKS addons or testing WAF rules). I kept these manual instead of fully automating them because they are useful for learning. You get to see exactly what is happening at each step. Once you are comfortable, you can script them if you want.

What It Costs

I tested this for a while then ran terraform destroy to clean up. While it was running, costs were around $200-300/month. That is mostly the EKS control plane, Fargate pods, ALB, and NAT gateways. Not cheap for a demo, but reasonable for a production workload with this much security built in.

Check Out the Code

I put everything on GitHub: https://github.com/sprider/aws-devsecops-demo

The repo has the full deployment guide, architecture diagrams, security configs, and screenshots from when I deployed it. I masked all the sensitive stuff so you can clone it and try it yourself.

Who is This For

This is not a perfect production-ready solution. There are things I would do differently for a real enterprise setup. But if you are trying to understand how to build a secure CI/CD pipeline or want a reference implementation to learn from, this is a solid starting point.

It is useful if you are:

Learning AWS security patterns
Building a reference pipeline for your team
Setting up security automation
Prepping for SOC2 or ISO 27001 audits
Understanding how security gates fit together

What You Could Add

If you want to extend this setup, here are some ideas worth exploring:

Multi-region setup for DR
GitOps with ArgoCD
GuardDuty integration
Spot instances to cut costs
Runtime security monitoring with Falco

Clone it, break it, improve it. That is how you learn.

AWS DevOps Agent: AI-Powered Incident Investigation in Seconds

Fri, 05 Dec 2025 00:00:00 +0000

Stop spending 30 minutes investigating incidents. Let AI do it in seconds. Here is a hands-on demo you can practice in 15 minutes.

The Problem

3 AM. Production is down. You are doing this:

Open CloudWatch → Check metrics
Open Datadog → Review traces
Open Splunk → Search logs
Check GitHub → Find recent deployments
Correlate everything manually → Find root cause

Time: 20-40 minutes of context switching and log correlation.

What if AI could do all of this in seconds?

The Solution: AWS DevOps Agent

Announced at AWS re:Invent 2025, AWS DevOps Agent is an AI service that automatically investigates incidents by:

Analyzing logs, metrics, and traces across multiple tools
Mapping infrastructure dependencies automatically
Recommending fixes to prevent future incidents
Integrating with your existing DevOps stack

Status: Public preview (us-east-1)

Free during preview

Who Should Use This?

Perfect For

On-call engineers who spend hours investigating incidents
SREs managing complex distributed systems
Platform teams running multi-account AWS environments
DevOps engineers correlating deployments with failures

Skip If

Simple applications with clear failure modes
Rarely experience incidents
Not heavily using AWS services

My Test: Real Results

I deployed a Lambda function with an intentional error and let the AI investigate.

Setup

Lambda function with division-by-zero error
CloudWatch alarm monitoring failures
3 error-generating invocations

Results

What the AI found in seconds:

The Lambda function contains intentional test code that throws ZeroDivisionError at line 9 in lambda_test.py with the literal expression ‘result = 1 / 0’. This is not a production bug but an expected test behavior.

What impressed me:

Context-aware: Understood it was test code, not a bug
Complete timeline: Linked deployment time to first error
Exact location: Found the error on line 9
Impact analysis: Calculated 100% failure rate
Fast: AI analysis in seconds + 4 minutes total

Before vs After

Task	Manual	AI Agent	Savings
Check metrics	2-3 min	Auto	100%
Review logs	3-5 min	Auto	100%
Check deployments	5-10 min	Auto	100%
Correlate timeline	5-10 min	Auto	100%
Root cause	5-10 min	sec	90%
Total	20-40 min	~4 min	80-90%

Three Core Features

1. AI Investigation

Auto-triggers from:

ServiceNow tickets
PagerDuty alerts
Datadog/Dynatrace/Splunk webhooks
Slack commands

What it analyzes:

CloudWatch metrics, logs, alarms
Third-party observability data
Deployment history from GitHub/GitLab
Infrastructure topology
Historical incident patterns

Delivers:

Root cause with reasoning
Event timeline
Blast radius analysis
Mitigation steps

2. Topology Discovery

Automatically maps your AWS infrastructure:

Resources across all accounts
Service dependencies
Links to source code
Deployment history

Use it to:

Understand blast radius during incidents
See cascading failure patterns
Assess change impact

3. Incident Prevention

After analyzing multiple incidents, the AI recommends:

Observability: “Add alarm for Lambda cold starts”
Testing: “Add load testing to pipeline”
Code: “Implement retry logic for API calls”
Infrastructure: “Enable Multi-AZ for RDS”

Integrations

Works with your existing tools:

Observability: CloudWatch • Datadog • Dynatrace • New Relic • Splunk

CI/CD: GitHub • GitLab

Ticketing: ServiceNow • PagerDuty

Chat: Slack

Kubernetes: Amazon EKS

Custom: MCP servers for proprietary tools

Try It: 15-Minute Demo

A hands-on demo using Terraform for infrastructure and manual Agent Space setup through the AWS Console.

Prerequisites

AWS account with admin access
AWS CLI v2 + Terraform installed
Region: us-east-1

Quick Start

1. Clone & Deploy Infrastructure

git clone https://github.com/sprider/aws-devops-agent-demo.git
cd aws-devops-agent-demo
chmod +x lambda-test.sh
./lambda-test.sh deploy

This automatically creates:

Lambda function with intentional error
CloudWatch alarm

2. Create Agent Space (Manual - AWS Console)

The Agent Space must be created through the AWS Console to ensure proper Primary source configuration.

Open the AWS DevOps Agent Console
Click “Begin setup” or “Create Agent Space”
Configure:
- Name: TestAgentSpace (or your preferred name)
- Description: Test Agent Space for Lambda error investigation demo
Click “Create”

3. Configure Cloud Capabilities (Primary Source)

After Agent Space creation, configure AWS account access:

In your Agent Space, go to “Settings” → “Cloud capabilities”
Click “Add cloud capability”
Select “AWS”
Choose “Primary source” (not Secondary)
Configuration:
- Account ID: Your AWS account (from terraform output aws_account_id)
- IAM Role: Use “Auto-create role” option
Click “Add”

Note: The IAM roles required for the DevOps Agent are automatically created by AWS when you select “Auto-create role” - you do not need to create them manually. The Primary source configuration ensures the agent can properly access CloudWatch alarms, Lambda logs, and other AWS resources needed for investigations.

4. Generate Lambda Errors

./lambda-test.sh test

5. Wait for Alarm to Trigger

After generating errors, wait 1-2 minutes for the CloudWatch alarm to evaluate and enter ALARM state:

./lambda-test.sh status

Wait until you see AlarmState: ALARM before proceeding to the next step.

6. Start Investigation

In the AWS DevOps Agent Console, click on your Agent Space name (e.g., “TestAgentSpace”)
Click the “Incident Response” tab
In the “Start an investigation” text box, type: Lambda function throwing errors
Click “Start investigation” button
A modal will appear - fill in the investigation details:
- Investigation details: Keep “Lambda function throwing errors”
- Investigation starting point: CloudWatch alarm AWS-AIDevOps-Lambda-Error-Test
- Date and time of incident: Get current time with date -u +"%Y-%m-%dT%H:%M:%SZ"
Click “Start investigating…“

7. Watch AI Work

Watch the investigation in real-time. The AI will:

Detect the alarm
Pull Lambda logs
Identify ZeroDivisionError
Correlate deployment time
Provide root cause

Investigation time: In seconds

8. Cleanup Everything

./lambda-test.sh destroy

Important: Manually delete the Agent Space and auto-created resources from the AWS Console before destroying infrastructure.

Delete Agent Space:
- Go to AWS DevOps Agent Console
- Select your Agent Space
- Click “Actions” → “Delete Agent Space”
- Confirm deletion
- Note: This automatically removes the IAM roles created by the Agent Space
Delete Lambda Log Group:
- Go to CloudWatch Console → Log groups
- Find /aws/lambda/AWS-AIDevOps-test-lambda
- Select it and click “Actions” → “Delete log group(s)”
- Confirm deletion
Verify IAM Roles Cleanup (Optional):
- Go to IAM Console → Roles
- Search for roles created by the Agent Space (they usually have “DevOpsAgent” or “AIDevOps” in the name)
- These should be automatically deleted when the Agent Space is deleted
- If any remain, manually delete them
Then run: ./lambda-test.sh destroy

All Available Commands

./lambda-test.sh deploy    # Deploy Lambda and CloudWatch alarm
./lambda-test.sh test      # Generate Lambda errors (invoke 3 times)
./lambda-test.sh status    # Check CloudWatch alarm status
./lambda-test.sh logs      # View Lambda function logs
./lambda-test.sh destroy   # Destroy all infrastructure

Cost

$0.00 - Everything covered by AWS Free Tier

Troubleshooting

Issue: “AWS account is not accessible” or “Monitor Association not found”

Error message in investigation:

Unable to investigate the Lambda function errors because AWS account XXX
is not accessible. The error 'Monitor Association with AgentSpace agentSpaceId
XXX not found' indicates this account is not associated with the monitoring system.

Root cause: Your AWS account is not configured as a Primary source in Cloud Capabilities.

Solution:

Open your Agent Space in AWS Console
Go to Settings → Cloud capabilities
Check if your AWS account is listed under “Primary sources”
If not listed or listed under “Secondary sources”:
- Click “Add cloud capability”
- Select “AWS”
- CRITICAL: Choose “Primary source” (NOT Secondary)
- Enter your AWS account ID (from terraform output aws_account_id)
- Use “Auto-create role” option
- Click “Add”
Verify your account now appears under “Primary sources”
Try the investigation again

Why this matters: Only Primary sources give the AI agent full access to CloudWatch alarms, Lambda logs, and other AWS resources needed for investigations.

Key Facts

What It Is

AI layer that connects your existing tools
Not a monitoring tool replacement
Reduces investigation time by 80-90%

Limitations (Preview)

Region: us-east-1 only
Quotas: 20 investigation hours/month, 10 prevention hours/month
Pricing: Free now, pricing TBD at GA

Security

Read-only permissions by default
IAM-based access control
Agent Space isolation
AWS IAM Identity Center support

Common Questions

Q: Does it replace my observability tools? A: No. It sits on top of them, connecting data across tools.

Q: What if the AI is wrong? A: You are in control. Ask follow-up questions, steer investigations, or escalate to AWS Support.

Q: How secure is it? A: Very. Read-only by default, IAM-controlled, data stays in your account.

Q: Works with non-AWS tools? A: Yes. Integrates with Datadog, Dynatrace, New Relic, Splunk, GitHub, GitLab, ServiceNow, Slack.

Next Steps

After testing:

Connect production - Create Agent Space for real environment
Enable auto-triggers - Set up ServiceNow/PagerDuty webhooks
Review recommendations - Implement prevention suggestions
Expand scope - Connect multiple AWS accounts

Files in This Repo

aws-devops-agent-demo/
├── README.md                 # This guide
├── lambda-test.tf            # Terraform: Lambda and CloudWatch alarm
├── lambda_test.py            # Test Lambda function (division by zero)
├── lambda-test.sh            # Automation script for deployment
├── .gitignore                # Git ignore file
└── screenshots/              # Step-by-step screenshots of the demo
    ├── 01-terraform-deploy.png
    ├── 02-terraform-output.png
    ├── 03-devops-agent-console.png
    ├── 04-create-agent-space.png
    ├── 05-cloud-capabilities.png
    ├── 06-lambda-errors-generated.png
    ├── 07-cloudwatch-alarm-triggered.png
    ├── 08-incident-response-dashboard.png
    ├── 10-investigation-details-modal.png
    ├── 11-investigation-in-progress.png
    ├── 12-investigation-completed.png
    ├── 13-investigation-summary.png
    ├── 14-mitigation-plan.png
    └── 15-terraform-destroy.png

What is Automated vs Manual?

Automated via Terraform:

Lambda function with intentional error
CloudWatch alarm monitoring

Manual via AWS Console:

Agent Space creation
Cloud Capabilities configuration (Primary source setup + IAM role auto-creation)
Agent Space deletion (which automatically removes auto-created IAM roles)

Why Manual? The Agent Space requires Primary source configuration through the console to ensure the AI agent can properly access AWS resources during investigations. The AWS CLI cannot currently configure this correctly. When you delete the Agent Space, AWS automatically cleans up the auto-created IAM roles.

About This Article This article and accompanying automation scripts were developed with assistance from Claude Code(Anthropic). All code has been tested in my personal AWS environment and verified against the official AWS DevOps Agent User Guide.

Resources

DynamoDB Just Made Your Life Easier: Multi-Attribute Composite Keys Explained

Tue, 25 Nov 2025 00:00:00 +0000

AWS just dropped a feature on November 19, 2025 that is going to save you from one of DynamoDB’s most annoying workarounds: multi-attribute composite keys for Global Secondary Indexes (GSIs). Let me show you why this matters with a real-world example.

Important: This is a GSI Feature

Critical Clarification: This new capability applies to Global Secondary Indexes (GSIs) only - NOT to your base table’s primary key. Your base table still uses the traditional structure of a single partition key + optional single sort key. However, when you create GSIs on your table, you can now use up to 4 partition key attributes and 4 sort key attributes!

The Scenario: E-Commerce Order Tracking

Imagine you are building an order management system. You need to query orders by:

Customer ID + Order Date + Order Status

Seems simple, right? Wrong. Until now, this was a pain.

The Old Way (aka The Painful Way)

Before this update, you had two bad options:

Option 1: Concatenate Fields (Yuck!)

GSI Configuration:
Partition Key: customer_id
Sort Key: date_status (concatenated: "2024-11-24_SHIPPED")

This meant:

Extra fields cluttering your table
String manipulation everywhere in your code
Maintenance nightmares when requirements change
Code that looks like sortKey = ${date}_${status}``

Option 2: Full Table Scan (Nope!)

Just scan the entire table filtering by all three fields. Slow, expensive, and scales terribly.

The New Way (Hello, Beautiful!)

Now you can do this with your GSI:

Global Secondary Index Configuration:
  Partition Key: customer_id
  Sort Key 1: order_date
  Sort Key 2: order_status

Up to 4 partition key fields and 4 sort key fields per GSI!

What This Means for You

// Before: String concatenation madness
const sortKey = `${orderDate}_${status}`;

// After: Clean, intuitive queries on your GSI
queryParams = {
  IndexName: 'CustomerOrdersIndex',
  partitionKey: customerId,
  sortKey1: orderDate,
  sortKey2: status
}

Real Benefits:

Cleaner Code: No more string concatenation hacks
Better Performance: Query (not scan) with multiple attributes on GSIs
Easier Maintenance: Add/remove query patterns without refactoring
Native Support: Let DynamoDB handle the complexity
Native Data Types: Keep numbers as numbers, dates as dates - no string conversion needed

Quick Example

Let us say you need to find all orders for customer “C123” placed on “2024-11-24” with status “PENDING”:

Before:

# Had to concatenate
sort_key = f"2024-11-24_PENDING"
response = table.query(
    IndexName='CustomerOrdersIndex',
    KeyConditionExpression='customer_id = :cid AND date_status = :ds',
    ExpressionAttributeValues={
        ':cid': 'C123',
        ':ds': sort_key
    }
)

After:

# Clean and intuitive with multi-attribute GSI
response = table.query(
    IndexName='CustomerOrdersIndex',
    KeyConditionExpression='customer_id = :cid AND order_date = :date AND order_status = :status',
    ExpressionAttributeValues={
        ':cid': 'C123',
        ':date': '2024-11-24',
        ':status': 'PENDING'
    }
)

Pro Tips

Do Not Go Crazy: Just because you CAN add 8 fields does not mean you SHOULD. GSIs consume additional storage and throughput.
Watch Your Capacity: Each GSI needs its own read/write capacity units. Plan accordingly.
Eventually Consistent: Remember, GSIs are eventually consistent. The more fields, the longer it might take to sync.
Query Pattern Rules:
- All partition key attributes must use equality (=) conditions
- Range conditions (<, >, BETWEEN) only work on the last sort key attribute
- You can not skip sort keys - use them left-to-right (SK1, or SK1+SK2, or SK1+SK2+SK3)
Base Table vs GSI: Your base table’s primary key structure has not changed - this feature is exclusively for GSIs to give you more flexible query patterns!

The Bottom Line

This is one of those updates that makes you wonder, “How did we live without this?” If you have been dealing with concatenated fields or complex workarounds in your GSIs, it is time to refactor and simplify.

Your future self (and your teammates) will thank you.

Ready to try it? Head to your DynamoDB console and create a GSI with multiple attributes. It is available now in all AWS regions at no additional cost beyond standard GSI pricing!

How Putting CDN After Your Reverse Proxy Creates a Single Point of Failure

Sat, 08 Nov 2025 00:00:00 +0000

This is a universal architecture anti-pattern that affects teams across all cloud providers and technology stacks. Whether you are using nginx, HAProxy, Envoy or cloud load balancers, the problem is the same: placing a CDN after your reverse proxy instead of before it defeats the CDN’s distributed architecture.

Let us understand why this happens and why it is dangerous, regardless of your technology choices.

Why This Architecture Fails: The Fundamental Problem

Your reverse proxy operates from a limited IP address space : Even with auto-scaling, multi-zone deployment and load balancing, your reverse proxy cluster runs on a finite set of IP addresses within your data center or cloud VPC. These IPs are geographically concentrated.
```
  # Example: 
  nginx cluster with 10 nodes nginx-1: 10.0.1.5 nginx-2: 10.0.1.8 nginx-3: 10.0.2.12 ... nginx-10: 10.0.3.45 
    
  All IPs from the same /16 or /24 subnet 
  All IPs in the same geographic region 
  Even with 100 nodes, still limited IP space!
```
CDN sees your entire proxy cluster as a single client location: CDNs route traffic based on source IP geolocation. When all requests originate from your proxy’s data center (even from multiple IPs), the CDN’s routing algorithm treats this as one geographic location requesting content.
```
  # Normal (user-facing CDN)
  User in Tokyo → CDN routes to Tokyo PoP
    
  # This anti-pattern 
  All traffic from proxy IPs (e.g. US-East) → CDN routes ALL to US-East PoP
```

100% of your traffic flows through ONE CDN Point of Presence: Instead of distributing globally across hundreds of edge locations, all your traffic is routed to the single PoP nearest to your reverse proxy cluster.

  # CORRECT: User-facing CDN 
  Origin User in Tokyo → Tokyo PoP  
  Origin User in London → London PoP 
  Origin User in New York → Virginia PoP 
  Origin User in Sydney → Sydney PoP 
    
  Origin Result: Distributed across 4 PoPs ✅ 

  # WRONG: CDN behind reverse proxy  
  All Users → Reverse Proxy (US-East IPs) → Virginia PoP ONLY 
    
  Backend Result: Single PoP = Single Point of Failure ❌

When that one PoP fails → 100% of users are impacted: Your carefully architected multi-zone reverse proxy becomes irrelevant. If the single CDN PoP experiences:
- Network congestion
- Hardware failure
- Software bug
- DDoS attack
- Maintenance downtime
ALL your users worldwide experience an outage

Why Teams Build This (By Accident)

Organic evolution: Started with Users → Proxy → Backend, then added “caching” without understanding CDN routing
Wrong mental model: Treating CDN as a cache layer (like Redis) instead of an edge network
Legacy migration: Lifted-and-shifted on-prem architecture without redesign
Quick fix that stuck: “Backend is slow, let us add caching here!” without proper architecture review

The Correct AWS Architectures

Now that we understand the problem, let us look at three correct ways to implement caching and content delivery in AWS. Each pattern solves specific use cases and eliminates the single point of failure.

ElastiCache for Internal Caching

Move caching inside your VPC using ElastiCache (Redis or Memcached). This provides distributed caching with true multi-AZ high availability, without any external routing layer to become a bottleneck.

Why This Works:

No external routing layer to become a bottleneck
Sub-millisecond latency within VPC
True multi-AZ high availability with automatic failover
Fine-grained cache control in application code
ElastiCache Cluster Mode for horizontal scaling

CloudFront in Front of ALB (User-Facing)

Place CloudFront where it belongs: directly facing users. This restores the CDN’s global distribution, provides DDoS protection, and delivers edge caching benefits without creating routing bottlenecks.

Why This Works:

Users route to nearest CloudFront edge → low latency for all
True global distribution across hundreds of locations
Built-in DDoS protection with AWS Shield Standard
SSL/TLS termination at edge reduces origin load
No single point of failure in routing layer
Cache static AND dynamic content at edge

Lambda@Edge for Edge Computing

Execute routing, A/B testing and composition logic at CloudFront edge locations using Lambda@Edge. This can eliminate the reverse proxy layer entirely for certain workloads.

Why This Works:

Logic executes at hundreds of edge locations (ultra-low latency)
No centralized reverse proxy to become a bottleneck
Dynamic routing, A/B testing, auth at edge
Can eliminate ALB costs for some workloads
Geo-based content delivery
Request/response manipulation at edge

Quick Decision Guide

Choose ElastiCache when

✅ You need internal caching within your VPC
✅ Sub-millisecond latency is critical
✅ You want full control over cache logic
✅ Session management across microservices
✅ Database query result caching
❌ Do not need global distribution

Choose CloudFront + ALB when

✅ You have a global user base
✅ You need DDoS protection
✅ You serve static or semi-static content
✅ SSL/TLS termination at edge is desired
✅ Cost-effective solution needed
❌ Do not need complex edge logic

Choose Lambda@Edge when

✅ You need complex logic at the edge
✅ A/B testing or personalization required
✅ Geographic content routing needed
✅ Authentication/authorization at edge
✅ You want to eliminate ALB for some workloads
❌ Do not mind higher complexity

Common Objection: What About Multi-Region Proxy Clusters?

A common question arises: “If I deploy my proxy clusters in multiple regions (US-East, EU-West, AP-Southeast), doesn’t that solve the single point of failure problem?”

Short answer: It reduces the blast radius but does not eliminate the anti-pattern.

What Multi-Region Proxies Give You

# Multi-region proxy setup
US Users → US Proxy Cluster → CDN Virginia PoP → Backend
EU Users → EU Proxy Cluster → CDN London PoP → Backend
Asia Users → Asia Proxy Cluster → CDN Singapore PoP → Backend

At first glance, this seems better—you are no longer routing all traffic through a single CDN PoP. Each regional proxy cluster routes to its nearest CDN location.

What Still Remains Wrong

Traffic still flows through your infrastructure first
- Users must reach your proxy before getting any CDN benefits
- Adds unnecessary latency: User → Your Proxy → CDN → Backend
- The CDN cannot optimize routing based on actual user location
You are duplicating what the CDN already does natively
- CDNs have hundreds of edge locations with intelligent routing
- You are building a 3-region routing layer when CDN offers hundreds of locations
- Your proxies become an expensive, manual version of CDN anycast

CDN routing is based on proxy location, not user location

# The Problem
User in Mumbai → Routes to Asia Proxy (Singapore) → CDN Singapore PoP
# CDN cannot optimize: Maybe Tokyo PoP would be faster for this user

# Correct Approach
User in Mumbai → CloudFront routes to closest PoP (Mumbai/Chennai) → Origin
# CDN intelligently selects from hundreds of locations

Cost and operational complexity
- Running multi-region proxy infrastructure is expensive
- Manual failover configuration between regions
- More moving parts = more failure modes

The Correct Multi-Region Approach

Instead of multi-region proxies, use CloudFront with native multi-region support:

CloudFront + Origin Groups (Automatic Failover):

User Anywhere → CloudFront (automatic routing to nearest PoP)
              → Primary Origin (US-East ALB)
              → Secondary Origin (EU-West ALB) [automatic failover]

Benefits:

CloudFront handles global routing automatically
Origin Groups provide automatic failover between regions
No proxy infrastructure to manage
Users always route to their nearest edge location
Lower latency, lower cost, higher availability

The fundamental issue is not single vs. multiple regions—it is placing the CDN after your infrastructure instead of in front of it. Multi-region proxies add cost and complexity while still defeating the CDN’s core purpose.

Key Takeaways

The anti-pattern is universal: Placing a CDN after your reverse proxy (whether it is nginx, HAProxy, ALB or API Gateway) defeats the CDN’s distributed architecture and creates a hidden single point of failure.

In AWS specifically: CloudFront must be user-facing to work correctly. Choose ElastiCache for internal caching needs, CloudFront in front of ALB for global content delivery or Lambda@Edge for edge computing.

The fix is architectural: This is not about tweaking configurations—it is about placing components in the right order. CDNs belong between users and your infrastructure, not between your infrastructure components.

The Game-Changer: Docker MCP Catalog and Toolkit

Sat, 18 Oct 2025 00:00:00 +0000

I will be honest – setting up MCP (Model Context Protocol) servers for AI agents has been a pain. You would spend time digging through documentation, manually editing JSON config files, and hoping you got the syntax right. Docker changed that with their MCP Toolkit, and it is pretty clever.

What Is MCP and Why Should You Care?

If you are unfamiliar, MCP is Anthropic’s protocol that lets AI agents (like Claude) interact with external services. Think of it as a standardized way for your AI to read Slack messages, create GitHub issues, fetch YouTube transcripts, or query databases. Before this, each integration required its own setup dance.

The problem was not the concept – it was the configuration overhead. Every MCP server required manual JSON editing in your Claude config, and if you wanted to try multiple servers, well… you would be editing that file a lot.

What Docker Actually Built

Docker Desktop now has a built-in MCP Toolkit that fundamentally changes how this works. Here is what makes it different:

The Catalog Interface

Instead of hunting down MCP servers on GitHub and figuring out how to configure them, you get a curated catalog in Docker Desktop. It shows you what is popular, what each server does, and you can add them with a single click.

The catalog handles the JSON configuration automatically. Adding a server updates the config file for any connected clients – Claude Desktop, Cursor, whatever you are using—no more manual editing.

How the Containers Work

This is the part that makes sense from a Docker perspective. Each MCP server runs in its own container, but Docker’s implementation is more innovative than you might expect:

Containers only spin up when a tool is actually called
They shut down automatically when the task completes
When idle, they consume zero memory
You get the isolation benefits of containers without the overhead of running everything 24/7

So if you have 10 MCP servers configured but only use one of them, you are only running one container now. It is efficient.

Authentication That Does Not Suck

Here is something that usually takes forever: OAuth flows. The catalog has built-in OAuth support for services like GitHub. You click to authenticate, and it handles the token dance. You are done. For API-key-based services, there is a straightforward interface to add your credentials.

Compare that to manually managing environment variables or config files. Yeah, this is better.

What’s Actually Available

The catalog has the servers you would expect if you have been following the MCP ecosystem:

Core Productivity Tools

YouTube – grab transcripts, summarize videos
Slack – read channels, post messages (helpful for monitoring or notifications)
GitHub – create issues, read repos, manage PRs
Notion, Obsidian – knowledge base integration

Development Tools

Database connectors (PostgreSQL, SQLite, etc.)
File system access
Memory/cache systems like ChromaDB
Fetch (for web scraping and HTTP requests)

There is also Gordon, Docker’s built-in test agent. It is in beta, but it is useful for quickly checking if an MCP server is working before you try using it with your actual workflow.

A Real Workflow Example

Let me give you a practical example of why this matters. Say you are researching a technical topic:

Use the YouTube MCP server to pull transcripts from conference talks
Have Claude summarize the key points

Before the catalog, setting up those five MCP servers meant editing JSON configs, debugging path issues, and restarting things a few times. Now it is 10 minutes of clicking through the catalog.

Setting It Up

The actual setup is straightforward:

Make sure you have Docker Desktop installed
Enable beta features in Settings → Beta features → Enable Docker MCP Toolkit
Open the MCP Catalog from the Docker Desktop’s MCP Toolkit section
Browse servers and click “Add MCP Server” for what you need
Configure any API keys or OAuth in the configuration section of the MCP server
In your MCP Tpplkit –> Clients section, click to connect to MCP clients
Restart your MCP clients

The whole thing takes less time than it took me to write this section.

For Custom Integrations

If you are building your own agents or using frameworks like N8N or Python-based systems, Docker has open-sourced the MCP Gateway. It lets you orchestrate MCP servers through HTTP with streamable protocol support.

Bottom Line

The Docker MCP Toolkit is worth checking out if you use AI agents for anything beyond basic chat. It automates the tedious parts of MCP server management while providing the control and isolation that containers provide.

The fact that it is built into Docker Desktop means there is one less tool to install, one less service to manage, and one less thing to forget when switching between projects.

Give it a try. Set up a few servers, test them with Gordon/Claude, and see if it fits your workflow. Worst case, you waste 15 minutes. Best case, you never manually edit an MCP config file again.

Kafka Crash Course: Learn with a Parent's Return to Office Mandate Use Case

Sat, 13 Sep 2025 00:00:00 +0000

Companies are mandating return-to-office. Parents now face a coordination challenge:

School bus drops kids at 3:15 PM at the community bus stop
Parents need to be there, but meetings run over
Group chats don’t work - messages get buried, no confirmation

Real scenario:

3:10 PM - Sarah's meeting runs over
3:11 PM - Posts in group chat: "Can someone watch Jake?"
3:15 PM - Bus arrives, no response yet

Neighbors want to help. They just need a reliable system.

Why Kafka Fits This Use Case

Before: Tightly Coupled Services

Parent App → Notification Service → Database → Neighbor App

Problems:

Notification service crashes = everything stops
Parent waits for entire chain to respond
Neighbor offline = message lost forever

With Kafka: Decoupled

Parent App → Kafka ← Neighbor Apps

Benefits:

Parent sends alert, doesn’t wait
Message stored safely in Kafka
Neighbors read when ready (even if offline before)
Multiple neighbors can all see it
Add new features without breaking existing ones

Think of Kafka as a bulletin board. Pin a message, walk away. Everyone sees it. First person to help responds.

Let’s Build It

What we need:

Docker (to run Kafka)
Python (to write producer/consumer)

Virtual Environment Setup

Create the project folder and navigate to it:

mkdir bus-stop-kafka
cd bus-stop-kafka

Create a virtual environment:

python3 -m venv venv

Activate the virtual environment:

source venv/bin/activate

Install librdkafka (required C library for macOS):

brew install librdkafka

Upgrade pip and install dependencies:

pip install --upgrade pip
pip install confluent-kafka

The virtual environment is now set up and isolated from your system Python installation.

Start Kafka

Create docker-compose.yml:

version: '3.8'
services:
  kafka:
    image: confluentinc/cp-kafka:7.8.3
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_KRAFT_MODE: "true"
      CLUSTER_ID: "bus-stop-demo"
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: "/var/lib/kafka/data"
    volumes:
      - kafka-data:/var/lib/kafka/data

volumes:
  kafka-data:

Start it:

docker-compose up -d
sleep 30  # Wait for Kafka to start

Producer (Sarah Sends Alert)

Create producer.py:

from confluent_kafka import Producer
import json

# Connect to Kafka
producer = Producer({'bootstrap.servers': 'localhost:9092'})

# Create alert message
alert = {
    'parent_name': 'Sarah',
    'child_name': 'Jake',
    'location': 'Oak Street Bus Stop',
    'message': 'Meeting ran over, will be 10 mins late'
}

# Send to Kafka topic
producer.produce(
    topic='bus-stop-alerts',           # Topic name
    value=json.dumps(alert).encode()   # Convert to bytes
)

producer.flush()  # Ensure it's sent

print(f"✅ Alert sent: {alert['parent_name']} needs help")

Run it (ensure your virtual environment is activated):

python producer.py
# Output: ✅ Alert sent: Sarah needs help

What happened:

Connected to Kafka at localhost:9092
Created JSON message with alert details
Sent to topic called bus-stop-alerts
Kafka stored it

Consumer (Mike Receives Alert)

Create consumer.py:

from confluent_kafka import Consumer
import json

# Connect to Kafka
consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'neighbors',              # Consumer group
    'auto.offset.reset': 'earliest'       # Read from beginning
})

consumer.subscribe(['bus-stop-alerts'])
print("🔔 Listening for alerts...\n")

try:
    while True:
        msg = consumer.poll(1.0)  # Check every second
        
        if msg is None:
            continue
        
        if msg.error():
            print(f"Error: {msg.error()}")
            continue
        
        # Got a message!
        alert = json.loads(msg.value().decode())
        
        print(f"🚨 {alert['parent_name']} needs help!")
        print(f"   Child: {alert['child_name']}")
        print(f"   Location: {alert['location']}")
        print(f"   Message: {alert['message']}\n")

except KeyboardInterrupt:
    print("Stopped")
finally:
    consumer.close()

Run it (ensure your virtual environment is activated):

python consumer.py

Output:

🔔 Listening for alerts...

🚨 Sarah needs help!
   Child: Jake
   Location: Oak Street Bus Stop
   Message: Meeting ran over, will be 10 mins late

What happened:

Consumer connected to Kafka
Subscribed to bus-stop-alerts topic
Read the message Sarah sent
Keeps running, waiting for more

Understanding Kafka Concepts

Topics

Like folders for messages
We used: bus-stop-alerts
Organizes different types of messages

Producers

Send messages to topics
Don’t wait for consumers
Don’t know who will read it

Consumers

Read messages from topics
Can start from beginning or latest
Keep polling for new messages

Consumer Groups

Multiple consumers with same group.id
Kafka distributes messages among them
Load balancing automatically

Try This: Messages Persist

Shows: Messages don’t disappear

Start consumer, then stop it (Ctrl+C)

Send 3 alerts:

python producer.py
python producer.py
python producer.py

Start consumer again

Result: Consumer shows all 3 alerts!

Why this matters: If Mike’s phone was off when Sarah sent alert, he still sees it when phone turns back on.

Important: Consumer Offset Tracking

Question: “If I sent 1 alert earlier and 3 alerts now, why don’t I see all 4 alerts?”

Answer: Kafka tracks where each consumer group left off reading using offsets.

Here’s what happens:

First run: producer.py sends alert #1
First run: consumer.py reads alert #1, Kafka marks “neighbors group read up to offset 0”
Second run: producer.py sends alerts #2, #3, #4
Second run: consumer.py only shows #2, #3, #4 (skips #1 because it was already read)

This is a feature, not a bug! Imagine if neighbors saw every alert from the past month every time they checked.

To see ALL messages from the beginning:

Option 1 - Change consumer group name (line 197 in consumer.py):

'group.id': 'neighbors-v2',  # New group = starts fresh

Option 2 - Delete the consumer group offset tracking:

docker exec -it kafka kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --delete --group neighbors

Try This: Multiple Neighbors

Shows: Multiple consumers share work

Open 3 terminals
Run python consumer.py in each (with venv activated)
Send alerts from 4th terminal

Result: Each consumer gets different messages (load balancing)

Why this matters: Multiple neighbors at bus stop, all see alerts, first one responds.

Important: Partitions Enable Load Balancing

Question: “All messages go to one consumer. Is load balancing actually working?”

Answer: With the default setup (1 partition), load balancing cannot work. Here’s why:

The Partition Rule:

Maximum parallel consumers = Number of partitions

By default, bus-stop-alerts has 1 partition, so:

Consumer #1 gets partition 0 (receives all messages)
Consumer #2 gets nothing (no partitions left)
Consumer #3 gets nothing

To see actual load balancing:

Delete the topic:

docker exec -it kafka kafka-topics \
  --bootstrap-server localhost:9092 \
  --delete --topic bus-stop-alerts

Recreate with 3 partitions:

docker exec -it kafka kafka-topics \
  --bootstrap-server localhost:9092 \
  --create --topic bus-stop-alerts \
  --partitions 3 \
  --replication-factor 1

Run 3 consumers in separate terminals:

python consumer.py  # Terminal 1
python consumer.py  # Terminal 2
python consumer.py  # Terminal 3

Send multiple alerts:

python producer.py  # Run this 6+ times

Now you’ll see: Messages distributed across all 3 consumers!

Key insight: More partitions = more parallelism. This is how Kafka scales to handle massive throughput.

The Power of Kafka

Real-World Flow

Sarah (3:10 PM)
  ↓ sends alert
Kafka (stores it)
  ↓ notifies consumers
Mike (3:11 PM) - sees alert
Lisa (3:11 PM) - sees alert
David (3:12 PM) - phone was locked, sees it now
  ↓
Mike responds "I'll watch Jake"
  ↓ sends confirmation through Kafka
Sarah (3:12 PM) - sees confirmation

Why This Architecture Works

Decoupling:

Services don’t talk directly
Add/remove services without breaking others

Persistence:

Messages stored on disk
Survive crashes and restarts

Scalability:

Add more consumers = faster processing
Add more producers = handle more load

Reliability:

One service down? Others keep working
Messages don’t get lost

Real-World Use Cases

Same pattern, different use cases:

E-commerce:

Order placed → Kafka
Payment service charges card
Inventory service updates stock
Email service sends confirmation

Uber:

Ride requested → Kafka
Driver matching finds nearby driver
Pricing calculates fare
Notifications alert driver

Your bus stop:

Alert sent → Kafka
Notification service alerts neighbors
Database logs the event
Analytics tracks usage

All use the same Kafka pattern you just learned.

Common Questions

“Why not just use a database?”

Database: Consumer constantly polls “any new data?”
Kafka: Consumer waits, Kafka notifies when ready
Result: Real-time, less load

“Why not just use REST API?”

REST: Consumer must be online NOW
Kafka: Consumer reads when ready
Result: More reliable, works offline

“When should I use Kafka?”

✅ High message volume
✅ Multiple systems need same data
✅ Can’t lose messages
✅ Need message history

What You Built

bus-stop-kafka/
├── docker-compose.yml  # Kafka setup
├── producer.py         # Send alerts
├── consumer.py         # Receive alerts
├── venv/               # Virtual environment
├── .gitignore          # Git ignore file
└── README.md           # Project documentation

Summary

You learned:

What Kafka is (message broker)
Why it’s useful (decoupling, persistence)
How to produce messages
How to consume messages
Consumer groups concept

You built:

Working producer that sends alerts
Working consumer that receives alerts
Everything runs locally with Docker

You can now:

Explain Kafka to anyone
Build event-driven systems
Apply this to other use cases

Resources

📦 Code: github.com/sprider/bus-stop-kafka
📚 Learn More: Kafka Docs
🎥 Watch: Nana’s Kafka Video

How to Build a Bible MCP Server: Complete Guide to Creating Custom AI Tools

Sun, 24 Aug 2025 00:00:00 +0000

A few months ago, I found something that changed my views on AI tools. It’s called MCP (Model Context Protocol); it allows AI models to connect to external tools and data sources.

What is Model Context Protocol (MCP) and Why Should You Care?

Think of MCP as the glue that binds different AIs into a unified network. Before MCP, if I wanted Claude to help me with Bible study, I had to copy and paste verses, look up references manually, or jump between apps. It was cumbersome and interrupted my flow.

With MCP, I can create a custom server that gives Claude AI direct access to Bible data—verses, cross-references, commentaries, and more. It’s like having a personal research assistant who never gets tired.

Step-by-Step Guide: Building a Bible MCP Server on Cloudflare Workers

I am a hands-on solutions architect, and I wanted to make something useful for my daily Bible study. Here’s how it went down:

Identifying the Problem: Why Build a Custom Bible AI Tool?

Every morning, I read scripture and take notes. I often wondered, What other verses relate to this theme?. But looking this up meant opening multiple tabs, losing my place, and breaking my focus.

I thought, What if Claude could just know this stuff?

How to Build Your First MCP Server: Technical Implementation

The beauty of building MCP servers is that you don’t need to be an expert programmer. I used Cloudflare Workers because they are free for small projects and easy to set up for custom AI integrations.

My server exposes two consolidated MCP tools (updated from an earlier 6-tool design):

bible_content — Search verses, get a single verse, passage, or full chapter (4 actions in one tool)
bible_reference — List books or chapters to navigate Bible structure

Each tool supports a response_format option (concise/detailed) to control token usage—I applied the principles from my AI Tool Optimization Guide to reduce tool count by 67% and token usage by 60–70%.

The MCP server code is surprisingly simple. Model Context Protocol handles all the complex communication, so I just focused on the Bible API integration and data logic.

Deploying MCP Server on Cloudflare Workers: Free and Fast

Here is where it gets interesting. I deployed my server to Cloudflare Workers, and now it runs all the time without my involvement. There’s no server maintenance, no hosting fees (thanks to Cloudflare’s generous free tier), and it works quickly because of Cloudflare’s global CDN network.

Then I connected it to Claude AI through the MCP protocol integration, and suddenly, my AI assistant became a personalized Bible study companion.

MCP Server Benefits: Real-World AI Integration Results

Now, when I study, I can ask Claude questions like:

How did Jesus feed 5,000 people?
What is the context around Romans 8:28?
Compare this verse across different translations

Claude doesn’t just give me generic answers; it pulls real data from my server and provides exactly what I need.

Beyond Bible Study: MCP Use Cases for Custom AI Development

What excites me most is not just my Bible server. It’s the concept behind MCP. We are transitioning from AI language models that act as isolated information bubbles to AI assistants engaging with real-world data and APIs through custom MCP servers.

Imagine connecting your AI to:

Your company’s internal database
Your personal calendar and task manager
Stock market APIs for real-time trading info
Your smart home devices
Medical databases for health research

The options are endless, and the barrier to entry is surprisingly low.

MCP Development Best Practices: Lessons from Building AI Tools

Building this MCP server for Bible study taught me several important lessons about custom AI development:

Start small: I didn’t try to build everything all at once. My first version just returned single verses. Then I added search. Iteration is your friend.
Think in tasks, not endpoints: I later consolidated six tools into two by grouping related actions (search, verse, passage, chapter into bible_content). Adding a response_format parameter (concise vs. detailed) cut token usage by 60–70%. See my AI Tool Optimization Guide for the full approach.
MCP does the heavy lifting: I spent much more time thinking about the Bible data structure than the protocol. MCP simplifies all the connection challenges.
Free tier is powerful: I built something useful without spending a dime using Cloudflare Workers and free Bible APIs.
Documentation matters: When I got stuck, the MCP documentation and community examples saved me hours of debugging.

Advanced MCP Features: Scaling Your Custom AI Server

I’ve already optimized the server for token efficiency: tools are consolidated by task, and a response_format parameter lets the agent request concise or detailed responses. Next, I’m considering commentary integration or connecting to my personal notes and highlights. The great thing about MCP is that adding new features doesn’t require rebuilding everything—I just extend the existing tools.

The Future of AI Customization: Why MCP Matters for Developers

We are witnessing the rise of personalized AI. Not AI that knows everything about everyone, but AI that knows exactly what you need it to know. MCP makes this possible by letting us build connections between AI models and our specific data sources.

My Bible MCP server is just one example, but it represents something larger: the opening up of AI customization. You don’t need to work at a tech company to empower your AI assistant. You just need an idea and a weekend.

If you’re interested in building your own MCP server, start with something you care about. For me, it was Bible study. For you, it might be cooking recipes, fitness tracking, or managing your book collection. The technical side is the easy part—the harder part is deciding how you want your AI to assist you.

Once you experience an AI assistant that truly understands your specific needs and can access your data, you won’t want to go back to generic chatbots.

You can find my Bible MCP server code on GitHub if you want to see how it works or build something similar. For more MCP examples, check out the official MCP documentation and MCP server examples.

AI Not Working Consistently? Here is How to Control Your Results

Sat, 12 Jul 2025 00:00:00 +0000

Imagine you are using a highend espresso machine at work. Sometimes you get the perfect cup(rich, smooth, and exactly the right strength). Other times even when you seem to follow the same process, you end up with bitter, weak, or overpowering coffee. You would probably think the machine is unreliable, right?

This is what business professionals face with AI tools every day. You ask for a marketing email and receive something brilliant. But ask again with the same prompt, and suddenly it sounds robotic. The issue is not that AI is unreliable; it is that most people do not realize there are hidden settings controlling every response.

Just like that espresso machine has temperature controls, grind settings, and pressure adjustments you might not notice, AI tools have seven key parameters that act as invisible control knobs. Once you understand what these knobs do, you can stop getting random results and start producing consistently excellent AI responses.

Today, we are going to uncover these seven hidden controls(LLM parameters) and show you how to adjust them. You would not need any technical expertise—just practical examples that will change your AI results from unpredictable to reliable and consistent.

Streamline Location-Relevant Answers with SharePoint, Amazon Nova and Bedrock

Sat, 19 Apr 2025 00:00:00 +0000

Global companies often face challenges in providing employees with location-relevant policies. For instance, leave policies in the USA differ significantly from those in India. However, when documents are stored together in systems like SharePoint without proper filtering, employees may waste time searching or risk following incorrect policies. The unfiltered content in Amazon Bedrock creates problems for its knowledge base by producing incorrect answers.

The Solution: Metadata Filtering with SharePoint and Amazon Bedrock

By integrating Amazon Bedrock Knowledge Bases with SharePoint and leveraging metadata filtering, companies can create intelligent Retrieval-Augmented Generation (RAG) systems. These systems automatically retrieve relevant policy documents based on location filters so employees get the right location-relevant information.

How It Works (In Simple Terms)

Organize Documents in SharePoint: Assign metadata (e.g., country-relevant tags) to each document.
Connect SharePoint to Amazon Bedrock: Sync SharePoint as a data source for Amazon Bedrock Knowledge Bases.
Apply Metadata Filters: Use filters to retrieve only location-relevant content when employees query the system.

Real-World Example: Leave Policies

Consider leave policies for the USA and India:

USA Policy: Based on ACME Corporation’s USA Employee Leave Policy, employees receive different types of leave: Vacation Leave (0-2 years of service: 10 days/80 hours), Sick Leave - 5 days (40 hours) per calendar year. Additionally, employees receive paid holidays (11 days), bereavement leave, and jury duty leave. Eligible employees may receive up to 12 weeks for parental leave.
India Policy: According to ACME Corporation India’s leave policy, you are entitled to the following types of leave: Privilege/Earned Leave: 24 days per year, Sick/Casual Leave: 12 days per calendar year. Optional Holidays: 2 days per year. The policy includes other types of leave such as Maternity Leave: 26 weeks.

Disclaimer: The leave policies uploaded to SharePoint for this demonstration were created using AI. The AI generated policies are for illustrative purposes only.

Using metadata filtering:

Employees in the USA see only the USA policy.
Employees in India see only the India policy.

This eliminates confusion and ensures compliance.

Implementation Steps

Add metadata to your SharePoint documents

First, ensure your documents have the right metadata in SharePoint:

We will use the default Title column in your SharePoint document library
Assign “Leave_Policy_USA” or “Leave_Policy_India” to the appropriate documents

Set up a connection between SharePoint and Amazon Bedrock

Next, set up a connection between SharePoint and Amazon Bedrock:

In AWS console, create a new Knowledge Base
Select SharePoint as your data source
Set up SharePoint App-Only authentication to connect to SharePoint
Sync the data source to begin indexing content from SharePoint

Note: I’m still exploring how custom metadata columns can be used for unstructured data formats. If I find a solution, I’ll create a separate blog post. For now, we’ll focus on using the out-of-the-box metadata fields generated by the OpenSearch collection.

Test metadata filtering using sample queries to ensure accuracy

Let us test a few questions both with and without filters to see how the selected model generates responses. This will help demonstrate the difference in relevance and accuracy when metadata filtering is used. For this example, I’ve used the Nova Pro 1.0 model to generate the responses.

No Filter

As you can see, the answers are a mix of both USA and India policies, with chunks being pulled from documents for both regions.

With x-amz-bedrock-kb-title ^ Leave_Policy_USA Filter

With the filter x-amz-bedrock-kb-title ^ Leave_Policy_USA, the response is clearly relevant to the USA, showing only the relevant policy for that region.

With x-amz-bedrock-kb-title ^ Leave_Policy_India Filter

With the filter x-amz-bedrock-kb-title ^ Leave_Policy_India, the response is clearly relevant to the India, showing only the relevant policy for that region.

Benefits of Metadata Filtering

Accurate Information: Employees access policies relevant to their region.
Time-Saving: Reduces time spent sifting through irrelevant documents.
Improved Compliance: Ensures employees follow the correct policies.
Centralized Management: All policies remain in one system for easy updates.

Conclusion

Combining SharePoint’s document management capabilities with Amazon Bedrock’s metadata filtering creates a powerful solution for global organizations. This approach simplifies policy management and ensures employees receive accurate, location-relevant information without requiring complex coding or major system changes.

Docker Model Runner: Run AI Models Locally with Seamless Integration

Mon, 07 Apr 2025 00:00:00 +0000

In the fast-paced world of AI development, tools that simplify the process of running and integrating AI models locally are in high demand. Docker’s latest beta feature, Model Runner, offers developers a seamless way to work with AI models directly within their existing Docker environment. This article explores the features, benefits, and practical applications of Docker Model Runner, making it an essential resource for developers looking to optimize their AI workflows.

What is Docker Model Runner?

Docker Model Runner is a beta feature designed for Docker Desktop users, enabling them to download, run, and manage AI models locally. By pulling models from Docker Hub, storing them locally, and loading them into memory only when needed, it optimizes system resources. For developers already familiar with Docker’s containerization tools, Model Runner provides a streamlined experience with OpenAI-compatible APIs for easy integration into applications.

Key Benefits of Docker Model Runner

Local AI model management: Pull, run, and remove models directly from the command line.
Resource optimization: Models are only loaded at runtime and unloaded when idle.
OpenAI-compatible APIs: Simplify integration with existing applications.
Familiar workflows: Leverage Docker commands you already know.

Key Features of Docker Model Runner

To make the most of Docker Model Runner, here are its standout features:

Pull AI models directly from Docker Hub.
Run models locally using simple commands.
Manage local models with options to add, list, or remove them.
Interact with models via prompts or chat mode.
Optimize resource usage, ensuring efficient memory management.
Access OpenAI-compatible APIs, enabling seamless integration into your applications.

These features make Docker Model Runner a game-changer for developers aiming to build custom AI assistants or agents.

How to Get Started with Docker Model Runner

Prerequisites

To start using Docker Model Runner, you’ll need:

Docker Desktop version 4.40 or later
A Mac with Apple Silicon (currently supported)
Beta features enabled in Docker Desktop under “Features in development”

Basic Commands

Here’s a quick guide to essential commands:

Check if Model Runner is active

docker model status

Pull a model from Docker Hub

docker model pull ai/smollm2

List downloaded models

docker model list

Run a model with a single prompt

docker model run ai/smollm2 "What is Kubernetes?"

Remove a model

docker model rm ai/smollm2

These commands allow you to efficiently manage and interact with AI models directly from your terminal.

Building AI Assistants with Docker Model Runner

One of the most exciting use cases for Docker Model Runner is building custom AI assistants. Developers can integrate these assistants into applications using OpenAI-compatible APIs. Here’s how you can access these APIs:

From Within Containers:

   #!/bin/sh

   curl http://model-runner.docker.internal/engines/llama.cpp/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
         "model": "ai/smollm2",
         "messages": [
               {
                  "role": "system",
                  "content": "You are a helpful assistant."
               },
               {
                  "role": "user",
                  "content": "What is Kubernetes?"
               }
         ]
      }'

From the Host (Unix Socket):

   #!/bin/sh

   curl --unix-socket $HOME/.docker/run/docker.sock \
      localhost/exp/vDD4.40/engines/llama.cpp/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
         "model": "ai/smollm2",
         "messages": [
               {
                  "role": "system",
                  "content": "You are a helpful assistant."
               },
               {
                  "role": "user",
                  "content": "What is Kubernetes?"
               }
         ]
      }'

From the host using TCP:

If you prefer to interact with the API directly from your host machine using TCP rather than a Docker socket, you can enable this functionality. TCP support can be activated either through the Docker Desktop graphical interface or by using the Docker Desktop command line with sh docker desktop enable model-runner --tcp <port>

Once TCP support is enabled, you can communicate with the API through localhost using either your specified port number or the default port, following the same request format shown in the previous examples.

   #!/bin/sh

   curl http://localhost:12434/engines/llama.cpp/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
         "model": "ai/smollm2",
         "messages": [
               {
                  "role": "system",
                  "content": "You are a helpful assistant."
               },
               {
                  "role": "user",
                  "content": "What is Kubernetes?"
               }
         ]
      }'

For hands-on examples, check out the official https://github.com/docker/hello-genai.git repository on GitHub. It includes sample applications in Python, Node.js, and Go.

Where to Find Models

Docker provides an extensive collection of pre-trained AI models on its Gen AI Catalog at https://hub.docker.com/catalogs/gen-ai. Popular options include:

SmolLM2: Tiny LLM built for speed, edge devices, and local development.
Llama Models: Available in various sizes for different use cases.
Other optimized models tailored for specific applications.

This centralized hub simplifies finding and deploying the right model for your needs.

Known Limitations

While Docker Model Runner shows great promise, it’s important to note some current limitations:

Lack of safeguards for oversized models that may exceed system resources.
Chat interface may still launch even if the model pull fails.
Progress reporting during model pulls can be inconsistent.

These issues are expected to improve as the feature evolves beyond its beta phase.

Why Choose Docker Model Runner?

For developers already working within the Docker ecosystem, Model Runner offers several compelling advantages:

Unified platform: Manage both containers and AI models in one environment.
Familiar commands: No steep learning curve for existing Docker users.
Resource efficiency: Load models only when needed to save memory.
Seamless integration: Easily connect AI capabilities to your applications via OpenAI-compatible APIs.

By leveraging these benefits, developers can enhance their productivity while simplifying their workflows.

Conclusion

As artificial intelligence becomes increasingly integral to modern applications, tools like Docker Model Runner are paving the way for more accessible and efficient development processes. With its ability to integrate seamlessly into existing workflows while optimizing resource usage, this beta feature holds immense potential for developers and DevOps engineers alike.

Start exploring Docker Model Runner today and take your AI development workflow to the next level!

DeepSeek vs. OpenAI: The AI Race Heats Up !

Sat, 01 Feb 2025 00:00:00 +0000

The world is buzzing about how DeepSeek has outperformed OpenAI on various benchmarks. This got me thinking that AI is evolving at an incredible speed, with endless opportunities. But beyond competition, what excites me is how AI can meaningfully solve real-world problems.

💡 A Personal Story

Due to my job, I have traveled to various cities in India and abroad. No matter where I go, one of the first things I look for is a church to attend. Over the years, I have noticed a familiar pattern: churches want to engage members meaningfully, and members wish to contribute through volunteering (kids’ ministry, small groups, outreach, etc.). However, most churches still rely on weekly announcements and flyers to match people with opportunities.

That is when I thought: Can AI help? 🤔

🎯 Introducing Ministry Matcher

I built an AI-driven hobby project “Ministry Matcher” to connect church members with service opportunities based on their backgrounds and interests. Instead of waiting for announcements, members can explore personalized recommendations in seconds!

To simplify this hobby project, I wrapped the logic using Python and OpenAI’s chat completion API with custom prompt instructions(which can be changed at any time) in a Docker image and deployed it using the Azure Container Apps.

The best part? This concept can be applied beyond churches

✅ New employees in a company finding the right team activities

✅ New customers in a bank discovering tailored services

✅ Community groups onboarding new members seamlessly

🌟 AI is more than just a race between models. It is about impact. Let us use technology to make life easier and more meaningful.

I would love to hear your thoughts! What are some ways AI can help in community engagement? 💙

Modernizing Bot Infrastructure: A Kubernetes Success Story

Sun, 26 Jan 2025 00:00:00 +0000

I led a project transforming our scattered bot infrastructure to Kubernetes. With bots spread across multiple servers and tech stacks, our teams faced maintenance challenges and rising costs.

🎲 The challenge

Bots were created for various projects using different tech stacks and deployed across multiple servers. It created a complex system with:

Inconsistent deployment processes
Varied maintenance requirements
Redundant infrastructure costs
Limited scalability options

How we tackled

💪 Here is how we tackled it at a high level using the Assess, Mobilize, and Modernize framework:

🔍 Assess: AWS Application Discovery Service (ADS) revealed crucial insights

Mapped bot dependencies across different environments
Identified resource utilization overlap
Uncovered opportunities to standardize common functionalities
Created detailed migration paths for each bot’s unique requirements

🏗️ Mobilize: Established our Kubernetes foundation

Prepared an existing Kubernetes cluster for hosting bot applications
Created standardized templates for bot containerization
Conducted hands-on workshops for team upskilling
Implemented centralized monitoring and logging

⚡ Modernize: Executed our transformation

Refactored bots into containerized applications
Established automated testing and validation
Deployed the bots via DevSecOps pipelines
Monitored and refined deployed resources

📕 Key Learnings

Using AWS Application Discovery Service helped us understand how our systems were connected and being used, which guided our migration planning
The team adoption process depended on enabling workshops and documentation
Standardized templates accelerated the containerization process
Ongoing feedback loops played a crucial role in improving our migration approach

🎯 Impact

The migration changed our operations. Deployment cycles shrank from hours to minutes. We cut our monthly spending by 60%. Our new infrastructure maintains consistent uptime with zero-downtime deployments as standard practice.

The impact extended beyond just technical enhancements. Because of this change in our work culture, our development cycles moved faster, inspiring innovation throughout our projects. Teams that used to work separately started collaborating regularly by exchanging knowledge and resources.

🤝 Would love to hear your modernization story! What challenges have you encountered so far?

Hidden Powers of Kubernetes Sidecar Containers: Beyond the Basics

Sun, 29 Dec 2024 00:00:00 +0000

While most Kubernetes engineers know what sidecar containers are and why they are necessary for logging and service mesh, these useful little components have other less obvious capabilities that are not well known. Let’s look at some powerful features that could revolutionize your containerized applications.

Shared Filesystem Superpowers

This enables real-time file watching, dynamic SSL renewal, and real-time log processing. Imagine configuring hot reload or running security scans without touching the primary application through shared volume mounts, allowing sidecar containers to communicate with the main container’s filesystem.

This example demonstrates filesystem sharing between containers in a pod using an emptyDir volume. The main nginx container and a sidecar container share access to the same volume, where the sidecar writes a timestamp every 10 seconds that nginx then serves as its index page. This showcases how containers within a pod can communicate through a shared filesystem.

shared-filesystem.yaml

apiVersion: v1
kind: Pod
metadata:
  name: shared-filesystem-demo
spec:
  containers:
  - name: main-app
    image: nginx:latest
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  - name: sidecar-config
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do date > /data/index.html; sleep 10; done"]
    volumeMounts:
    - name: shared-data
      mountPath: /data
  volumes:
  - name: shared-data
    emptyDir: {}

# Deploy the pod
kubectl apply -f shared-filesystem.yaml

# Watch the main container's filesystem changes
kubectl exec shared-filesystem-demo -c main-app -- /bin/sh -c 'while true; do clear; cat /usr/share/nginx/html/index.html; sleep 2; done'

# Verify the content is updating
kubectl port-forward shared-filesystem-demo 8080:80
curl localhost:8080

The Init Sidecar Pattern

The init sidecar pattern—where a sidecar container in the pod spec finishes its execution before the main container can start—is less well-known. This pattern is particularly useful for dynamic resource configuration and runtime dependency injection, offering more flexibility than traditional init containers.

This Kubernetes manifest does sequential configuration management through three containers: an init container sets up an initial config file, then the main container reads this file every 30 seconds, while simultaneously a sidecar container updates the same file with timestamps every 60 seconds. All containers share a common emptyDir volume to enable this file-based communication.

init-sidecar.yaml

apiVersion: v1
kind: Pod
metadata:
  name: init-sidecar-demo
spec:
  initContainers:
  - name: init-config
    image: busybox
    command: ['sh', '-c', 'echo "Initial config" > /config/config.ini']
    volumeMounts:
    - name: config-vol
      mountPath: /config
  containers:
  - name: main-app
    image: ubuntu:latest
    command: ['sh', '-c', 'while true; do cat /app/config.ini; sleep 30; done']
    volumeMounts:
    - name: config-vol
      mountPath: /app
  - name: sidecar-config-updater
    image: busybox
    command: ['sh', '-c', 'while true; do echo "Updated config $(date)" > /config/config.ini; sleep 60; done']
    volumeMounts:
    - name: config-vol
      mountPath: /config
  volumes:
  - name: config-vol
    emptyDir: {}

# Deploy and monitor
kubectl apply -f init-sidecar.yaml
kubectl logs init-sidecar-demo -c main-app
kubectl logs init-sidecar-demo -c sidecar-config-updater

Process-Level Communication

When configured with shareProcessNamespace: true, you can send UNIX signals from the sidecar to processes within the main container. With standard container interactions, you won’t be able to perform graceful shutdowns, health management, and other forms of sophisticated debugging.

This manifest demonstrates inter-container process communication using Linux signals. With shared process namespace enabled, the signal-sender container sends a SIGHUP signal every 10 seconds to the main container, which is configured with a trap handler to respond with “Received SIGHUP!” message when it receives the signal, while also displaying the date every 5 seconds.

signal-demo.yaml

apiVersion: v1
kind: Pod
metadata:
  name: signal-demo
spec:
  shareProcessNamespace: true  # This enables processes to see each other across containers
  containers:
  - name: main-app
    image: busybox
    command: ["/bin/sh", "-c"]
    args:
      - |
        echo "Starting main process..."
        # trap command sets up a signal handler
        # When SIGHUP is received, it will execute 'echo "Received SIGHUP!"'
        trap 'echo "Received SIGHUP!"' HUP
        while true; do
          date
          sleep 5
        done
  - name: signal-sender
    image: busybox
    command: ["/bin/sh", "-c"]
    args:
      - |
        sleep 2
        while true; do
          echo "Sending SIGHUP to main process..."
          # pkill finds and sends signals to processes based on their name/pattern
          # -HUP: sends SIGHUP signal
          # -f: matches against full command line
          # "Starting main process": pattern to match (from the echo in main-app)
          pkill -HUP -f "Starting main process"
          sleep 10
        done

# Deploy the signal demo
kubectl apply -f signal-demo.yaml

# Watch the logs from both containers
kubectl logs signal-demo -c main-app -f
kubectl logs signal-demo -c signal-sender -f

Dynamic Configuration Management

Even after the pod has been created, sidecar containers can receive and alter environment variables from the Kubernetes Downward API. Combining this with the shared filesystem capability allows you to perform runtime secret rotation and other adaptive container techniques.

The manifest creates a pod where a sidecar container updates the pod’s version label every 30 seconds using the current timestamp, while the main container continuously reads this version through a Downward API volume mount and prints it alongside the current time. The RBAC configuration grants the pod permissions to modify its own labels.

env-inherit.yaml

apiVersion: v1
kind: Pod
metadata:
  name: env-inherit-demo
  labels:
    version: "1.0.0"
spec:
  serviceAccountName: pod-labeler
  containers:
  - name: main-app
    image: ubuntu:latest
    command: ['/bin/bash', '-c']
    args:
      - |
        while true; do 
          echo "Current time: $(date)"
          echo "APP_VERSION=$(cat /etc/podinfo/version)"
          sleep 10
        done
    volumeMounts:
    - name: podinfo
      mountPath: /etc/podinfo
  - name: version-updater
    image: bitnami/kubectl
    command: ['/bin/bash', '-c']
    args:
      - |
        while true; do
          NEW_VERSION="$(date +%H-%M-%S)"
          echo "Updating version to $NEW_VERSION"
          kubectl label pod $POD_NAME version=$NEW_VERSION --overwrite
          sleep 30
        done
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
  volumes:
  - name: podinfo
    downwardAPI:
      items:
      - path: "version"
        fieldRef:
          fieldPath: metadata.labels['version']

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-labeler
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-labeler
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-labeler
subjects:
- kind: ServiceAccount
  name: pod-labeler
roleRef:
  kind: Role
  name: pod-labeler
  apiGroup: rbac.authorization.k8s.io

# First create the RBAC resources
kubectl apply -f rbac.yaml

# Then create the pod
kubectl apply -f env-inherit.yaml

# Watch the main-app logs to see the version changes
kubectl logs env-inherit-demo -c main-app -f

# In another terminal, you can watch the version-updater logs
kubectl logs env-inherit-demo -c version-updater -f

# You can also verify the label changes
kubectl get pod env-inherit-demo --show-labels

Resource Management

Kubernetes’ Quality of Service (QoS) features are used by sidecar containers and the primary container to manage the CPU and memory allocations of the sidecar containers and the primary container. These features can cause resources to be dynamically redistributed leading to improved efficiency of the cluster. This is useful for optimizing cloud application performance and helps with cost effective resource management.

The manifest creates a pod with three containers sharing a process namespace: an nginx container, a monitoring container that displays system stats every 5 seconds, and a load generator running multiple dd commands. Each container has specific CPU and memory limits, demonstrating Kubernetes resource management and container resource isolation.

resource-demo.yaml

apiVersion: v1
kind: Pod
metadata:
  name: resource-demo
spec:
  shareProcessNamespace: true
  containers:
  - name: main-app
    image: nginx:latest
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: resource-monitor
    image: busybox
    command: ['sh', '-c', 'while true; do echo "---$(date)---"; top -b -n 1 | head -n 20; sleep 5; done']
    resources:
      requests:
        memory: "32Mi"
        cpu: "100m"
      limits:
        memory: "64Mi"
        cpu: "200m"
  - name: load-generator
    image: busybox
    command: ['sh', '-c', 'while true; do for i in $(seq 1 4); do dd if=/dev/zero of=/dev/null bs=1M count=1024 & done; wait; done']
    resources:
      requests:
        cpu: "100m"
        memory: "32Mi"
      limits:
        cpu: "200m"
        memory: "64Mi"

# Deploy the new version
kubectl apply -f resource-demo.yaml

# Watch the resource monitor output
kubectl logs resource-demo -c resource-monitor -f

Wrap-Up

These capabilities make sidecars very powerful and should be used properly. One must design for security and failure scenarios because sharing filesystems and process namespaces creates a strong coupling between containers. The key is balancing these advanced features with maintainable and reliable architectures.

Thus, by understanding and properly using these underutilized features, one can develop more complex yet efficient and manageable container-based applications. As you use these features, remember that with great power comes great responsibility—use them wisely and document them for your team.

Intelligent Kubernetes Event Summarizer: A Step-by-Step Guide with a Demo

Sun, 10 Nov 2024 00:00:00 +0000

Kubernetes is powerful but it also can be complicated due to the number of events and logs it generates. It is crucial to have the tools in place that enable you to summarise these events quickly when someone points out the root cause of an issue amongst all those logs or error traces. This article proposes a intelligent Kubernetes event summarizer that utilizes OpenAI’s language model to summarize complex event logs into abstract human-readable form. Such a tool can also help the operations team save time and respond to issues faster.

What This Tool Does?

This tool connects to your kubernetes cluster pulls out pod events and uses OpenAI language model under the hood to summarize them in a concise format. Teams can easily access the summarization output as it is exposed through RESTful API.

Key Features

Gets events reported about the pod from Kubernetes cluster.
Summarizes events using OpenAI API.
Has a REST API to access event summary

Technical Architecture

Overview

This tutorial works with kubernetes v1.30.2 to set up a local development environment with Docker Desktop 4.35.1 (173168). This version is a simple set up for experimentation.
I use the Python Kubernetes client to connect to the cluster and Flask as my web framework for creating the API.

Components Explained

Kubernetes Cluster: The source of pod events.
OpenAI API: Summarizes the events into human-readable text.
Flask App: Serves as an API that returns the event summary.

Note for Other Clusters:

If you use a different cluster setup (e.g., Minikube, EKS, GKE, or AKS), make sure to handle authentication correctly. The line config.load_kube_config() in app.py loads the default kubeconfig file for authentication. Adjustments may be needed to match your cluster’s authentication method.
For in-cluster deployments, use config.load_incluster_config() to load the cluster’s service account credentials for proper authentication.

Demo Walkthrough

Setting Up the Environment

Clone the Repository:

git clone https://github.com/sprider/k8s-event-summarizer.git
cd k8s-event-summarizer

Create a Virtual Environment and Install Dependencies:

python3 -m venv venv
source venv/bin/activate
pip3 install -r app/requirements.txt

Set Up Environment Variables:

Create a .env file in the app/ directory and add the OpenAI API key:

OPENAI_API_KEY=your_openai_api_key

Running the Flask App Locally

Start the Flask App:

python3 app/app.py

Access the API:

Open a browser or use Postman to visit http://localhost:8000/summarize/<pod-name> to get summary for a specific pod.

Using Docker

Build the Docker Image:

docker build -t k8s-event-summarizer .

Run the Docker Container:

docker run -p 8000:8000 --env-file app/.env k8s-event-summarizer

Deploying a Test Pod in Your Cluster

To generate events for testing, use the test-pod.yaml file provided in the repository:

kubectl apply -f test-pod.yaml

This will create a test pod that can trigger events, which the event summarizer can then process and summarize.

Accessing Summarized Events via the API

Use curl or Postman to send a GET request to the API endpoint:

curl http://localhost:8000/summarize/test-pod

Replace test-pod with the name of the pod for which you want to retrieve event summary.

Understanding the Code Structure

app.py: The main Python script that sets up the Flask API, connects to the Kubernetes cluster, fetches events, and sends them to OpenAI for summarization.
requirements.txt: Lists all the Python dependencies needed for the app.
.env: Stores environment variables such as the OpenAI API key.
Dockerfile: Contains the instructions to build and run the app as a Docker container.
test-pod.yaml: A sample YAML file for creating a test pod in the Kubernetes cluster to generate events.

Wrap-Up

Kubernetes Event Summarizer — it helps in summarizing the events from the logs by creating human-friendly summaries of pod events. It can be customized to cover other use cases as well, for instance to monitor events of deployments and services and hence its very flexible. This project is a basic version which can be extended to address multiple operational requirements. This little tool can also be a game changer for your DevOps workflow if coupled with alerting systems or dashboards.

While this demo focuses on summarizing pod events, the same approach can be applied to monitoring other types of events, such as deployment events, service and ingress events, and more.

In real-world scenarios, clusters can generate millions of events. A practical next step would be to ingest these events into a vector database with embeddings, allowing the tool to function as a question-and-answer system. This would enable users to ask specific questions, with NLP capabilities extracting key attributes and calling relevant methods, resulting in a more versatile system that covers a wide range of use cases.

I hope you found this article helpful!

Leveraging eBPF for Container Network Monitoring with Cilium

Thu, 10 Oct 2024 00:00:00 +0000

In modern Kubernetes environments, monitoring network traffic and securing containers can be challenging due to their dynamic nature. Cilium, powered by eBPF(Extended Berkeley Packet Filter), offers high-performance visibility and security for containerized applications. This article will guide you through setting up Cilium on a Docker Desktop-based Kubernetes cluster to monitor network traffic and detect suspicious outbound connections.

What is Cilium and eBPF?

Cilium is an open-source networking and security project that leverages eBPF to provide efficient and flexible network observability and enforcement. eBPF runs programs directly in the Linux kernel, enabling fine-grained monitoring of network activity with minimal performance overhead.

Setting Up Cilium on Docker Desktop Kubernetes

Before we dive into monitoring traffic, let’s get Cilium up and running on your Docker Desktop Kubernetes setup.

Ensure Kubernetes is enabled in Docker Desktop:
Open Docker Desktop settings and enable Kubernetes. After Kubernetes starts, verify with:

kubectl config current-context

Install Cilium using Helm:

Helm is the easiest way to install Cilium in Kubernetes. First, add the Cilium Helm repository:

helm repo add cilium https://helm.cilium.io/
helm repo update

Then install Cilium:

helm install cilium cilium/cilium --namespace kube-system

Verify Cilium is running:

kubectl get pods -n kube-system

Check Cilium’s status:

After installation, run the following to ensure Cilium is functioning correctly:

kubectl exec -n kube-system -it cilium-xxxxx -- cilium status

Monitoring Network Traffic with Cilium

Now that Cilium is installed, you can begin monitoring network traffic. Cilium’s monitor command gives you detailed, real-time visibility into all packets flowing through your cluster.

Start monitoring network traffic:

To see live network flows, run:

cilium monitor

You’ll start seeing information about network packets, such as source and destination IPs, ports, and protocols. This visibility is powered by eBPF, which runs directly in the kernel for low-latency, high-performance monitoring.

Simulating Suspicious Outbound Connection

Now let’s simulate a suspicious outbound network connection to demonstrate how Cilium can be used to monitor and potentially block unexpected traffic.

Run a container making an outbound HTTP request:

To simulate suspicious behavior, run a pod that makes an HTTP request to an external server (e.g., example.com):

kubectl run external-connect --image=curlimages/curl --restart=Never --stdin --tty --command -- curl http://example.com

Monitor traffic with Cilium:

With the pod making a connection, use Cilium’s monitor command to capture and analyze the network flow:

cilium monitor -t flow

You should see output like:

12:34:56.789 10.244.1.4 -> 93.184.216.34 HTTP

This shows an HTTP request from the pod (10.244.1.4) to the external IP (93.184.216.34, which resolves to example.com). While this may be harmless in some cases, in a real-world scenario, unexpected outbound traffic like this could be a sign of data exfiltration, malware activity, or an unauthorized connection attempt.

Applying Network Policies to Control Outbound Traffic

To enforce security, you can apply network policies in Cilium to control which services or pods are allowed to make outbound connections.

Create a network policy that restricts outbound traffic:

Here’s an example policy that blocks all outbound traffic except within the same namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
    name: deny-all-except-same-namespace
spec:
    podSelector: {}
    policyTypes:
    - Ingress
    - Egress
    egress:
    - to:
        podSelector: {}
    ingress:
    - from:
        podSelector: {}

Apply the network policy:

Apply the policy using kubectl:

kubectl apply -f deny-all-except-same-namespace.yaml

With this policy in place, any attempt by a pod to send traffic outside its namespace will be blocked, including the suspicious outbound request to example.com.

Monitor policy violations:

If the pod tries to make a connection outside the allowed network, Cilium will log a policy violation, helping you identify unauthorized traffic attempts:

cilium monitor --type drop

Cleaning Up

After the demo, you can clean up the resources:

Delete the test pod:

kubectl delete pod external-connect

Uninstall Cilium (if no longer needed):

helm uninstall cilium --namespace kube-system

Conclusion

By using Cilium with eBPF on Docker Desktop’s Kubernetes cluster, you can gain real-time visibility into network traffic, making it easier to detect suspicious outbound connections. With Cilium, monitoring becomes more efficient and detailed, providing granular insights into network flows. By applying network policies, you can also proactively block unauthorized outbound traffic, adding an extra layer of security to your containerized environment.

Cilium’s eBPF-based approach allows for high-performance monitoring and security enforcement with minimal overhead, making it an excellent choice for Kubernetes environments, whether for local demos or production use.

Building a Unified Bible Platform: Q&A, Insights, and Ministry Matching

Thu, 10 Oct 2024 00:00:00 +0000

In today’s ever-evolving technological landscape, there are infinite possibilities to delve into and interact with Bible verses. I myself have recently evolved my personal endeavor from a single Q&A platform into a comprehensive unified application that combines three powerful Bible-focused services: AI-powered Q&A specifically tailored to the King James Version (KJV), curated insights browsing, and ministry recommendation matching.

Extracting the KJV Bible

The first step was to acquire the content. Thanks to the API, extracting the full text of the KJV Bible was a straightforward process. This API offers a structured format of the Bible verses, making the extraction process seamless.

Note: Bible verses taken from the King James Version (KJV) and sourced from api.bible (bible id: de4e12af7f28f599-02).

Generating Embeddings with OpenAI

After obtaining the Bible verses, the next challenge was understanding and contextualizing them. I employed OpenAI’s text-embedding-ada-002 model. This model, renowned for its capabilities, transforms textual data into embeddings - numerical representations of the text that capture the essence and context of the content.

Storing Embeddings in Pinecone

Quickly retrieving relevant embeddings was crucial to creating a responsive Q&A interface. Pinecone, a vector search service, provided the solution. After generating embeddings from the Bible verses, I stored them in a Pinecone index. Pinecone index ensured efficient and quick retrieval during user interactions.

Creating the Unified Platform with Python Flask

With the backend prepared, it was time to focus on the user interface. Python’s Flask framework was the tool of choice, now enhanced with a modular Blueprint architecture. Flask, known for its lightweight nature and flexibility, was perfect for setting up a unified platform that houses three distinct services:

Bible Q&A Service - The original AI-powered question answering
Bible Insights Service - Browse curated insights from Bible books and chapters
Ministry Matching Service - Church ministry recommendation system using OpenAI

Leveraging OpenAI for Response Generation

When users posed questions, the system needed to provide meaningful responses. By forwarding the user’s question to OpenAI, the model via LangChain’s Pinecone vector store would sift through the stored embeddings in Pinecone, corresponding to each Bible verse. This would pinpoint verses with content most relevant to the question.

Additionally, the ministry matching service leverages OpenAI’s conversational capabilities to provide personalized ministry recommendations based on user interests and calling.

Displaying Results

The unified platform now displays results from all three services through a cohesive interface. Users can seamlessly navigate between AI-powered Q&A, browse curated biblical insights, or discover ministry opportunities - all within a single application with shared navigation and consistent styling.

Modern Deployment with Azure Container Apps

Moving beyond traditional hosting, I deployed the unified application using Azure Container Apps, which provides several advantages over previous AWS EC2 deployment:

Serverless scaling: Automatically scales from 1-10 replicas based on demand
Managed SSL certificates: Automatic HTTPS with custom domain support
Container-based deployment: Docker containers for consistent environments
Integrated monitoring: Built-in logging and health checks
Cost optimization: Pay-per-use model with efficient resource utilization

The unified platform is now live at https://uba.josephvelliah.com, showcasing all three services in one cohesive experience.

The platform has evolved to include additional services beyond the original Q&A functionality. Curated insights are now integrated directly into the unified platform as the “Bible Insights” service, making scripture more accessible and engaging for users. The containerized deployment on Azure Container Apps ensures smooth operation and automatic scaling based on user demand.

Enhanced Technical Architecture

The unified platform demonstrates modern application design principles:

Modular Services: Clean separation of concerns with independent service functionality
Graceful Degradation: Services work independently; missing configuration doesn’t break the app
Security Features: Rate limiting, input sanitization, and Content Security Policy headers
Production Ready: Health checks, structured logging, and monitoring capabilities
Docker Containerization: Consistent deployment across environments

In conclusion, evolving from a single Bible Q&A application to a unified platform combining OpenAI’s powerful models, Pinecone’s efficient storage, curated insights, and ministry matching capabilities resulted in a comprehensive Bible study resource. It demonstrates how technology can enhance our understanding of Bible verses while providing practical tools for spiritual growth and community engagement, all brought closer to the modern user through thoughtful platform design.

Visit the live unified platform at https://uba.josephvelliah.com to experience all three services.