LLM calls are expensive and slow, and people often ask the same thing in different words. “What’s your refund policy?” and “How do I get my money back?” are different strings but the same question. Without a semantic cache, you pay full price for those repeats.
I spent a weekend building a cache that matches by meaning, not exact text, using only AWS services: S3 Vectors for similarity search, Bedrock for embeddings and the LLM, and Lambda for compute. No external vector DB. Cache hits were about 10x faster and cheap compared with calling the model again.
The Problem
Every Bedrock call costs money and usually takes 1-3 seconds. A large share of queries are the same question rephrased, so you end up paying for answers you already have.
The Solution
Instead of matching exact strings, I used vector embeddings to match meaning. When a new query comes in:
- Convert the query into a vector embedding (using Titan V2)
- Search for similar queries in the cache (using S3 Vectors)
- If similarity is above 85%, return the cached response
- Otherwise, call the LLM and cache the result for next time
Simple concept. The trick was making it work with AWS-native services only.
The Tech Stack
| Component | Service |
|---|---|
| Vector Storage | Amazon S3 Vectors |
| Embeddings | Bedrock Titan V2 |
| LLM | Bedrock Claude Haiku 4.5 |
| Compute | Lambda |
| API | API Gateway HTTP API |
The stack is serverless, so there is no always-on baseline cost.
Results
After running some tests:
- Cache hits are ~10x faster than calling the LLM
- Semantic matching works - “capital of France” matches “France’s capital city”
- Graceful degradation - if the cache fails, it falls back to the LLM
What I Learned
- S3 Vectors gave me similarity search without running a separate vector database.
- Cold starts were fine for this demo; requests began in about 300ms.
- The similarity threshold matters. 0.85 avoided most false matches while still catching rephrases.
Try It Yourself
The complete code is available on GitHub. One-click deploy, one-click cleanup.
GitHub: github.com/sprider/semantic-cache-demo
The repo includes:
- Full infrastructure as code (ready to deploy)
- 71 unit tests
- One-click deploy and cleanup scripts
- Architecture diagrams
Fair warning: it creates AWS resources that cost money. But the scripts make cleanup easy, and a few hours of testing costs less than a dollar.
What’s Next?
This is a demo, not production-ready code. For real use, you’d want:
- API authentication
- Cache invalidation strategy
- Multi-region deployment
- Better observability
Design alternatives
This demo uses S3 Vectors only for the cache layer. S3 Vectors has its own trade-offs (e.g. no built-in TTL, 40 KB metadata limit per vector). Combining S3 Vectors with DynamoDB—for example, storing vectors in S3 Vectors for similarity search and payloads or TTL in DynamoDB—lets you design differently for larger payloads, expiry, or exact-key lookups without changing the core flow shown here.
But as a proof of concept? It works. And it’s a pattern worth knowing.
Questions? Found a bug? Open an issue on the repo. Happy to chat about semantic caching, AWS architecture, or why vector databases are the future.