The promise and the problem
AI agents are everywhere. Every week brings a new framework, a new API, a new cloud service promising to make your AI smarter, faster, more capable. But if you’re running a homelab — or just care about your data — you’ve probably hit the same walls I have:
Cloud APIs are expensive. A single deep research session with GPT-4 can cost dollars. Run that daily for a month and you’re paying real money. For a homelabber or indie researcher, that’s not sustainable.
Your data leaves your machine. Every prompt, every document, every search query goes to someone else’s server. For personal research, medical papers, or proprietary work, that’s a non-starter.
Rate limits kill workflows. You’re deep in a research loop, 20 papers downloaded, building a knowledge base — and then you’re throttled. Or the API changes. Or the service shuts down.
Ephemeral containers lose your work. This one surprised me. Docker makes containers disposable, which is great for isolation but terrible for persistence. Download 50 papers into a container that disappears on restart? Gone.
Context windows are tiny. Your local LLM has 16k tokens. Each MCP tool description eats 100-200 tokens. Expose 60 tools and you’ve spent 25% of your context before the user says anything.
These aren’t edge cases. They’re the default experience for anyone trying to run AI agents locally.
What is MCP (and why should you care)?
The Model Context Protocol (MCP) is an open standard that lets AI models interact with external tools through a consistent interface. Instead of every AI app hardcoding integrations for every service, MCP provides a single protocol:
AI Model → MCP Client → MCP Gateway → MCP Servers (tools)Think of it like USB for AI tools. You don’t need a special cable for every device — you plug into a standard port. MCP is that standard port.
The protocol defines:
- Tools: Functions the AI can call (search papers, fetch a URL, take a screenshot)
- Resources: Data the server can provide (files, database records)
- Prompts: Templates for common interactions
Docker’s MCP Gateway takes this further by running each MCP server as an isolated Docker container. Want paper search? Spin up the paper-search container. Want Wikipedia? Spin up wikipedia-mcp. The gateway manages the lifecycle, routing, and security.
The vision: one gateway, many servers, any LLM
Here’s what I wanted to build:
┌─────────────┐ ┌──────────────┐ ┌─────────────────────────┐
│ Local LLM │────▶│ MCP Gateway │────▶│ 12+ MCP Servers │
│ (Ollama/LMS) │ │ (Docker) │ │ ├ paper-search │
│ │ │ │ │ ├ wikipedia-mcp │
│ │ │ Port 8811 │ │ ├ playwright │
│ │◀────│ SSE/Stream │◀────│ ├ markdownify │
│ │ │ │ │ ├ duckduckgo │
└─────────────┘ └──────────────┘ │ ├ sequentialthinking │
│ ├ fetch │
┌──────────────┐ │ ├ time │
│ Host Data │ │ ├ task-orchestrator │
│ /workspace │◀─────│ ├ memory/sqlite │
│ (persisted) │ │ └ ...more │
└──────────────┘ └─────────────────────────┘One gateway that any OpenAI-compatible LLM can connect to. Many servers running as containers, each with a specific purpose. Persistent data on the host filesystem, surviving container restarts. Tool selection that respects context window limits.
This isn’t a theoretical architecture. It’s running on my homelab right now. And the challenges I hit along the way are worth sharing — because you’ll hit them too.
The challenges I hit
1. Config overrides don’t work
Docker MCP Gateway reads server configurations from a catalog (remote YAML). You can override settings in a local config.yaml, but the gateway ignores your overrides when the catalog already has values for those fields. Want to add a volume mount to the paper-search server? The catalog says “no volumes” — but the gateway doesn’t fall back to your config, it just… doesn’t mount anything.
This meant ephemeral containers couldn’t save data to the host. I had to patch the gateway source code.
2. Ephemeral containers, persistent data
MCP servers run as --rm Docker containers. They’re created on demand, destroyed after use. That’s great for isolation but catastrophic for data. The paper-search server downloads PDFs into /app/downloads/ inside the container — and then the container exits and the files vanish.
The fix: volume mounts. But volume mounts require config overrides (see challenge 1), and they also require listing every writable path in a special environment variable with a non-obvious separator syntax.
3. Tool count vs. context window
I started with 12 servers exposing 61 tools. Each tool description is 3-5 lines of text that gets injected into every conversation. At 40-80 tokens per tool, that’s 2,400-4,800 tokens just for tool definitions. On a 16k context model, that’s 15-30% of your context before the user types a single word.
I needed a tier system: core (5 tools), standard (24), extended (37), full (61). And I discovered that the gateway’s tools.yaml filtering is broken when using the --servers flag — another bug to work around.
4. Docker networking quirks
Containers on a bridge network can’t always reach external APIs. Wikipedia returned 429 (rate limit) from inside Docker. DNS resolution works differently. The MCP_GATEWAY_DOCKER_BIND_ALLOW_WRITABLE_PATHS variable uses : as a separator (Linux path convention), not commas — and this is undocumented.
5. No playbook exists
There’s no “how to run MCP servers locally for homelab AI” guide. The Docker MCP Gateway docs assume you’re using Docker Desktop with its GUI. If you’re running Linux with Docker Compose, you’re on your own. The catalog has 300+ servers but no guidance on which ones work locally, which need API keys, and which are actually useful.
What this series covers
This blog series is the guide I wish I’d had. Here’s what we’ll walk through:
Post 2: Patching the Gateway — When open source doesn’t quite work. The config override bug, the tools.yaml filtering bug, the writable paths separator, and how I diagnosed, patched, and tested the fix.
Post 3: Homelab Setup — Full Docker Compose configuration for 12 MCP servers. Volume strategy, per-server config, the tier system, host directory layout, and step-by-step instructions for adding and removing servers.
Post 4: Connecting Your Local LLM — How MCP SSE/streaming works, configuring Ollama and LM Studio, testing end-to-end, and managing context windows for different model sizes.
Post 5: Use Case Deep Dives — Four complete walk-throughs you can replicate:
- Academic research pipeline (papers → knowledge base)
- Web research and consolidation (multi-source with screenshots)
- Document processing (PDF/docx → markdown → knowledge graph)
- Multi-iteration research with task orchestration
Post 6: Lessons Learned and What’s Next — The gotchas, the architectural decisions, and the roadmap for local-first AI agents.
Why local-first matters
I’m not anti-cloud. Cloud APIs are powerful and convenient. But for certain use cases — academic research with paywalled papers, personal knowledge bases, medical data, anything you don’t want leaving your network — local-first is the only option.
And it’s more achievable than you think. A consumer GPU running a 7B or 13B parameter model can handle research, summarization, and tool use. MCP gives that model access to Wikipedia, paper search, web browsing, and file operations. The gateway orchestrates everything. And your data stays on your machine.
The rest of this series shows you exactly how to set it up.
Next: Patching the Gateway — when open source doesn’t quite work