Connecting Your Local LLM to MCP - Blog

Part 4 of the Local-First AI Agents series — putting a real LLM in front of the gateway.

How MCP clients connect

The MCP Gateway exposes an SSE (Server-Sent Events) endpoint that any OpenAI-compatible client can use. The connection flow is:

Local LLM (Ollama/LM Studio/anything)
    │
    │  HTTP POST (OpenAI function calling format)
    ▼
MCP Client (in your app or agent framework)
    │
    │  SSE/Streamable HTTP on port 8811
    ▼
MCP Gateway (Docker container)
    │
    │  Docker API (stdio)
    ▼
MCP Servers (ephemeral containers)

The key insight: your LLM doesn’t speak MCP directly. It speaks OpenAI’s function calling format. A client library translates between the two. This client can be:

A Python script
An agent framework (LangChain, CrewAI, AutoGen)
A custom application
An OpenAI-compatible API wrapper

Option 1: Direct Python client

The simplest approach. A minimal client demonstrates the whole protocol:

import httpx
import json

GATEWAY_URL = "http://localhost:8811/mcp"

def call_tool(server_name, tool_name, arguments):
    client = httpx.Client(timeout=300.0)

    # 1. Initialize session
    r = client.post(GATEWAY_URL, json={
        "jsonrpc": "2.0", "id": "1",
        "method": "initialize",
        "params": {
            "protocolVersion": "2024-11-05",
            "capabilities": {},
            "clientInfo": {"name": "my-client", "version": "1.0"},
        },
    }, headers={
        "Content-Type": "application/json",
        "Accept": "application/json, text/event-stream",
    })
    session_id = r.headers.get("mcp-session-id", "")

    # 2. Call tool
    r = client.post(GATEWAY_URL, json={
        "jsonrpc": "2.0", "id": "2",
        "method": "tools/call",
        "params": {"name": tool_name, "arguments": arguments},
    }, headers={"Mcp-Session-Id": session_id})

    # 3. Parse SSE response
    for line in r.text.split("\n"):
        if line.startswith("data: "):
            data = json.loads(line[6:])
            for content in data.get("result", {}).get("content", []):
                if content.get("type") == "text":
                    return content["text"]
    return None

Usage:

# Search for papers
result = call_tool("paper-search", "search_papers", {
    "query": "knowledge distillation",
    "max_results_per_source": 20
})

# Get current time
result = call_tool("time", "get_current_time", {"timezone": "America/New_York"})

# Take a screenshot
result = call_tool("playwright", "browser_navigate", {
    "url": "https://arxiv.org/abs/2402.13116"
})

Option 2: Ollama with function calling

Ollama supports function calling with models like llama3.1, mistral-nemo, and qwen2.5. Here’s how to connect it to your MCP gateway:

Setup Ollama

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model with function calling support
ollama pull qwen2.5:14b

# Verify it's running
ollama list

Bridge script

Ollama doesn’t speak MCP natively. You need a bridge that:

Gets the tool list from the MCP gateway
Converts it to OpenAI function calling format
Sends it to Ollama with the user’s prompt
If Ollama decides to call a tool, executes the MCP call
Feeds the result back to Ollama

#!/usr/bin/env python3
"""Bridge between Ollama and MCP Gateway."""

import httpx
import json
import sys

OLLAMA_URL = "http://localhost:11434/api/chat"
MCP_URL = "http://localhost:8811/mcp"

def get_mcp_tools():
    """Fetch available tools from MCP gateway and convert to Ollama format."""
    client = httpx.Client(timeout=30)

    # Initialize MCP session
    r = client.post(MCP_URL, json={
        "jsonrpc": "2.0", "id": "1",
        "method": "initialize",
        "params": {
            "protocolVersion": "2024-11-05",
            "capabilities": {},
            "clientInfo": {"name": "ollama-bridge", "version": "1.0"},
        },
    }, headers={"Content-Type": "application/json", "Accept": "application/json, text/event-stream"})

    session_id = r.headers.get("mcp-session-id", "")

    # List tools
    r = client.post(MCP_URL, json={
        "jsonrpc": "2.0", "id": "2",
        "method": "tools/list", "params": {},
    }, headers={"Mcp-Session-Id": session_id})

    tools = []
    for line in r.text.split("\n"):
        if line.startswith("data: "):
            data = json.loads(line[6:])
            for t in data.get("result", {}).get("tools", []):
                tools.append({
                    "type": "function",
                    "function": {
                        "name": t["name"],
                        "description": t.get("description", ""),
                        "parameters": t.get("inputSchema", {
                            "type": "object",
                            "properties": {},
                        }),
                    }
                })

    client.close()
    return tools, session_id

def call_mcp_tool(session_id, tool_name, arguments):
    """Call an MCP tool and return the result."""
    client = httpx.Client(timeout=300)
    r = client.post(MCP_URL, json={
        "jsonrpc": "2.0", "id": "3",
        "method": "tools/call",
        "params": {"name": tool_name, "arguments": arguments},
    }, headers={"Mcp-Session-Id": session_id})

    for line in r.text.split("\n"):
        if line.startswith("data: "):
            data = json.loads(line[6:])
            for c in data.get("result", {}).get("content", []):
                if c.get("type") == "text":
                    return c["text"]
    client.close()
    return "No result"

def chat_with_tools(model, prompt, max_iterations=5):
    """Run a chat loop with Ollama, enabling tool calls through MCP."""
    tools, session_id = get_mcp_tools()

    messages = [{"role": "user", "content": prompt}]

    for i in range(max_iterations):
        print(f"\n--- Iteration {i+1} ---")

        # Call Ollama with tools
        response = httpx.post(OLLAMA_URL, json={
            "model": model,
            "messages": messages,
            "tools": tools,
            "stream": False,
        }, timeout=120)

        result = response.json()
        msg = result.get("message", {})
        messages.append(msg)

        # If model wants to call a tool
        if msg.get("tool_calls"):
            for tc in msg["tool_calls"]:
                fn = tc["function"]
                print(f"  Calling: {fn['name']}({json.dumps(fn['arguments'], indent=2)[:200]})")

                tool_result = call_mcp_tool(session_id, fn["name"], fn["arguments"])
                print(f"  Result: {str(tool_result)[:300]}...")

                messages.append({
                    "role": "tool",
                    "name": fn["name"],
                    "content": str(tool_result),
                })
        else:
            # Model responded with text, we're done
            print(f"\n{msg.get('content', '')}")
            break

if __name__ == "__main__":
    model = sys.argv[1] if len(sys.argv) > 1 else "qwen2.5:14b"
    prompt = " ".join(sys.argv[2:]) if len(sys.argv) > 2 else "What is knowledge distillation?"
    chat_with_tools(model, prompt)

Usage:

python3 ollama_mcp_bridge.py qwen2.5:14b "Search for recent papers on knowledge distillation and summarize the top 3"

Option 3: LM Studio

LM Studio provides a GUI for running local models and supports an OpenAI-compatible server. To connect it to MCP:

Start LM Studio and load a model (e.g., Qwen 2.5 14B)
Enable the local server in LM Studio (default: http://localhost:1234/v1)
Use the same bridge script, changing the URL:

LMSTUDIO_URL = "http://localhost:1234/v1/chat/completions"

The OpenAI-compatible format is the same — the bridge just needs a different URL.

Context window management

This is the critical constraint. Here’s the math:

Model	Context Window	Available for Content	Max Tools (at ~40 tok each)
Llama 3.2 3B	8k	~6,000 tokens	~15 tools
Qwen 2.5 7B	32k	~24,000 tokens	~40 tools
Qwen 2.5 14B	32k	~24,000 tokens	~40 tools
Llama 3.1 70B	128k	~100,000 tokens	All 61 tools

Practical recommendations:

8k models: Use Core tier (5 tools). No browser, no document conversion.
16k models: Use Core or Standard with limited tools. Comment out Wikipedia tools you don’t need.
32k models: Standard tier (24 tools) works well. Research pipelines with paper search + Wikipedia + fetch.
128k+ models: Full tier (61 tools). Everything available.

Reducing tool overhead

The tools.yaml file is your control panel. Even though the filtering bug (post 2) means it doesn’t restrict tools yet, it documents what’s available and will work once the upstream fix lands.

For now, the tier system in --servers is the effective control:

# 8k context — Core only
- --servers=paper-search,fetch,time

# 16k context — Core + essential
- --servers=paper-search,fetch,time,sequentialthinking,wikipedia-mcp

# 32k context — Standard
- --servers=paper-search,fetch,time,sequentialthinking,wikipedia-mcp,duckduckgo,playwright,markdownify

# 128k context — Full
- --servers=paper-search,fetch,time,sequentialthinking,wikipedia-mcp,duckduckgo,playwright,markdownify,markitdown,mcp-hackernews,task-orchestrator,mcp-code-interpreter,memory,sqlite,filesystem

Session strategy

For research workflows, consider a multi-session approach:

Discovery session (Standard tier): Search papers, Wikipedia, web — find what’s relevant
Processing session (Core + markdownify): Convert PDFs to markdown, extract key facts
Analysis session (Core + memory/sqlite): Build knowledge graph, store structured data
Synthesis session (Core + sequentialthinking): Reason over collected data, produce final output

Each session uses only the tools it needs, keeping context lean.

Testing the connection

# 1. Verify gateway is healthy
curl http://localhost:8811/health

# 2. Test with direct MCP call
python3 scripts/mcp_client.py call time get_current_time '{"timezone": "UTC"}'

# 3. Test with Ollama bridge
python3 ollama_mcp_bridge.py qwen2.5:14b "What time is it in Tokyo?"

# 4. Test paper search
python3 ollama_mcp_bridge.py qwen2.5:14b \
  "Search for papers about knowledge distillation and list the top 5 titles"

# 5. Test web search + fetch
python3 ollama_mcp_bridge.py qwen2.5:14b \
  "Search DuckDuckGo for recent advances in model compression"

Common issues

”Connection refused” on port 8811

The gateway isn’t running or hasn’t finished starting. Check:

docker compose logs gateway --tail 20
# Look for "Start streaming server on port 8811"

“Session not found” errors

MCP sessions expire. Each initialize creates a new session. Don’t reuse session IDs across restarts.

Tool calls return empty results

Check the server container logs:

docker logs mcp-gateway 2>&1 | grep "paper-search" | tail -10

Context window exceeded

Reduce --servers to a smaller tier. Remove Wikipedia (22 tools) first — it’s the biggest contributor.

Next: Use Case Deep Dives