← mcp-for-kibana

With MCP, Claude Code did the same Kibana job cheaper and faster than with Direct API, with each of the three models. Median cost per run fell 24% (Sonnet 5.5) to 69% (Haiku 5.5); median time per run fell 32% (Sonnet 5.5) to 65% (Haiku 5.5). All six 95% intervals lie wholly below 1.

These are warm-cache numbers: 593 of 600 runs (99%) started with the prompt already cached. On a cold cache, with one run each, MCP cost more for Sonnet 5.5 and Opus 5.5, and still saved for Haiku 5.5.

Savings per model

Haiku 5.5 saved the most on both measures. With 100 runs per model and setup, the intervals are narrow.

Per model and setup

On Direct API, Haiku 5.5 took the longest of all six combinations; on MCP it was the cheapest by far. All 600 of 600 counted runs passed every check.

Where the time goes exploratory

Not planned before the runs: it suggests where to look, not what holds. On MCP, the time saved is the model's time. A median run spent 25.9 s in model requests instead of 92.7 s with Haiku 5.5, 23.8 s instead of 37.3 s with Sonnet 5.5, and 45.0 s instead of 76.9 s with Opus 5.5. Tool time outside those requests stayed between 5.1 and 8.7 s.

One MCP tool call takes 6 to 9 times as long as one shell command, yet tool time does not grow: the CLI starts a tool while the model's response is still streaming, so on MCP a median 2.1 to 4.6 s of tool time per run runs under the model's requests.

The time saving is not harness overhead: by the CLI's own duration the ratios are 0.32 (Haiku 5.5), 0.64 (Sonnet 5.5) and 0.60 (Opus 5.5), each interval wholly below 1 (time_ratio_cli in study/data/exploratory.json).

Cache: cold against warm exploratory

On MCP, the cached prompt carries the server's tool definitions: a warm run's smallest cache read per request is 18,373 tokens larger than on Direct API. On a warm cache they cost little; a cold run first has to write them to the cache. 593 of 600 runs started warm. The 7 cold ones are the first run of each model and setup, and one more (Sonnet 5.5 on Direct API, started 308 s after the previous run of its model and setup had ended).

Cold against cold, MCP cost Sonnet 5.5 and Opus 5.5 more, and still saved for Haiku 5.5; all three cold runs finished sooner on MCP. One run per model and setup is an observation, not a result.

How it ran

• Design: 100 blocks × 3 models × 2 setups = 600 runs. A block is one run of each model in each setup, in a seeded order, so each block pairs the two setups.
• When: 17 to 18 October 2026, 23:42 to 01:45 UTC, on 6 workers running whole blocks one run at a time.
• The job: one long Kibana job on the web-logs and flights sample data, 6 steps in order, then 27 checks against Kibana:
  1. create a Kibana space and work inside it
  2. create data views on the web-logs and flights indices, each with its time field
  3. build a two-panel web-logs dashboard and a server-errors alert rule
  4. build a two-panel flights dashboard and a cancelled-flights alert rule
  5. change the web-logs dashboard in place: retitle a panel, add a third
  6. read back the dashboard's panel count and the rule's settings
• Held fixed: harness frozen at commit 9d4ec85, mcp-for-kibana 0.4.0 (write tier), Kibana 9.5.5, Claude Code CLI 2.1.302, effort medium. Each run worked in its own Kibana space.
• Measurement: each run sent one root span, efficiency.run, to Honeycomb, with its time, cost, tokens, turns, tool calls, checks and versions. Every number on this page comes from those spans and the CLI's own spans.
• Analysis, fixed before any run: medians per run with 95% percentile-bootstrap intervals (10,000 resamples, seed 20261017). A saving counts only if the pass-rate difference (MCP minus Direct API, bootstrapped over blocks) does not lie wholly below 0.
• Integrity: 600 of 600 planned runs counted, each exactly once, from 601 root spans (1 aborted run, rerun). Recomputed from each run's tokens, the price table reproduced the CLI's cost of all 601 priced runs. One trace per run. Every run recorded the same harness, CLI and Kibana version.
• Spend: $69.45 at list prices for every run.

Limits

• Warm cache. 99% of runs started on a warm prompt cache, so the ratios describe repeated use. A first, cold run can come out the other way, as it did for Sonnet 5.5 and Opus 5.5.
• Success does not separate the setups. All 600 runs passed all 27 checks, so the study cannot say whether either setup does the job better; it compares cost and time at equal success.
• One job, one stack. One task, one Kibana version (9.5.5), one Claude Code CLI version (2.1.302), one server version (0.4.0), one effort level. Other jobs, versions or agents may differ.
• List prices. Cost is computed at Anthropic's published list prices, not what any account was billed.
• Exploratory sections (where the time goes, cache) were not planned before the runs.

Reproduce

The rows, spans, analysis output and the frozen harness are in study/ on this branch. To recompute every number from the rows, with uv:

git clone -b gh-pages https://github.com/pedro-angel/mcp-for-kibana.git mcp-savings-study
cd mcp-savings-study/studies/mcp-savings/study/harness
uv run python mcp_savings_analyze.py analyze --phase study \
  --rows ../data/rows.jsonl --harness-sha 9d4ec85 --out /tmp/mcp-savings
uv run python mcp_savings_explore.py --phase study \
  --rows ../data/rows.jsonl --harness-sha 9d4ec85 --out /tmp/mcp-savings \
  --spans ../data/spans/spans.jsonl \
  --min-reads ../data/cache/min-request-cache-read.json \
  --pilot-rows ../data/cache/pilot-root-spans.json
cmp /tmp/mcp-savings/analysis.json ../data/analysis.json
cmp /tmp/mcp-savings/exploratory.json ../data/exploratory.json
cd ../.. && python3 study/build_site.py   # rebuilds this page

How the rows were pulled from Honeycomb: study/docs/PULL_ROWS.md. What each file is: study/README.md.