Backtesting with an AI agent via MCP

AI coding agents work best when market data is a tool, not a pasted CSV. AymarData’s MCP server exposes catalog and market-data operations so Cursor, Claude Code, and similar clients can list markets, pull books, and scaffold backtests under your API key. This guide shows a safe workflow, example prompts, and REST fallbacks when you need deterministic scripts.

Platform page: MCP. Docs: Using with AI coding tools. Product angle: AI agents.

What MCP gives you

Model Context Protocol lets the host IDE call remote tools with structured arguments. With AymarData connected, an agent can:

• Discover sources and markets
• Fetch point-in-time books and trades
• Read schema / docs hints
• Draft Python that calls the same REST API your production jobs will use

MCP does not replace risk controls. You still review fills logic, fees, and leaky labels before trading. AymarData remains market data — not an order gateway.

MCP vs REST vs bulk

See Bulk and Pricing when agents graduate to large offline jobs.

Setup checklist

1. Create an ad_live_ key in the dashboard (Authentication)
2. Enable the AymarData MCP server in your client (see MCP for current install snippets)
3. Confirm the agent can call a harmless tool (list markets) before large pulls
4. Set rate-limit expectations from Rate limits

Never commit live keys to git. Prefer environment variables or the client’s secret store. If a prompt transcript might be shared, redact Authorization headers and key substrings.

Example agent prompt

Using AymarData MCP (or REST if tools fail):
1) List polymarket markets matching q=btc-updown-15m, limit 10.
2) For the first resolved-looking market, fetch Up book at history_from + 60s, depth 5.
3) Fetch settlement for that market.
4) Write a Python script using requests + Bearer $AYMARDATA_API_KEY that loops 20 windows,
   enters long Up when mid < 0.45 at that sample, and prints resolved counts vs settlement.
5) Do not invent market ids — only use API responses.
6) If you print any P&L / win rate / edge, prefix with: Illustrative output — not a real result.

Good prompts force tool use, forbid hallucinated tickers, and ask for scripts that re-run without the agent.

Stronger prompt patterns

Add constraints the model tends to skip:

Constraints:
- Pass outcome=up explicitly on every Polymarket book call.
- Use ask for buys and bid for sells; never fill at mid.
- Skip windows without history_from or settlement outcome.
- Print skip counts (empty book, HTTP errors) alongside resolved trade counts.
- Label any P&L / win-rate / edge line: Illustrative output — not a real result.
- Emit backtest.py that imports only stdlib + requests.

Ground the agent on public pages: Polymarket data, BTC Up/Down 15m, and the manual backtest guide.

REST fallback the agent should emit

Agents sometimes draft directly against REST — that is desirable for CI:

import os
import requests
from datetime import datetime, timedelta, timezone

API = "https://api.aymardata.com/v1"
H = {"Authorization": f"Bearer {os.environ['AYMARDATA_API_KEY']}"}

def markets(source: str, q: str, limit: int = 20):
    r = requests.get(
        f"{API}/markets",
        headers=H,
        params={"source": source, "q": q, "limit": limit},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["data"]

def book(source: str, market: str, at: str, outcome: str | None = None, depth: int = 5):
    params = {"source": source, "market": market, "at": at, "depth": depth}
    if outcome:
        params["outcome"] = outcome
    r = requests.get(f"{API}/book", headers=H, params=params, timeout=30)
    r.raise_for_status()
    return r.json()

def settlement(source: str, market: str):
    r = requests.get(
        f"{API}/settlement",
        headers=H,
        params={"source": source, "market": market},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

def sample_at(history_from: str, offset_s: int = 60) -> str:
    t = datetime.fromisoformat(history_from.replace("Z", "+00:00"))
    return (t + timedelta(seconds=offset_s)).astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")

print(len(markets("polymarket", "btc-updown-15m")))

cURL sanity check:

export AYMARDATA_API_KEY=ad_live_YOUR_KEY
curl -s "https://api.aymardata.com/v1/markets" \
  -H "Authorization: Bearer $AYMARDATA_API_KEY" \
  -G \
  --data-urlencode "source=polymarket" \
  --data-urlencode "q=btc-updown-15m" \
  --data-urlencode "limit=5"

Multi-source agent workflows

Ask the agent to join venues explicitly:

Fetch binance-spot BTCUSDT mid and polymarket btc-updown-15m Up mid at the same UTC timestamp.
Print both and the difference. Use only tool/API data.

Data hubs for grounding: Polymarket, Binance, Hyperliquid, Kalshi.

Source ids the agent must not invent

Instruct the agent to call /v1/markets (or MCP list tools) before any book pull when status might still be Coming.

Guardrails for backtests written by agents

Human-write acceptance tests: empty book handling, HTTP 429 retries, settlement missing. Prefer comparing agent row counts to a short known window set from the manual Polymarket backtest guide. Label any agent P&L as illustrative.

Interpreting agent-produced metrics

Treat every agent-printed P&L, win rate, or edge figure as Illustrative output — not a real result until you recompute it from live AymarData rows. If a summary looks too good, check: (1) outcome skew in the sample, (2) mid fills, (3) unresolved windows treated as wins, (4) reused at= after resolution. Ask the agent to dump a CSV of per-window rows before trusting summaries.

Pairing with docs and llms.txt

Agents can read /llms.txt and /llms-full.txt for product maps, then deep-link into Docs. Prefer documenting your house style (Polars vs pandas, timezone rules) in the repo so the agent stays consistent. Endpoint references: book, trades, settlement, deltas.

When not to use MCP

• Locked-down CI that only allows outbound REST with pinned scripts
• Ultra-HFT replay where you already have local Parquet from Bulk
• Compliance environments that forbid third-party tool bridges
• Batch jobs that would issue thousands of exploratory tool calls instead of one scripted pull

In those cases, generate code once via MCP locally, then run offline.

Reviewing agent output like a data engineer

Agents optimize for looking done. Your job is to verify the data path. Read the generated script top to bottom and check four things before you run a long loop. First, every Polymarket book call must pass outcome explicitly. Second, entry prices must use ask for buys and bid for sells unless the script documents a passive-fill model. Third, settlement handling must skip missing outcomes instead of coercing them to losses. Fourth, the script must print how many windows were skipped and why.

Next, force a tiny dry run: five markets, one sample each, print raw JSON for one book and one settlement. If the agent summarized P&L without showing rows, ask it to emit CSV and prefix metrics with Illustrative output — not a real result. Compare one window by hand against the manual Polymarket backtest guide and against the live data hub naming. When multi-source joins appear, confirm binance-spot versus binance-perps and Hyperliquid coin ids against /data/binance and /data/hyperliquid.

Prompt retries work better when you paste the failing HTTP status and response body snippet back to the agent with “fix the client, do not invent data.” That keeps the loop grounded in AymarData’s actual endpoints (book, trades, settlement) instead of hallucinated fields. Once the script is stable, stop using MCP for the batch: commit the file, run it under CI or a research runner, and keep MCP for the next exploratory question.

Human gates worth automating later

• Reject runs where skip rate exceeds a threshold you choose (for example 30%).
• Reject runs that never touch settlement.
• Reject runs whose only apparent edge appears before fees and disappears after a one-cent fee sweep (still label any printed metrics as illustrative).
• Require a holdout window list checked into the repo.

Those gates turn agent speed into research discipline without giving up the IDE workflow that MCP enables.

What “done” looks like for an agent-assisted backtest

Declare victory only when three artifacts exist in the repo: a script that runs with a single environment variable for the API key, a short markdown note listing market filters and fee assumptions, and a sample output CSV from a holdout set. MCP transcripts are not artifacts; they are scaffolding. If a teammate cannot reproduce the result without the chat, the work is not finished.

That bar also clarifies when to stop prompting. Once the script passes the dry run and the holdout CSV looks sane, further agent turns tend to churn thresholds and invent features. Switch to deliberate human edits, or ask the agent only for narrow diffs (“add Retry-After handling on 429”) rather than open-ended “make the strategy better.” Better, in backtesting, usually means more honest accounting — not more signals — and honesty is easier to verify in REST code than in a free-form tool loop.

End-to-end mini plan

1. Agent lists markets for btc-updown-15m
2. Agent writes backtest.py using REST helpers above
3. You run it with a Builder/Pro key from Pricing
4. You compare results to the manual guide How to backtest Polymarket Up/Down
5. You promote the script to your research repo; keep MCP for exploration
6. Optionally graduate path-dependent logic to deltas (deltas vs snapshots)

FAQ

Does MCP consume API quota?
Yes — tool calls hit the same backend limits as REST.

Can the agent place orders?
AymarData is market data, not a trading gateway. Execution stays on your venue accounts.

Where is OpenAPI?
See platform/docs links from API and Quickstart.

What if MCP tools fail mid-session?
Fall back to the REST helpers in this guide; do not let the agent invent JSON books. Re-check authentication and rate limits.