agentery
$ agentery.com/api/mcpsubmit agent
← Back to search

Market around AI Crawler Index

Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. Ask it to identify a crawler from a raw User-Agent, look one up, list them by category or operator, generate a robots.txt for a

Visit pathwren.workers.dev ↗

Explore the market

Prices and medians update for the tier you select.

Compare at
AI Crawler Index · Pro
No Pro plan
—
Pro-provider median
—
insufficient comparable pricing
vs Pro providers
No comparison
AI Crawler Index doesn't sell Pro
ⓘ Comparisons are same provider type (provider) and same buyer tier (Pro). Never across tiers.
Market density
53rd pctile
87 competing vendors
Comparable listings · 30d
—
MCP servers · Pro median
$39.5/mo
8 priced · $29–$71.5 mid 50%
APIs · Pro median
$53/mo
2 priced · $50–$56 mid 50%
Price spectrum · Pro plans · 8 of 14 providers priced · log scale · $19 → $149/mo
MCP servers APIs◻ shaded = middle 50% · line = median (all types)AI Crawler Index has no comparable Pro price — not plotted

Products in this market

Ranked by how closely each one matches AI Crawler Index's job. Prices show each provider's Pro state; entry prices are labelled as such. Unpriced products still belong to the market.

OOther3Ranked by relevance
AI Crawler Indexthis listing
● OTHER · relevance 100%
no public price
centinel_agent_trust
● OTHER · relevance 78%
no public price
CrawlEyes
● OTHER · relevance 71%
no public price
AAPIs6Ranked by relevance
Crawl Census
● API · relevance 77%
no public price
fastcrw
● API · relevance 75%
$59/mo Pro
+11% vs Pro median
Exa
● API · relevance 72%
no public price
Exa Deep Research API
● API · relevance 72%
no public price
broke2built Data & Intelligence Skills
● API · relevance 72%
no public price
Tavily Search
● API · relevance 71%
no public price
MMCP servers38Ranked by relevance
Crawl Census Client
● MCP · relevance 76%
no public price
WebVector
● MCP · relevance 75%
no public price
SW
swarm_web_crawller
● MCP · relevance 74%
no public price
crawlbase
● MCP · relevance 74%
no Pro plan
entry $99/mo
GEO Tool MCP Server
● MCP · relevance 74%
no public price
mcp_server_webcrawl
● MCP · relevance 74%
no public price
SEO Authority MCP
● MCP · relevance 73%
no public price
SM
smartfetch
● MCP · relevance 73%
no public price
anakin
● MCP · relevance 73%
$19/mo Pro
-52% vs Pro median
string_ai_web_access
● MCP · relevance 73%
no public price
web_intelligence
● MCP · relevance 73%
no public price
wigolo_2026
● MCP · relevance 72%
no public price
crawlie_cloud
● MCP · relevance 72%
$29/mo Pro
-27% vs Pro median
AgentCrush
● MCP · relevance 72%
$29/mo Pro
-27% vs Pro median
SMESH Crawler
● MCP · relevance 72%
no public price
MC
mccauleyloren56_agent_prefligh
● MCP · relevance 72%
no public price
agent_ready_kit
● MCP · relevance 71%
$99 one-time Pro
not monthly-comparable
x1_baas
● MCP · relevance 71%
no public price
BotHub - The Trust Layer for AI Agents
● MCP · relevance 71%
no public price
Neurobird Search
● MCP · relevance 71%
no Pro plan
entry $20/mo
scrapfly_mcp
● MCP · relevance 71%
$30/mo Pro
-24% vs Pro median
tavily_mcp
● MCP · relevance 71%
no public price
agentready
● MCP · relevance 71%
$49/mo Pro
+24% vs Pro median
multi-scraper-mcp
● MCP · relevance 71%
no public price
CR
crw
● MCP · relevance 71%
no Pro plan
entry $10/mo
trirank
● MCP · relevance 71%
$149/mo Pro
+277% vs Pro median
unified-fetch
● MCP · relevance 71%
no public price
mcp_2
● MCP · relevance 71%
no public price
anishdasmail_groundtruth
● MCP · relevance 70%
no public price
threadsnoop_reddit
● MCP · relevance 70%
no public price
web_search
● MCP · relevance 70%
no public price
wigolo
● MCP · relevance 70%
no public price
foura_mcp
● MCP · relevance 70%
$62/mo Pro
+57% vs Pro median
Tool Directory MCP Server
● MCP · relevance 70%
no public price
Web Search MCP
● MCP · relevance 70%
no public price
AEO Scanner
● MCP · relevance 70%
no public price
web_forager
● MCP · relevance 70%
no public price
smithery_acc_google_search_con
● MCP · relevance 70%
no public price
AAgents2Ranked by relevance
Agent Disco
● AGENT · relevance 75%
no public price
CrawlBit
● AGENT · relevance 72%
no Pro plan
entry $390/mo
PPlatforms1Ranked by relevance
Scrapeless
● PLATFORM · relevance 70%
no Pro plan
entry $49/mo

Market = the products most similar to this one by capability; prices are median / quartiles over its priced members, separated by provider type and buyer tier.