feat!: migrate MCP server to ScrapeGraph API v2#16
Open
VinciGit00 wants to merge 3 commits intomainfrom
Open
Conversation
Align with scrapegraph-py PR #82: base URL /api/v2, Bearer + SGAI-APIKEY headers, X-SDK-Version, and v2 endpoints for scrape, extract, search, crawl, credits, history, and monitor. BREAKING CHANGE: Removes sitemap, agentic_scrapper, markdownify_status, and smartscraper_status. Crawl supports markdown/html only (no AI crawl mode). Adds crawl_stop, crawl_resume, credits, sgai_history, and monitor_* tools. Optional SCRAPEGRAPH_API_BASE_URL for custom API hosts. Made-with: Cursor
…Config Align MCP server with scrapegraph-py PR #82 proxy changes: - Replace stealth/render_heavy_js/render_js booleans with mode enum (auto, fast, js, direct+stealth, js+stealth) - Rename wait_ms → wait, add timeout, cookies, country params - Add LlmConfig support (llm_model, llm_temperature, llm_max_tokens) to smartscraper, searchscraper, and monitor_create - Add fetch_config params to smartcrawler_initiate and monitor_create - Add output_schema support to searchscraper - Remove deprecated v1 params (website_html, website_markdown, stream, total_pages, same_domain_only, time_range, number_of_scrolls) - Update parameter reference documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove llm_model, llm_temperature, llm_max_tokens from smartscraper, searchscraper, and monitor_create tools. Remove _llm_config helper and llm_config_dict from all client methods. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Member
Author
✅ Integration Test Results — localhost:3002Tested all three SDKs/tools against the local dev API ( scrapegraph-py v2 (PR #82 — built from
|
| Test | Endpoint | Result |
|---|---|---|
client.credits() |
GET /credits | ✅ {remaining: 248935, plan: "Pro Plan"} |
client.scrape(url, format="markdown") |
POST /scrape | ✅ Returns markdown content |
client.extract(url, prompt=...) |
POST /extract | ✅ Returns structured JSON |
client.search(query, num_results=3) |
POST /search | ✅ Returns search results |
client.history(limit=3) |
GET /history | ✅ Returns request history |
scrapegraph-mcp v2 (this PR — ScapeGraphClient)
| Test | Endpoint | Result |
|---|---|---|
credits() |
GET /credits | ✅ |
scrape_v2(url, "markdown") |
POST /scrape | ✅ |
scrape_v2(url, "html") |
POST /scrape | ✅ |
scrape_v2(url, "screenshot") |
POST /scrape | ✅ |
scrape_v2(url, "branding") |
POST /scrape | ✅ |
extract(prompt, url) |
POST /extract | ✅ |
search_api(query, num_results=3) |
POST /search | ✅ |
history(limit=3) |
GET /history | ✅ |
crawl_start(url, depth=1) |
POST /crawl | ✅ |
monitor list |
GET /monitor | ✅ |
just-scrape CLI (scrapegraph-cli + scrapegraph-js v2)
| Command | Result |
|---|---|
just-scrape credits --json |
✅ |
just-scrape scrape https://example.com --json |
✅ |
just-scrape extract https://example.com -p "..." --json |
✅ |
just-scrape search "ScrapeGraph AI" --json |
✅ |
just-scrape history scrape --json |
✅ |
All three client implementations (Python SDK, MCP server, JS CLI) are aligned and working against the v2 API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the ScrapeGraph MCP server to API v2, matching scrapegraph-py#82.
Changes
https://api.scrapegraphai.com/api/v2(override withSCRAPEGRAPH_API_BASE_URL)Authorization: Bearer,SGAI-APIKEY,X-SDK-Version: scrapegraph-mcp@2.0.0/scrape,/extract,/search,/crawl(+ stop/resume),/credits,/history,/monitor(+ lifecycle)crawl_stop,crawl_resume,credits,sgai_history,monitor_create,monitor_list,monitor_get,monitor_pause,monitor_resume,monitor_deletemarkdownify,smartscraper,searchscraper,scrape(format options),smartcrawler_initiate(markdown/html crawl; defaultextraction_mode=markdown)sitemap,agentic_scrapper,markdownify_status,smartscraper_statusBreaking changes
Clients relying on removed tools or v1-only behavior must migrate per the README and
.agentdocs.Test plan
ruff check src/andmypy src/scrapegraph_mcp/server.pySGAI_API_KEYand smoke-testmarkdownify,credits(if deployed on target API)Made with Cursor