/ Verzeichnis / Playground / web-scraper
← Alle Server ↗ GitHub
● Community yfe404 ⚡ Sofort

web-scraper

von yfe404 · yfe404/web-scraper

Phased reconnaissance scraper that prefers APIs > sitemaps > HTML, picks the right Apify template (Cheerio vs Playwright), and only escalates anti-detection when signals warrant.

Intelligent web scraping via Claude Code. Runs a 5-phase reconnaissance (Phases 0–5) that tries the cheapest extraction path first: public APIs, sitemap feeds, structured data. Only when those fail does it consider browser automation, and only when protection signals appear does it layer on stealth. Built around TypeScript-first Apify Actor development, with templates (Cheerio for static, Playwright for JS-heavy).

▶ Demo ansehen 📦 Installieren 💡 Anwendungsfälle

Warum nutzen

Hauptfunktionen

Live-Demo

In der Praxis

web-scraper-skill.replay ▶ bereit
0/0

Installieren

Wählen Sie Ihren Client

~/Library/Application Support/Claude/claude_desktop_config.json  · Windows: %APPDATA%\Claude\claude_desktop_config.json
{
  "mcpServers": {
    "web-scraper-skill": {
      "command": "git",
      "args": [
        "clone",
        "https://github.com/yfe404/web-scraper",
        "~/.claude/skills/web-scraper"
      ],
      "_inferred": true
    }
  }
}

Öffne Claude Desktop → Settings → Developer → Edit Config. Nach dem Speichern neu starten.

~/.cursor/mcp.json · .cursor/mcp.json
{
  "mcpServers": {
    "web-scraper-skill": {
      "command": "git",
      "args": [
        "clone",
        "https://github.com/yfe404/web-scraper",
        "~/.claude/skills/web-scraper"
      ],
      "_inferred": true
    }
  }
}

Cursor nutzt das gleiche mcpServers-Schema wie Claude Desktop. Projektkonfiguration schlägt die globale.

VS Code → Cline → MCP Servers → Edit
{
  "mcpServers": {
    "web-scraper-skill": {
      "command": "git",
      "args": [
        "clone",
        "https://github.com/yfe404/web-scraper",
        "~/.claude/skills/web-scraper"
      ],
      "_inferred": true
    }
  }
}

Klicken Sie auf das MCP-Servers-Symbol in der Cline-Seitenleiste, dann "Edit Configuration".

~/.codeium/windsurf/mcp_config.json
{
  "mcpServers": {
    "web-scraper-skill": {
      "command": "git",
      "args": [
        "clone",
        "https://github.com/yfe404/web-scraper",
        "~/.claude/skills/web-scraper"
      ],
      "_inferred": true
    }
  }
}

Gleiche Struktur wie Claude Desktop. Windsurf neu starten zum Übernehmen.

~/.continue/config.json
{
  "mcpServers": [
    {
      "name": "web-scraper-skill",
      "command": "git",
      "args": [
        "clone",
        "https://github.com/yfe404/web-scraper",
        "~/.claude/skills/web-scraper"
      ]
    }
  ]
}

Continue nutzt ein Array von Serverobjekten statt einer Map.

~/.config/zed/settings.json
{
  "context_servers": {
    "web-scraper-skill": {
      "command": {
        "path": "git",
        "args": [
          "clone",
          "https://github.com/yfe404/web-scraper",
          "~/.claude/skills/web-scraper"
        ]
      }
    }
  }
}

In context_servers hinzufügen. Zed lädt beim Speichern neu.

claude mcp add web-scraper-skill -- git clone https://github.com/yfe404/web-scraper ~/.claude/skills/web-scraper

Einzeiler. Prüfen mit claude mcp list. Entfernen mit claude mcp remove.

Anwendungsfälle

Praxisnahe Nutzung: web-scraper

Scrape a static listing site into a structured dataset

👤 Data engineers pulling public data (directories, price lists, public records) ⏱ ~45 min intermediate

Wann einsetzen: You need a dataset from a public site that doesn't have an API.

Voraussetzungen
  • Skill installed — git clone https://github.com/yfe404/web-scraper ~/.claude/skills/web-scraper
  • Node 20 for Apify Actors — nvm install 20
Ablauf
  1. Let the skill do recon
    Use web-scraper. Target: https://example.com/listings. I want name + URL + category. Recon first — tell me the cheapest extraction path.✓ Kopiert
    → Skill reports: 'sitemap.xml available, use Cheerio'
  2. Scaffold the Apify Actor
    Scaffold a TypeScript Apify Cheerio actor for that extraction.✓ Kopiert
    → Actor tree + main.ts ready to run
  3. Run and iterate
    Run locally on 10 pages; tighten selectors if needed.✓ Kopiert
    → Clean JSON output

Ergebnis: An Apify Actor you can deploy for scheduled scrapes.

Fallstricke
  • Jumping to Playwright when Cheerio would do — Trust the recon — headful browsers 10x the cost unnecessarily
Kombinieren mit: apify · filesystem

Discover and use a site's undocumented JSON API instead of HTML

👤 Scraper devs who want reliability ⏱ ~30 min intermediate

Wann einsetzen: The page is a SPA and HTML is gross, but the XHR calls are clean JSON.

Ablauf
  1. Run API-discovery phase
    Use web-scraper phase 1 — API discovery on https://example.com/app. Enumerate XHR/fetch endpoints.✓ Kopiert
    → List of endpoints with observed payloads
  2. Build the JSON-based actor
    Generate an actor that hits those endpoints directly with auth as needed.✓ Kopiert
    → Lightweight fetch-based actor

Ergebnis: A far more stable scrape than HTML parsing.

Fallstricke
  • Private/session-auth APIs that break when token rotates — Plan token-refresh logic or fall back to browser flow

Kombinationen

Mit anderen MCPs für 10-fache Wirkung

web-scraper-skill + apify

Deploy the scaffolded actor to Apify for scheduled runs

Deploy this actor to my Apify account and schedule it daily.✓ Kopiert
web-scraper-skill + filesystem

Keep the actor code in-repo alongside the consuming app

Scaffold into scrapers/ and commit with the main project.✓ Kopiert

Werkzeuge

Was dieses MCP bereitstellt

WerkzeugEingabenWann aufrufenKosten
recon url Always first 0
scaffold_actor template (cheerio|playwright), target After recon picks template 0
record_session url Debugging dynamic sites 0
run_local actor path, limit Iteration phase 0

Kosten & Limits

Was der Betrieb kostet

API-Kontingent
Apify has its own compute + proxy quotas
Tokens pro Aufruf
Moderate — scaffold and iteration loops
Kosten in €
Free skill; Apify costs separate
Tipp
Prefer Cheerio — one of the cheapest run profiles on Apify

Sicherheit

Rechte, Secrets, Reichweite

Minimale Scopes: Apify API token with actor:read + actor:write
Credential-Speicherung: APIFY_TOKEN in env
Datenabfluss: Whatever sites you target + Apify platform

Fehlerbehebung

Häufige Fehler und Lösungen

Cheerio returns empty selectors

Content is JS-rendered — rerun recon, expect Playwright template

Playwright times out

Bump navigation timeout; consider waiting for specific selector instead of networkidle

403 / bot-block page

Stop and reconsider. This is the legit signal to re-check ToS, not a cue to escalate stealth.

Alternativen

web-scraper vs. andere

AlternativeWann stattdessenKompromiss
Direct Apify consoleYou already know which template you needNo recon phase
firecrawlYou just need markdown of a page, not structured extractionNo actor scaffolding

Mehr

Ressourcen

📖 Offizielle README auf GitHub lesen

🐙 Offene Issues ansehen

🔍 Alle 400+ MCP-Server und Skills durchsuchen