web-eval-agent MCP — Cas d'usage, Installer & Démo en direct

Pourquoi l'utiliser

Fonctionnalités clés

Describe tests in plain English, not selectors and asserts
Captures screenshots, console errors, network requests
setup_browser_state persists logins so tests can run behind auth
Works in Cursor, Claude, and similar coding agents

Démo en direct

Aperçu en pratique

web-eval-agent.replay ▶ prêt

0/0

Installer

Choisissez votre client

~/Library/Application Support/Claude/claude_desktop_config.json · Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Ouvrez Claude Desktop → Settings → Developer → Edit Config. Redémarrez après avoir enregistré.

~/.cursor/mcp.json · .cursor/mcp.json

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Cursor utilise le même schéma mcpServers que Claude Desktop. La config projet l'emporte sur la globale.

VS Code → Cline → MCP Servers → Edit

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Cliquez sur l'icône MCP Servers dans la barre latérale Cline, puis "Edit Configuration".

~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Même format que Claude Desktop. Redémarrez Windsurf pour appliquer.

~/.continue/config.json

{
  "mcpServers": [
    {
      "name": "web-eval-agent",
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ]
    }
  ]
}

Continue utilise un tableau d'objets serveur plutôt qu'une map.

~/.config/zed/settings.json

{
  "context_servers": {
    "web-eval-agent": {
      "command": {
        "path": "uvx",
        "args": [
          "web-eval-agent"
        ]
      }
    }
  }
}

Ajoutez dans context_servers. Zed recharge à chaud à la sauvegarde.

claude mcp add web-eval-agent -- uvx web-eval-agent

Une seule ligne. Vérifiez avec claude mcp list. Supprimez avec claude mcp remove.

Cas d'usage

Usages concrets : web-eval-agent

Smoke-test a web deploy with web-eval-agent

👤 Solo devs, small teams without a Playwright suite ⏱ ~10 min beginner

Quand l'utiliser : You deployed and want a quick 'did I break anything obvious' check.

Prérequis

Free API key from operative.sh/mcp — Sign up, copy the key
Playwright deps — npx playwright install (the MCP will prompt if missing)

Déroulement

Describe the test

On staging.example.com, verify I can: sign up with a new email, create a project, log out. Report what fails.✓ Copié

→ Pass/fail with screenshots
Dig into failures

For the failed step, show the console errors and the network request that returned 500.✓ Copié

→ Stack-level evidence

Résultat : Post-deploy confidence in 2 minutes.

Pièges

Test accounts clutter your prod DB — Always run against staging; if prod, use a dedicated QA account and clean up

Combiner avec : sentry

Exploratory UX evaluation of a new flow

👤 Designers, PMs ⏱ ~20 min intermediate

Quand l'utiliser : You want an outside perspective on a flow without scheduling user tests.

Déroulement

Describe user intent, not steps

As a first-time user, try to share a project with a colleague. Note every friction point.✓ Copié

→ Free-form UX critique with screenshots of each confusion
Contrast with the happy path

Now do the same flow as a power user who knows the UI. How much faster? What confused the noob but not the expert?✓ Copié

→ Comparative friction map

Résultat : Cheap UX heuristics before putting real users in front of it.

Test features behind login with persisted browser state

👤 Anyone testing authenticated flows ⏱ ~15 min intermediate

Quand l'utiliser : Your feature requires login; you don't want the agent handling your password.

Déroulement

Seed the session

Call setup_browser_state opening https://app.example.com/login — I'll sign in myself.✓ Copié

→ Interactive browser opens; you log in; session saved
Run the test using the saved state

Test the billing settings page: load it, verify the current plan shows, try downgrading.✓ Copié

→ Test runs with your authenticated session

Résultat : Authed testing without sharing credentials with the agent.

Combinaisons

Associez-le à d'autres MCPs pour un effet X10

web-eval-agent + sentry

Run an eval, any new errors go to Sentry for post-hoc review

Run the signup eval, then check Sentry for new error events captured during that window.✓ Copié

web-eval-agent + playwright

Prototype with web-eval-agent, harden into Playwright for CI

Convert the working web-eval-agent test into a Playwright spec I can run in CI.✓ Copié

Outils

Ce que ce MCP expose

Outil	Entrées	Quand appeler	Coût
web_eval_agent	url: str, task: str, headless_browser?: bool	Any natural-language web test	LLM calls + browser time
setup_browser_state	url?: str	Once per service, to persist logged-in state	0

Coût et limites

Coût d'exécution

Quota d'API: Free tier from operative.sh
Tokens par appel: A full eval can be 5-20k tokens (screenshots described)
Monétaire: Free for low volume
Astuce: For repetitive tests, graduate them to Playwright; use web-eval-agent for exploration

Sécurité

Permissions, secrets, portée

Stockage des identifiants : operative.sh API key in env; browser state saved locally

Sortie de données : Target sites + operative.sh for eval orchestration

Original project is archived/discontinued; team now at withrefresh.com. Still functional but no new features expected.

Dépannage

Erreurs courantes et correctifs

Browser fails to launch

Install Playwright deps: npx playwright install-deps

Session keeps expiring

Some sites rotate cookies; re-run setup_browser_state. Or use Playwright's storageState for finer control

Agent misunderstands the task

Be specific: URLs, selectors or text to look for, expected outcomes

Alternatives

web-eval-agent vs autres

Alternative	Quand l'utiliser	Compromis
Playwright MCP	You want scriptable, reproducible tests	You write the code
Browserbase MCP	You need cloud-hosted browsers for CI	Paid per minute

Plus

Ressources

📖 Lire le README officiel sur GitHub

🐙 Voir les issues ouvertes

🔍 Parcourir les 400+ serveurs MCP et Skills