web-eval-agent MCP — Casos de uso, Instalar & Demo ao vivo

Por que usar

Principais recursos

Describe tests in plain English, not selectors and asserts
Captures screenshots, console errors, network requests
setup_browser_state persists logins so tests can run behind auth
Works in Cursor, Claude, and similar coding agents

Demo ao vivo

Como fica na prática

web-eval-agent.replay ▶ pronto

0/0

Instalar

Escolha seu cliente

~/Library/Application Support/Claude/claude_desktop_config.json · Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Abra Claude Desktop → Settings → Developer → Edit Config. Reinicie após salvar.

~/.cursor/mcp.json · .cursor/mcp.json

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Cursor usa o mesmo esquema mcpServers que o Claude Desktop. Config de projeto vence a global.

VS Code → Cline → MCP Servers → Edit

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Clique no ícone MCP Servers na barra lateral do Cline, depois "Edit Configuration".

~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ],
      "_inferred": true
    }
  }
}

Mesmo formato do Claude Desktop. Reinicie o Windsurf para aplicar.

~/.continue/config.json

{
  "mcpServers": [
    {
      "name": "web-eval-agent",
      "command": "uvx",
      "args": [
        "web-eval-agent"
      ]
    }
  ]
}

O Continue usa um array de objetos de servidor em vez de um map.

~/.config/zed/settings.json

{
  "context_servers": {
    "web-eval-agent": {
      "command": {
        "path": "uvx",
        "args": [
          "web-eval-agent"
        ]
      }
    }
  }
}

Adicione em context_servers. Zed recarrega automaticamente ao salvar.

claude mcp add web-eval-agent -- uvx web-eval-agent

Uma linha só. Verifique com claude mcp list. Remova com claude mcp remove.

Casos de uso

Usos do mundo real: web-eval-agent

Smoke-test a web deploy with web-eval-agent

👤 Solo devs, small teams without a Playwright suite ⏱ ~10 min beginner

Quando usar: You deployed and want a quick 'did I break anything obvious' check.

Pré-requisitos

Free API key from operative.sh/mcp — Sign up, copy the key
Playwright deps — npx playwright install (the MCP will prompt if missing)

Fluxo

Describe the test

On staging.example.com, verify I can: sign up with a new email, create a project, log out. Report what fails.✓ Copiado

→ Pass/fail with screenshots
Dig into failures

For the failed step, show the console errors and the network request that returned 500.✓ Copiado

→ Stack-level evidence

Resultado: Post-deploy confidence in 2 minutes.

Armadilhas

Test accounts clutter your prod DB — Always run against staging; if prod, use a dedicated QA account and clean up

Combine com: sentry

Exploratory UX evaluation of a new flow

👤 Designers, PMs ⏱ ~20 min intermediate

Quando usar: You want an outside perspective on a flow without scheduling user tests.

Fluxo

Describe user intent, not steps

As a first-time user, try to share a project with a colleague. Note every friction point.✓ Copiado

→ Free-form UX critique with screenshots of each confusion
Contrast with the happy path

Now do the same flow as a power user who knows the UI. How much faster? What confused the noob but not the expert?✓ Copiado

→ Comparative friction map

Resultado: Cheap UX heuristics before putting real users in front of it.

Test features behind login with persisted browser state

👤 Anyone testing authenticated flows ⏱ ~15 min intermediate

Quando usar: Your feature requires login; you don't want the agent handling your password.

Fluxo

Seed the session

Call setup_browser_state opening https://app.example.com/login — I'll sign in myself.✓ Copiado

→ Interactive browser opens; you log in; session saved
Run the test using the saved state

Test the billing settings page: load it, verify the current plan shows, try downgrading.✓ Copiado

→ Test runs with your authenticated session

Resultado: Authed testing without sharing credentials with the agent.

Combinações

Combine com outros MCPs para 10× de alavancagem

web-eval-agent + sentry

Run an eval, any new errors go to Sentry for post-hoc review

Run the signup eval, then check Sentry for new error events captured during that window.✓ Copiado

web-eval-agent + playwright

Prototype with web-eval-agent, harden into Playwright for CI

Convert the working web-eval-agent test into a Playwright spec I can run in CI.✓ Copiado

Ferramentas

O que este MCP expõe

Ferramenta	Entradas	Quando chamar	Custo
web_eval_agent	url: str, task: str, headless_browser?: bool	Any natural-language web test	LLM calls + browser time
setup_browser_state	url?: str	Once per service, to persist logged-in state	0

Custo e limites

O que custa rodar

Cota de API: Free tier from operative.sh
Tokens por chamada: A full eval can be 5-20k tokens (screenshots described)
Monetário: Free for low volume
Dica: For repetitive tests, graduate them to Playwright; use web-eval-agent for exploration

Segurança

Permissões, segredos, alcance

Armazenamento de credenciais: operative.sh API key in env; browser state saved locally

Saída de dados: Target sites + operative.sh for eval orchestration

Original project is archived/discontinued; team now at withrefresh.com. Still functional but no new features expected.

Solução de problemas

Erros comuns e correções

Browser fails to launch

Install Playwright deps: npx playwright install-deps

Session keeps expiring

Some sites rotate cookies; re-run setup_browser_state. Or use Playwright's storageState for finer control

Agent misunderstands the task

Be specific: URLs, selectors or text to look for, expected outcomes

Alternativas

web-eval-agent vs. outros

Alternativa	Quando usar	Troca
Playwright MCP	You want scriptable, reproducible tests	You write the code
Browserbase MCP	You need cloud-hosted browsers for CI	Paid per minute

Mais

Recursos

📖 Leia o README oficial no GitHub

🐙 Ver issues abertas

🔍 Ver todos os 400+ servidores MCP e Skills