vision

Fresh

Source: BIBLE.md (vision.mjs section)

core/vision.mjs takes a screenshot from a page (desktop CDP) or device (ADB screencap), sends it to a VLM with a natural language instruction, and gets back pixel coordinates for the requested element.

Key functions

Function	What
captureScreenshot(page, opts)	CDP screenshot or adbScreenshot(serial) based on opts.backend
visionFind(page, instruction, opts)	Find element, return {x, y, confidence}
visionClick(page, instruction, opts)	Find plus humanized click (desktop) or adbHumanTap (mobile)
visionType(page, instruction, text, opts)	Find input, click/tap, type
visionVerify(page, description, opts)	Check if page/screen matches description

Providers

Three VLM providers. Set via VISION_PROVIDER env var.

Provider	Cost	Accuracy	Setup
claude (default, claude-sonnet-4-6)	~$0.003/call	Best	ANTHROPIC_API_KEY
ollama (local)	Free	Lower	Pull llava, set OLLAMA_URL
openai (or compatible)	Varies	Varies	OPENAI_API_KEY, custom endpoint

Cost control

Vision calls are about $0.003 each (Claude Sonnet). Used only for navigation decisions in ghost mode.

Data reads (get text, check URL, extract content) always go through CDP regardless of mode.

Confidence threshold

Default confidence: 0.5 with 2 retries.

If a visionFind returns confidence below threshold, retry with a different prompt phrasing or escalate to a fallback (see vision-fallback plugin).

Backend selection

// Desktop CDP screenshot
captureScreenshot(page, { backend: 'cdp' })

// Mobile ADB screencap
captureScreenshot(page, { backend: 'adb', serial: '192.168.1.100:5555' })

Example

# Desktop ghost mode click
curl -X POST http://localhost:4700/browser/abc123/vision-click \
  -H "Content-Type: application/json" \
  -d '{"instruction":"Click the blue Sign In button at the top right"}'

# Returns:
# { "success": true, "x": 1234, "y": 56, "confidence": 0.92 }

Related

• Vision Navigation concept
• vision-fallback plugin
• Ender Eyes scripts: 11 visual agent scripts