Vision Navigation

Fresh

Source: BIBLE.md (vision.mjs section), bible/understand-everything.md (How vision navigation works)

vision.mjs replaces DOM selectors with visual recognition.

How it works

flowchart LR
  P[Page or device] -- screenshot --> C[Capture]
  C --> V[VLM provider: Claude / Ollama / OpenAI]
  V -- "Click the blue Sign In button" --> R[Pixel coordinates]
  R --> A[Act: OS click or ADB tap]

Four steps:

1. Capture. Screenshot the page (via CDP Page.captureScreenshot) or device screen (via adb exec-out screencap -p)
2. Send. Transmit the screenshot plus a natural-language instruction to a VLM: "Find the blue Sign In button"
3. Receive. Get back pixel coordinates: {x: 684, y: 423}
4. Act. Click those coordinates via OS-level input or ADB

Zero DOM interaction for navigation. No document.querySelector, no page.$(selector), no XPath. Detection scripts cannot hook what does not exist.

Three VLM providers

Provider	Cost	Accuracy	Setup
Claude API (default, claude-sonnet-4-6)	~$0.003 per call	Best	Set ANTHROPIC_API_KEY
Ollama on localhost:11434	Free	Lower	Pull llava model
Any OpenAI-compatible endpoint	Varies	Varies	Set custom endpoint

Configure via VISION_PROVIDER env var.

Key functions

Function	What it does
captureScreenshot(page, opts)	CDP screenshot or adbScreenshot(serial) based on opts.backend
visionFind(page, instruction, opts)	Find element, return {x, y, confidence}
visionClick(page, instruction, opts)	Find plus humanized click (desktop) or adbHumanTap (mobile)
visionType(page, instruction, text, opts)	Find input, click/tap, then type
visionVerify(page, description, opts)	Check if page/screen matches description

API routes

Desktop vision routes

# Click the visible element matching a description
curl -s -X POST http://localhost:4700/browser/abc123/vision-click \
  -H "Content-Type: application/json" \
  -d '{"instruction":"Click the blue Sign In button"}'

# Type into the element matching a description
curl -s -X POST http://localhost:4700/browser/abc123/vision-type \
  -H "Content-Type: application/json" \
  -d '{"instruction":"the search box at the top","text":"junk removal dallas","clear":true}'

# Find without clicking
curl -s -X POST http://localhost:4700/browser/abc123/vision-find \
  -H "Content-Type: application/json" \
  -d '{"instruction":"the login button"}'

# Verify the page state
curl -s -X POST http://localhost:4700/browser/abc123/vision-verify \
  -H "Content-Type: application/json" \
  -d '{"description":"a Google search results page with at least 5 results"}'

# Screenshot history
curl http://localhost:4700/browser/abc123/vision-history

Mobile vision routes

# Find on the device screen
curl -s -X POST http://localhost:4700/mobile/mob-abc123/vision-find \
  -H "Content-Type: application/json" \
  -d '{"instruction":"the compose button"}'

# Verify device state
curl -s -X POST http://localhost:4700/mobile/mob-abc123/vision-verify \
  -H "Content-Type: application/json" \
  -d '{"description":"the Reddit home feed"}'

Cost control

Vision is expensive. About $0.003 per VLM call (Claude Sonnet). The router enforces three rules:

1. Only used for navigation decisions in ghost mode. Speed and hybrid modes do not call vision unless explicitly requested
2. Data reads always go through CDP. Getting text, checking URL, extracting content does not need vision
3. History is cached. Repeated screenshots within a short window may be reused

Reliability

Vision is probabilistic, not deterministic. The confidence threshold defaults to 0.5 with 2 retries. Complex UIs with many similar elements may confuse the VLM.

Best for:

• Clear UI elements (buttons, links, input fields)
• Distinct visual landmarks
• Simple, well-laid-out screens

Worse for:

• Dense data tables
• Ambiguous icons
• Dynamic UIs with similar-looking siblings

When vision fails, the vision-fallback plugin can route to alternative strategies. See vision-fallback plugin.

Vision and ghost mode

Vision is the navigation layer that makes ghost mode work. In ghost mode:

• All input events go through OS-level (desktop) or ADB (mobile): isTrusted: true
• All navigation goes through vision: zero CDP traces

This means a ghost-mode session has zero CDP interaction traces. Reddit cannot see CDP. Reddit cannot see DOM queries. Reddit sees a real Chrome window with real input events.

Related

• vision core module
• vision-fallback plugin: CDP to Ender Eyes routing
• Three I/O Modes