Ghost Engine

Appearance

os-input ​

Fresh

Source: BIBLE.md (os-input.mjs section), bible/understand-everything.md (How OS-level input actually works)

core/os-input.mjs sends real mouse and keyboard events through Windows user32.dll.

How it works ​

flowchart LR
  N[Node.js] --> PS[spawn PowerShell]
  PS --> AT[Add-Type inline C#]
  AT --> PI[P/Invoke user32.dll]
  PI --> SI[SendInput / SetCursorPos]
  SI --> OS[OS input stream]
  OS --> CH[Chrome: isTrusted=true]
  1. Spawn a PowerShell process
  2. Use Add-Type to compile inline C# code
  3. The C# code calls user32.dll SendInput via P/Invoke
  4. All operations for one action (e.g., a full Bezier mouse path) are batched into one PowerShell spawn

Overhead: about 50 ms per action, mostly PowerShell startup. 10x slower than CDP's 5 ms, but events are physically indistinguishable from real hardware input.

Why it matters ​

CDP's Input.dispatchMouseEvent produces isTrusted: false. OS-level SendInput produces isTrusted: true. Advanced detection JS checks event.isTrusted on click and keypress handlers.

Also: zero Input.* domain calls in CDP protocol logs.

Key functions ​

FunctionWhat it does
osMoveMouse(waypoints)Move cursor along path with per-point delays
osClick(x, y, button)move + mouseDown + delay + mouseUp
osTypeText(text, delays)SendInput UNICODE per character
osPressKey(vkCode)virtual key event
osKeyCombo(vkCodes)multi-key combo
osScroll(amount)mouse wheel
osScrollSteps(steps)multi-step scroll
osGetWindowBounds(title)GetWindowRect for Chrome
osFocusWindow(title)SetForegroundWindow
pageToScreen(page, x, y)Convert page-relative to absolute screen coords

Coordinate conversion ​

Puppeteer gives page-relative coordinates. SendInput needs absolute screen coordinates. pageToScreen() handles the conversion, accounting for:

  • Chrome window position (from osGetWindowBounds)
  • Chrome toolbar height (~108 px on Windows at 125% DPI scaling, estimated)
  • Window-vs-screen scaling

Window bounds are cached and re-queried every 5 seconds to handle window moves.

Constraints ​

Window focus required ​

SendInput sends events to the foreground window. Without focus, events go to the wrong window.

This means only one desktop ghost session at a time. phase-runner should serialize desktop ghost work.

Windows only ​

os-input.mjs uses Windows-only APIs (user32.dll). VPS deployment requires a different input backend.

Planned: port to xdotool on Linux. Same interface, different backend.

ADB backend works everywhere ADB is installed.

PowerShell spawn overhead ​

About 50 ms per action for the Add-Type C# compilation. Acceptable for human-speed interaction but noticeable in batch operations.

Future optimization: compile C# helper once to a DLL, or use a persistent PS process.

Integration ​

human-mouse.mjs, human-keyboard.mjs, and human-scroll.mjs accept outputMode: 'os'. The hybrid router sets this flag for ghost and hybrid mode sessions.

javascript
// Ghost mode session
humanMove(page, fromX, fromY, toX, toY, { outputMode: 'os' })
// Routes Bezier path through osMoveMouse instead of CDP