Input, the UI tree and building for automation
8 min read
Two things decide whether an app feels right in a browser and whether it can be automated: how input gets into it, and how well its controls describe themselves. The SDK handles most of both for you. This page covers what it does and what to do to help it.
Input is delivered for you
By default the SDK posts browser mouse and keyboard events straight to the window you passed to Onscryn_AcquireSurface, as real window messages. Clicks become button messages at the right position, and typing becomes character messages. A linked app gets working input with no extra code.
Pointer positions arrive as fractions of the window, from zero to one. If your surface covers more than the main window, set its screen rectangle so they map correctly.
When you want the input yourself
Set the disable flag for input injection in the options and handle the input callback instead. Each event carries a type such as mousedown or keydown, the pointer position, the button, the key code from the browser, wheel movement, and for the clipboard the text and whether to paste.
The input callback still fires when automatic delivery is on, so you can watch events, handle hotkeys or window commands, and leave the rest to the SDK.
The UI tree
About three times a second the SDK reads your app's controls through UI Automation and publishes a tree of them: each node has a role such as button, textbox or list, its id, its text, its bounds in surface pixels, and whether it is enabled. Unchanged trees are not sent again.
That tree is what lets the browser offer selectable text and search over the picture, and what the workflow builder and the Copilot use to find a button or a field by name instead of by position. Opt out with the disable flag if you would rather send your own.
Design for automation
These rules come from the reference app, which was built to be automated. They are cheap to follow and they decide whether a workflow is a five-step success or a fragile guess.
- Give every control a stable name. In WinForms that is the Name property, in WPF the AutomationId. Workflows refer to controls by that id, so rename it and every workflow that uses it breaks.
- Show one screen at a time. Hidden controls drop out of the tree, so ids on other screens never clash with the visible ones.
- Prefer radio buttons and list rows to combo boxes. A drop-down opens in a separate window that the tree does not walk.
- When a search leaves exactly one result, select it. Then a workflow can type an id and press the action button without having to click a row.
- Write every outcome to a status label that has its own name. Workflows can wait for it or branch on its text.
- Keep data resettable. A reset action or button that restores seed data lets tests and demos start from the same place every time.
- Do not rely on owner-drawn controls for anything a workflow needs to read. Custom-painted controls expose nothing to UI Automation unless you implement it.
Resize and viewer information
When the browser window changes size, the resize callback gives you the new width and height so you can resize your window to fit. Once per session the browser info callback delivers a JSON object with the viewer's screen size, orientation, browser, pixel ratio and language, which you can use to pick a layout or a locale.