Automating an app with no API: why the control tree beats coordinates
Most screen automation breaks the first time a window moves. The difference between a fragile script and a durable one is how it finds what to click.
By the Onscryn team · November 1, 2026 · 9 min read
A great deal of valuable software has no API. The only integration point is the screen a person uses, so the only way to automate it is to do what a person does: click, type, wait, read. The idea is old. What separates automation that lasts for years from automation that breaks on Tuesday is one question: how does it find the thing to click?
Three ways to point at a button
The first way is by coordinates: click at x 102, y 101. It is the simplest thing that can possibly work and the first to fail. Move the window, resize it, change the display scaling or open the program on a different monitor and every click lands somewhere else. Coordinates encode where something was, once, on one machine.
The second way is by appearance: find a picture of the button and click where it matches. This is more forgiving, since position no longer matters. But it assumes the button always looks the same. A new theme, a different font size, a high-density display or a small redesign in a vendor update and the match fails, or worse, matches something else.
The third way is by identity. Windows applications expose a structured description of their controls to the system: a tree of windows, panels, buttons and fields, each with a name and a type. Automation that reads that tree can say 'the button named Save' and find it wherever it is, however it is drawn.
Why the tree wins
It is not magic. It is simply that the tree describes the program in the program's own terms. A button's name rarely changes when the window does. Its position changes constantly. Pointing at the stable thing instead of the unstable one is the whole trick.
It also makes the script readable. 'Click Save in the Invoice window' is something a colleague can review. 'Click at 102, 101' is a riddle.
Where it still goes wrong
A durable selector does not make a script durable. Applications are slow, dialogs appear unannounced and a screen takes a moment to load. Three habits matter as much as the way you point:
- Wait for the thing you expect, not for a fixed number of seconds.
- Assert that it worked: that the record saved, the value appeared, the dialog closed.
- Decide what happens on the unexpected: stop, retry or carry on, and say so in the script.
Some programs draw their own custom controls that do not appear in the tree. For those you may need a fallback, such as an image match, and you should expect to maintain them. Knowing which of your screens are which is part of scoping the work.
Describe the goal, not the clicks
Once automation can read the tree, a new option opens. Instead of someone writing each step, an assistant can read the same description, plan the actions and carry them out, with every step visible and recorded. It is still early, and it deserves to be treated as something to supervise rather than something to leave alone, but it is a direct consequence of having a structured view of the screen.
If you are about to automate a legacy screen, spend ten minutes inspecting what the program exposes before you write a single step. It will tell you how long the automation will last.