Clicking, typing, scrolling, dragging
GitHub has put computer use into public preview in Copilot CLI and the Copilot desktop app on macOS and Windows. The assistant can now read accessible application content and visual context, click controls, enter and edit text, press keys, scroll, drag, and move between applications.
The stated target is not modern software. GitHub says the feature covers “legacy and GUI-only software that do not provide an API, command-line interface, or MCP integration” — the expense system, the procurement portal, the terminal emulator sitting in front of a mainframe. Anything with an API was already automatable. The point of driving a screen is to reach the things that never got one.
The permission trail
Copilot asks for approval before controlling an application, and users can review or reset the applications they have chosen to always allow. On macOS the feature walks the user through granting Accessibility and Screen Recording permissions, which is what reading and clicking another app’s interface requires. Organisations can disable the whole thing through managed settings.
It is turned on explicitly: /computer on in the CLI, or a toggle under Settings in the app. GitHub’s advice is to describe the outcome, the applications involved and the constraints, rather than the clicks.
The same week, from the other side
The timing is the story. Apple published a developer post on Friday saying it will add new controls around Full Disk Access on macOS, and named agent autonomy as the reason: “As AI agents become increasingly capable and autonomous, the risks associated with this level of access will grow substantially.”
Full Disk Access and Screen Recording are not the same permission, but they are adjacent problems. One hands an app every file on the machine; the other lets it watch and operate every window. A desktop agent that is useful needs something in that neighbourhood, and the platform owner has just said the neighbourhood is too permissive.
So developers now have a capability shipping into public preview and a platform signalling it intends to narrow the ground under it. What to watch is whether Apple’s forthcoming controls treat screen access the way it is treating disk access, and whether GitHub’s approval prompts survive contact with users who approve everything once and never look again.
