gui-agent
GUI automation via visual detection. Clicking, typing, reading content, navigating menus, filling forms — all through screenshot → detect → act workflow. Supports macOS and Linux.
Best use case
gui-agent is best used when you need a repeatable AI agent workflow instead of a one-off prompt.
GUI automation via visual detection. Clicking, typing, reading content, navigating menus, filling forms — all through screenshot → detect → act workflow. Supports macOS and Linux.
Teams using gui-agent should expect a more consistent output, faster repeated execution, less prompt rewriting.
When to use this skill
- You want a reusable workflow that can be run more than once with consistent structure.
When not to use this skill
- You only need a quick one-off answer and do not need a reusable workflow.
- You cannot install or maintain the underlying files, dependencies, or repository context.
Installation
Claude Code / Cursor / Codex
Manual Installation
- Download SKILL.md from GitHub
- Place it in
.claude/skills/gui-claw/SKILL.mdinside your project - Restart your AI agent — it will auto-discover the skill
How gui-agent Compares
| Feature / Agent | gui-agent | Standard Approach |
|---|---|---|
| Platform Support | Not specified | Limited / Varies |
| Context Awareness | High | Baseline |
| Installation Complexity | Unknown | N/A |
Frequently Asked Questions
What does this skill do?
GUI automation via visual detection. Clicking, typing, reading content, navigating menus, filling forms — all through screenshot → detect → act workflow. Supports macOS and Linux.
Where can I find the source code?
You can find the source code on GitHub using the link provided at the top of the page.
Related Guides
Best AI Skills for Claude
Explore the best AI skills for Claude and Claude Code across coding, research, workflow automation, documentation, and agent operations.
ChatGPT vs Claude for Agent Skills
Compare ChatGPT and Claude for AI agent skills across coding, writing, research, and reusable workflow execution.
Top AI Agents for Productivity
See the top AI agent skills for productivity, workflow automation, operational systems, documentation, and everyday task execution.
SKILL.md Source
# GUI Agent
## STEP 0: Activate Platform (MANDATORY FIRST STEP)
Before any GUI operation, run:
```bash
python3 {baseDir}/scripts/activate.py
```
This detects your OS, sets up the correct action commands, and outputs platform context.
After running, `{baseDir}/actions/_actions.yaml` contains your platform's commands.
## Workflow
```
OBSERVE → LEARN → ACT → VERIFY → SAVE
```
1. **OBSERVE** — Take screenshot → run OCR + detector → understand current state
→ `read {baseDir}/skills/gui-observe/SKILL.md`
2. **LEARN** — First time with an app? Save components to memory
→ `read {baseDir}/skills/gui-learn/SKILL.md`
→ `learn_from_screenshot()` auto-outputs app tips if available
3. **ACT** — Pick target → execute using `_actions.yaml` commands → verify
→ `read {baseDir}/skills/gui-act/SKILL.md`
→ `read {baseDir}/actions/_actions.yaml` for available commands
4. **VERIFY** — Screenshot again → confirm action succeeded
5. **SAVE** — Record state transitions to memory
→ `read {baseDir}/skills/gui-memory/SKILL.md` for memory structure
## Core Rules
- **Coordinates from detection only** — OCR or GPA-GUI-Detector, NEVER from guessing
- **Look before you act** — every action must be justified by what you observed
- **image tool = understanding only** — use it to decide WHAT to click, get WHERE from OCR/detector
## Sub-Skills Reference
| Sub-Skill | When to read |
|-----------|-------------|
| `skills/gui-observe/SKILL.md` | Before screenshots or detection |
| `skills/gui-learn/SKILL.md` | Before learning a new app |
| `skills/gui-act/SKILL.md` | Before any click/type action |
| `skills/gui-memory/SKILL.md` | For memory structure details |
| `skills/gui-workflow/SKILL.md` | For multi-step navigation |
| `skills/gui-setup/SKILL.md` | For first-time machine setup |
| `skills/gui-report/SKILL.md` | For task performance reporting |Related Skills
---
name: article-factory-wechat
humanizer
Remove signs of AI-generated writing from text. Use when editing or reviewing text to make it sound more natural and human-written. Based on Wikipedia's comprehensive "Signs of AI writing" guide. Detects and fixes patterns including: inflated symbolism, promotional language, superficial -ing analyses, vague attributions, em dash overuse, rule of three, AI vocabulary words, negative parallelisms, and excessive conjunctive phrases.
find-skills
Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill.
tavily-search
Use Tavily API for real-time web search and content extraction. Use when: user needs real-time web search results, research, or current information from the web. Requires Tavily API key.
baidu-search
Search the web using Baidu AI Search Engine (BDSE). Use for live information, documentation, or research topics.
agent-autonomy-kit
Stop waiting for prompts. Keep working.
Meeting Prep
Never walk into a meeting unprepared again. Your agent researches all attendees before calendar events—pulling LinkedIn profiles, recent company news, mutual connections, and conversation starters. Generates a briefing doc with talking points, icebreakers, and context so you show up informed and confident. Triggered automatically before meetings or on-demand. Configure research depth, advance timing, and output format. Walking into meetings blind is amateur hour—missed connections, generic small talk, zero leverage. Use when setting up meeting intelligence, researching specific attendees, generating pre-meeting briefs, or automating your prep workflow.
self-improvement
Captures learnings, errors, and corrections to enable continuous improvement. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Claude ('No, that's wrong...', 'Actually...'), (3) User requests a capability that doesn't exist, (4) An external API or tool fails, (5) Claude realizes its knowledge is outdated or incorrect, (6) A better approach is discovered for a recurring task. Also review learnings before major tasks.
botlearn-healthcheck
botlearn-healthcheck — BotLearn autonomous health inspector for OpenClaw instances across 5 domains (hardware, config, security, skills, autonomy); triggers on system check, health report, diagnostics, or scheduled heartbeat inspection.
linkedin-cli
A bird-like LinkedIn CLI for searching profiles, checking messages, and summarizing your feed using session cookies.
notebooklm
Google NotebookLM 非官方 Python API 的 OpenClaw Skill。支持内容生成（播客、视频、幻灯片、测验、思维导图等）、文档管理和研究自动化。当用户需要使用 NotebookLM 生成音频概述、视频、学习材料或管理知识库时触发。
小红书长图文发布 Skill
## 概述