Sync coding-agent knowledge base

2026-09-25 17:24:12 +08:00 · 2026-07-28 01:27:08 -07:00
parent ecdb39c2f8
commit 0ef0273ac4
3 changed files with 268 additions and 4 deletions
@@ -388,8 +388,271 @@ Do not panic or wait indefinitely. Once a native window is suspected, switch to

 ## Linux Specific

-(to be editing...)
+Use the GacUI automation service for all GacUI-owned windows. On wGac, normal
+and hosted applications expose `Controls` and `IO`; the native remote renderer
+exposes `Dom` and renderer-side `IO`. Linux computer use is the fallback for an
+OS-native modal window, not a replacement for those endpoints.
+
+A native portal dialog runs a synchronous nested modal loop while the portal
+backend shows a window from another process. wGac keeps Wayland events, timers,
+and queued MiniHTTP main-thread work advancing during that loop, but the native
+dialog still belongs to the portal process and is absent from GacUI
+`Controls`. Inspect and operate it from a separate process with AT-SPI. Avoid
+driving controls in the underlying GacUI window until the modal call finishes,
+because doing so would re-enter application code.
+
+On GNOME Wayland, wGac's XDG Desktop Portal file chooser is normally exposed
+through AT-SPI by `xdg-desktop-portal-gnome`. The accessible application and
+PID therefore need not match `Test_FullControlTest`. Locate the window by its
+application-supplied title and inspect its descendants. The Full Control Test
+uses `The Title`; do not assume that title for another application.
+Hosted Full Control Test uses `FakeDialogService`, so its dialogs stay in the
+GacUI `Controls` tree and must be operated through `IO`, not AT-SPI.
+
+Install the AT-SPI Python introspection packages if they are unavailable:
+
+```bash
+sudo apt install python3-gi gir1.2-atspi-2.0
+```
+
+For an unattended visual check on Wayland, call the XDG Desktop Portal
+`Screenshot` method with `interactive=false`. This captures the complete
+desktop without opening a rectangle picker or requiring user input:
+
+```bash
+/usr/bin/python3 - "/tmp/gacui-linux-desktop.png" <<'PY'
+import os, sys, uuid, gi
+gi.require_version("Gio", "2.0")
+from gi.repository import Gio, GLib
+
+OUTPUT = os.path.abspath(sys.argv[1])
+bus = Gio.bus_get_sync(Gio.BusType.SESSION, None)
+token = "gacui_" + uuid.uuid4().hex
+sender = bus.get_unique_name()[1:].replace(".", "_")
+request_path = f"/org/freedesktop/portal/desktop/request/{sender}/{token}"
+result = {}
+loop = GLib.MainLoop()
+
+def on_response(connection, sender_name, object_path, interface_name, signal_name, parameters):
+    code, values = parameters.unpack()
+    result["code"] = code
+    result["uri"] = values.get("uri")
+    loop.quit()
+
+subscription = bus.signal_subscribe(
+    "org.freedesktop.portal.Desktop",
+    "org.freedesktop.portal.Request",
+    "Response",
+    request_path,
+    None,
+    Gio.DBusSignalFlags.NONE,
+    on_response,
+)
+returned_path = bus.call_sync(
+    "org.freedesktop.portal.Desktop",
+    "/org/freedesktop/portal/desktop",
+    "org.freedesktop.portal.Screenshot",
+    "Screenshot",
+    GLib.Variant("(sa{sv})", ("", {
+        "handle_token": GLib.Variant("s", token),
+        "interactive": GLib.Variant("b", False),
+    })),
+    GLib.VariantType("(o)"),
+    Gio.DBusCallFlags.NONE,
+    -1,
+    None,
+).unpack()[0]
+if returned_path != request_path:
+    raise SystemExit(f"Unexpected portal request path: {returned_path}")
+timeout = GLib.timeout_add_seconds(15, lambda: (loop.quit(), False)[1])
+loop.run()
+GLib.source_remove(timeout)
+bus.signal_unsubscribe(subscription)
+if result.get("code") != 0 or not result.get("uri"):
+    raise SystemExit(f"Screenshot failed: {result}")
+source = Gio.File.new_for_uri(result["uri"])
+target = Gio.File.new_for_path(OUTPUT)
+source.copy(target, Gio.FileCopyFlags.OVERWRITE, None, None)
+print(OUTPUT)
+PY
+```
+
+Require a zero exit code and inspect the saved PNG. The desktop portal may
+return a temporary source URI, so keep the explicitly copied output file as
+the evidence artifact.
+
+Run inspection from the same logged-in graphical session as the application.
+This helper finds every accessible object with the requested title and prints
+its complete subtree, including action names:
+
+```bash
+/usr/bin/python3 - "The Title" <<'PY'
+import sys, gi
+gi.require_version("Atspi", "2.0")
+from gi.repository import Atspi
+TITLE = sys.argv[1]
+def children(n):
+    try: return [n.get_child_at_index(i) for i in range(max(0, n.get_child_count()))]
+    except Exception: return []
+def walk(n):
+    yield n
+    for c in children(n): yield from walk(c)
+def showing(n):
+    try: return n.get_state_set().contains(Atspi.StateType.SHOWING)
+    except Exception: return False
+def actions(n):
+    try: return [n.get_action_name(i) for i in range(max(0, n.get_n_actions()))]
+    except Exception: return []
+def describe(n, depth=0):
+    try: print(f"{'  '*depth}{n.get_role_name()} name={(n.get_name() or '')!r} actions={actions(n)}")
+    except Exception as e: print(f"{'  '*depth}<unavailable: {e}>")
+    for c in children(n): describe(c, depth+1)
+desktop = Atspi.get_desktop(0)
+found=[]
+for app in children(desktop):
+    for n in walk(app):
+        try:
+            if n.get_role() == Atspi.Role.DIALOG and (n.get_name() or "") == TITLE and showing(n): found.append((app,n))
+        except Exception: pass
+if len(found) != 1: raise SystemExit(f"Expected one visible dialog {TITLE!r}, found {len(found)}")
+app, dialog = found[0]
+print(f"owner={(app.get_name() or '')!r} pid={app.get_process_id()}")
+describe(dialog)
+PY
+```
+
+Read the reported names, roles, hierarchy, and actions before interacting. A
+GNOME portal file chooser exposes `Cancel` and `Select` or an operation-specific
+accept button as push buttons with a `click` action. When the goal is only to
+unblock a test, prefer the least destructive named action, normally `Cancel`.
+
+The following helper clicks one exact push-button name inside one exact titled
+window:
+
+```bash
+/usr/bin/python3 - "The Title" "Cancel" <<'PY'
+import sys, time, gi
+gi.require_version("Atspi", "2.0")
+from gi.repository import Atspi
+TITLE, BUTTON = sys.argv[1:3]
+def children(n):
+    try: return [n.get_child_at_index(i) for i in range(max(0, n.get_child_count()))]
+    except Exception: return []
+def walk(n):
+    yield n
+    for c in children(n): yield from walk(c)
+def showing(n):
+    try: return n.get_state_set().contains(Atspi.StateType.SHOWING)
+    except Exception: return False
+def dialogs():
+    result=[]
+    desktop=Atspi.get_desktop(0)
+    for app in children(desktop):
+        for n in walk(app):
+            try:
+                if n.get_role() == Atspi.Role.DIALOG and (n.get_name() or "") == TITLE and showing(n): result.append(n)
+            except Exception: pass
+    return result
+ds=dialogs()
+if len(ds) != 1: raise SystemExit(f"Expected one visible dialog {TITLE!r}, found {len(ds)}")
+candidates=[]
+for n in walk(ds[0]):
+    try:
+        if n.get_role() == Atspi.Role.PUSH_BUTTON and (n.get_name() or "") == BUTTON and showing(n):
+            clicks=[i for i in range(max(0,n.get_n_actions())) if n.get_action_name(i)=="click"]
+            if len(clicks)==1: candidates.append((n,clicks[0]))
+    except Exception: pass
+if len(candidates) != 1: raise SystemExit(f"Expected one visible clickable {BUTTON!r}, found {len(candidates)}")
+if not candidates[0][0].do_action(candidates[0][1]): raise SystemExit("AT-SPI click rejected")
+deadline=time.monotonic()+5
+while dialogs() and time.monotonic()<deadline: time.sleep(0.1)
+if dialogs(): raise SystemExit("Dialog remained visible after click")
+PY
+```
+
+After the action, require the native dialog to disappear. Then resume the
+GacUI `Controls`/`Dom` and `IO` checks and verify the expected application
+state. If the dialog remains, enumerate the desktop again: a validation error
+or a second modal window may have appeared.
+
+Do not use X11-only tools such as `xdotool` as the primary Wayland procedure.
+Do not guess screen coordinates when semantic AT-SPI actions are available.
+For a different desktop portal backend or a non-portal crash dialog, enumerate
+the complete AT-SPI desktop first and identify the actual owning application,
+title, prompt, and named actions before choosing a response.

 ## macOS Specific

-(to be editing...)
+`CocoaAutomationService` handles GacUI windows in a normal application,
+`CocoaAutomationServiceHosted` handles hosted mode, and
+`CocoaAutomationServiceRenderer` handles a native remote renderer. Use their
+HTTP `Controls`/`Dom` and `IO` endpoints for all GacUI-owned UI. Computer use is
+needed only for OS-native modal dialogs.
+
+Full Control Test without `--hosted` deliberately uses native AppKit dialogs.
+A native modal dialog blocks the application's UI thread, so an automation
+request that needs that thread can time out until the dialog is dismissed.
+Stop polling the GacUI endpoint when this happens and inspect the target process
+from a separate `osascript` process.
+
+The terminal, IDE, or agent host running `osascript` needs Accessibility
+permission under **System Settings > Privacy & Security > Accessibility**.
+Do not work around a permission error by repeatedly polling the blocked
+application. Grant permission to the actual host process, then retry the
+read-only inspection.
+
+First list the target application's top-level windows:
+
+```bash
+osascript -e \
+  'tell application "System Events" to tell process "Test_FullControlTest" to return name of every window'
+```
+
+An `NSOpenPanel` or `NSSavePanel` can be a separate top-level window rather
+than a sheet. Its title can also be the application-supplied dialog title, not
+`Open` or `Save`. Inspect every window and its recursively nested controls:
+
+```bash
+osascript <<'APPLESCRIPT'
+tell application "System Events"
+  tell process "Test_FullControlTest"
+    repeat with targetWindow in windows
+      log "WINDOW: " & (name of targetWindow)
+      try
+        log entire contents of targetWindow
+      end try
+    end repeat
+  end tell
+end tell
+APPLESCRIPT
+```
+
+The resulting object descriptions include the exact hierarchy needed for the
+next command. For example, the Full Control Test open-file panel exposes
+`Cancel` and `Open` under its first splitter group. Prefer the least destructive
+action when the goal is only to unblock the test:
+
+```bash
+osascript -e \
+  'tell application "System Events" to tell process "Test_FullControlTest" to click button "Cancel" of splitter group 1 of window "The Title"'
+```
+
+Do not assume that hierarchy for another macOS version or dialog type. Re-read
+`entire contents`, identify the button by its visible name, and use the exact
+reported containment path. For a message dialog, choose the button after
+reading its prompt. For file, color, and font panels, use the named controls
+reported by Accessibility; canceling is preferable unless the scenario
+requires accepting a specific value.
+
+If a visual check is necessary, take a screenshot without interacting with the
+dialog:
+
+```bash
+screencapture -x /tmp/gacui-native-dialog.png
+```
+
+After dismissing or accepting the native dialog, require the top-level dialog
+window to disappear, then return to the GacUI automation endpoint and verify
+the resulting application state. Hosted Full Control Test uses
+`FakeDialogService`; those dialogs remain inside the GacUI control tree and do
+not require this native-dialog procedure.