` buttons with onclick, dropdowns inside React portals, autocomplete options, form inputs lacking role/label — none show up in the snapshot. Agents flying blind either guess selectors from memory or give up. ## The core rule **Selectors persisted to main.py must be DETERMINISTIC across future runs.** A deterministic selector resolves to the same element on every replay — across browser restarts, page rebuilds, deploys that rename auto-generated classes (`css-8xy3zb`), React key changes, re-hydration. Anything that depends on session state (refs) or on unstable DOM shape (nth-child chains, auto-gen class names) is NOT deterministic and will silently match the wrong element tomorrow. Refs (`@e1`, `e68`, `"ref": "abc123"`) are session-local identifiers the browser tools assign per snapshot. They are valid only for the immediate next tool call in the same session. **Any ref hardcoded into main.py is a bug.** Two equally valid paths to a deterministic selector: - **Path A — snapshot + act.** `browser_snapshot` gives you role + accessible name + widget state. Pick a locator from the priority list, then act via individual tool calls (`browser_click`, `browser_type`, `browser_select`, `browser_navigate`) OR via Playwright code (`browser_run_code` with `page.getByRole(...)`, `page.locator(...)`). Tool-call style is a debugability preference, not a durability dimension. - **Path B — DOM probe via eval.** Run the canonical read-only probe below via `browser_evaluate` (Playwright) or `agent-browser eval` to get a structured inventory. Use when the a11y snapshot misses elements (custom `

` buttons, portal/popover children, form inputs the tree skips). Both paths terminate in a durable locator expressed in the persisted script. `browser_run_code` using Playwright's locator API (`page.getByRole('button', { name: 'Continue' }).click()`) is durable and is the right shape for chained multi-step interactions. `browser_evaluate` for inspection is allowed; `browser_evaluate` that hand-rolls `document.querySelector` for an action and then writes that selector into the script is fine IF the selector is in the durability tier below — avoid structural CSS chains that break on DOM rearrangement. ## Durable-selector priority When writing locators, agents must pick the highest-priority hook that uniquely identifies the element: 1. **`data-testid`** / `data-test` / `data-cy` / `data-qa` — ideal, rare on production sites. 2. **Hand-written semantic `id` or `name`** — e.g. `#panAdhaarUserId`, `#loginPasswordField`. **Skip auto-generated ids**: `radix-_rN_`, `mat-mdc-*`, React `:rNN:`, UUID-shaped `8-4-4-4-12` — all rotate across rebuilds. 3. **`aria-label`** — very durable when present. 4. **Role + accessible name** — `page.get_by_role("button", name="Sign in")`. 5. **`get_by_label(...)`** / `get_by_placeholder(...)` / `get_by_text(...)`. 6. **Structural CSS / XPath with nth-child chains** — last resort; flag in learnings. This priority mirrors Playwright's own recommendation but is explicitly enforced in prompts so code-gen doesn't default to fragile nth-child paths. ## The DOM probe When the a11y snapshot is insufficient (custom divs, portal dropdowns, missing form inputs), the agent runs a **read-only DOM probe** via `browser_evaluate`. The probe is a canonical JS snippet embedded in the prompt — agents copy it verbatim so results stay comparable across runs. ### What the probe returns ```jsonc { "url": "https://example.com/page", "framework": "angular-material" | "radix" | "headlessui" | "react" | "unknown", "stableHookInventory": { "data-testid": 0, "aria-label": 38, "id": 80, "name": 5, "role": 73, ... }, "popoverItems": [ // Visible children of floating/portal containers, auto-detected { "source": "popover", "tag": "div", "text": "Option A", "attrs": { "role": "option", "aria-label": "..." }, "cssPath": "[aria-label=\"Option A\"]" } ], "actionableItems": [ // Cursor-pointer / onclick / interactive-role elements body-wide { "source": "actionable", "tag": "button", "text": "Continue", "attrs": { "aria-label": "Continue", "type": "submit" }, "cssPath": "[aria-label=\"Continue\"]" } ], "counts": { "popover": 3, "actionable": 42 } } ``` ### Why each field matters - **`stableHookInventory`** tells the agent which hook strategy applies site-wide: "38 aria-labels, 0 testids → use aria-label + role+name across this workflow". Record this once per site in learnings. - **`framework`** drives known-bad-id filters (Radix → skip `_rN_`, Angular Material → skip `mat-mdc-*`). - **`popoverItems`** captures floating/portal content (React portals, Radix Popover, Headless UI menus) that the a11y tree often misses. - **`actionableItems`** catches custom `

` buttons and form inputs the a11y tree skips because the tag itself implies interactive. - **`cssPath`** is pre-filtered against auto-generated id patterns. If non-null, use it directly in main.py. If null, fall back to `role+name` from the a11y snapshot. ### When to run the probe - After first navigation to a new site (for the stable-hook inventory). - When a specific element isn't in `browser_snapshot` output but should be clickable. - When debugging a selector failure — the probe's current DOM view is ground truth. ### When NOT to run the probe - As a substitute for `browser_snapshot`. The snapshot is faster, lighter, and carries widget state (disabled/expanded). Probe the DOM only when the snapshot is insufficient. - For actions. The probe is strictly read-only. Writes go through `browser_click`, `browser_type`, `browser_select`, `browser_navigate`. ### Multi-backend invocation The probe JS body is identical across backends. Only the wrapper differs: | Backend | Invocation | |---|---| | Playwright MCP | `call_mcp('playwright', 'browser_evaluate', {'function': ''})` | | agent-browser CLI | `agent-browser eval ""` — returns JSON on stdout; pipe to a file if large | ## Shared CDP tab contract When a workflow uses `agent_browser` in CDP mode, it is controlling the user's real Chrome through a shared CDP port. Chrome tabs are global to that browser, so the workflow must not rely on agent-browser's "latest active tab" behavior. The project wrapper enforces an explicit tab on every page action. CDP is not a background-isolated mode. Because it drives a visible, user-profile Chrome window, tab selection, navigation, clicks, fills, uploads, and sometimes snapshots may bring Chrome to the foreground and interrupt typing in Codex or another app. Use CDP only when the workflow needs the user's real logged-in browser or a site blocks headless automation. For schedules and unattended work, prefer headless mode or a dedicated automation Chrome profile/port that the user does not use interactively. ### Required flow 1. Try the real tab list first. The wrapper gives it 15 seconds, then falls back to remembered selected-tab state if Chrome/CDP is stuck: ```python browser("tab", []) ``` 2. Reuse a listed or selected tab if it matches the workflow. 3. Create one stable labeled tab when no tab is selected or the selected tab is unrelated: ```python browser("tab", ["new", "--label", "upwork-profile", "https://www.upwork.com/"]) ``` 4. Navigate with URL-only `open`, then include the tab inline on later page actions: ```python browser("open", ["https://www.upwork.com/freelancers/example"]) browser("snapshot", ["tab", "upwork-profile", "-i"]) browser("click", ["tab", "upwork-profile", "@e1"]) browser("fill", ["tab", "upwork-profile", "@e2", "search text"]) browser("eval", ["tab", "upwork-profile", "document.title"]) ``` `["--tab", "", ...]` is also accepted. ### Enforcement The MCP wrapper rejects CDP page actions that omit a tab. The error includes the selected-tab hint and tells the agent to retry with either `["tab", "", ...]` or `["--tab", "", ...]`. `tab` and `reset` commands are exempt because they are used to inspect or repair tab state. The wrapper translates the inline tab into native agent-browser behavior by selecting the tab immediately before the command, then running the command under a per-CDP-port mutex. This keeps `select tab -> action` atomic enough to avoid interleaving with another workflow command on the same CDP port. Commands against different CDP ports do not share that lock. ### Important distinction Native agent-browser uses a session-level active tab (`agent-browser --session ... tab t2`, then `agent-browser --session ... snapshot`). The inline tab form is this project's MCP wrapper API; it is intentionally stricter so the LLM chooses the tab on every browser action instead of inheriting whichever tab happened to be active. ### Direct CDP scripts Raw CDP scripts can bypass the wrapper, so they must follow the same tab discipline manually: - Read existing targets from `/json/list`. - Reuse a matching target by URL/title when possible. - Prefer `agent_browser eval` with an inline tab for complex JavaScript because it uses the shared CDP command lock. - Treat direct CDP as read-only diagnostics by default. It bypasses the wrapper lock, so do not use it for navigation, clicking, filling, scrolling, uploads, or multi-page loops. - Do not call `window.location`, `element.click()`, `Target.createTarget`, `/json/new`, `Target.closeTarget`, or `/json/close` unless the task explicitly requires disposable raw-CDP control and the user accepts that it bypasses shared-browser locking. - If direct CDP is used for diagnostics, connect to the chosen target's WebSocket and close only the WebSocket client, not the Chrome tab. ## Site-access resilience Production sites often block Playwright-launched browsers. When `browser_navigate` returns "Permission Denied" / a blank page / a native `alert()` freeze, the agent should: 1. **Stop launching a fresh browser** — the Playwright fingerprint is the problem. 2. **CDP-attach to an already-running Chrome** with a real user profile: - `agent-browser --cdp ` / `--auto-connect` - Playwright's `connect_over_cdp("http://localhost:9222")` in `main.py`. 3. **Register a dialog handler** before interacting if the page shows native alerts. 4. **Document the access preamble in learnings** so future steps detect-and-switch automatically. Cloudflare-style interstitials may also be bypassable by using an alternate subdomain (e.g. `prod.gcp.example.com` vs `prod.example.com`). Record these in the site-profile learning. ## Ephemeral refs — explicit ban Refs like `@e1`, `e68`, `"ref": "abc123"` appear in snapshot output and are usable **only for the immediate next tool call in the same session**. They are: - **Not stable across snapshots** — a fresh snapshot reassigns them. - **Not stable across sessions** — every new browser spawn rebuilds the ref map. - **Never safe to save** to main.py, learnings, or any other artifact. The validator `reviewMainPyScript` at `controller_learn_code.go` runs a regex check (`Check 10`) that rejects any saved main.py containing `['"]@e\d+['"]` or `{"ref": "abc..."}` when the script calls browser tools. This catches the failure mode at save time before it costs you a broken run. ## What learnings must capture for browser steps When `learnings_access="read-write"` and the step used a browser tool, the post-step learning agent MUST produce the following content in `learnings/_global/SKILL.md` (typically split across `references/site-profile.md` and `references/selectors.md`): 1. **Site access preconditions** — CDP required, Cloudflare interstitial on apex, dialog handler needed, failure signatures so future steps can detect-and-switch. 2. **Stable-hook inventory** — once per site: counts of testids / aria-labels / ids, framework, recommended locator strategy. 3. **Per-action intents, not raw selectors** — record semantic identity plus 1–2 alternates: ``` Step [login.fill_user_id] intent: {by: "id", value: "panAdhaarUserId"} alt: {by: "placeholder", value: "PAN/ Aadhaar/ Other User ID"} alt: {by: "role+name_contains", role: "textbox", name: "User ID"} notes: Continue button stays disabled until input has a value. ``` 4. **Behavioral quirks** — multi-step flows (User ID → Continue → Password), cross-domain redirects (e-Filing → TRACES), disabled-until-valid gates, OTP/captcha branches, phantom controls (e.g. a `#btn` that looks like Proceed but does nothing — the real action is a link below). 5. **Known-bad selector patterns** — explicit per-site "do NOT use" list (Radix auto-ids, mat-mdc-*, whatever bit this workflow). Intents (not raw selectors) are the core contract: future fix loops and harden runs re-derive the selector from the intent against the current snapshot/probe output. This is why intents are 1–2 alternates per action rather than a single locator. ## Which agents see these rules The browser-authoring block (`BuildBrowserAuthoringRules`) is injected into every prompt that authors, patches, or reviews main.py — **provided the workflow has a browser MCP** (`HasBrowserCapability()` returns true): | Agent | Reads the block? | Why | |---|---|---| | Execution agent (learn-code mode) | ✅ | Writes main.py; must use durable locators | | Execution agent (code-exec-only mode) | ✅ | Even throwaway scripts shouldn't bake refs — same discipline | | Learning agent | ✅ | Needs to know what learnings SHOULD contain for browser steps | | Harden agent | ✅ | Patches main.py during eval-driven fixes; must enforce the contract | | Review agent | ✅ | Detects drift in main.py; must know what "drift" means | | Todo-task orchestrator | ✅ | Same execution semantics as regular steps | Non-browser workflows (workflow config has no browser server/skill/CDP) skip the entire block — saves ~60 lines of prompt tokens per step. ## Code locations - **`prompt_sections.go:BuildBrowserAuthoringRules()`** — the full browser-rules block including the canonical DOM probe JS. Single source of truth. - **`prompt_sections.go:browserAuthoringRulesIfBrowserEnabled()`** — helper for call sites that have direct access to the orchestrator (workshop manager). - **`prompt_sections.go:BrowserAuthoringRulesFromTemplateVars()`** — helper for call sites that only have templateVars (harden, review, execution-only). - **`controller.go:HasBrowserCapability()`** — the gate. Checks explicit non-`none` browserMode, CDP port, registered servers (`playwright`), `agent-browser` skill. Never use `GetBrowserMode() != ""` as a proxy — empty means auto-detect, not disabled. - **`execution_only_agent.go`** — template has `{{.BrowserAuthoringRules}}` slot; populated by `BrowserAuthoringRulesFromTemplateVars(templateVars)`. - **`interactive_workshop_manager.go`** — workshop/harden/review call sites set `HasBrowserAccess` from `iwm.controller.HasBrowserCapability()`. - **`learning_agent.go`** — `{{if .HasBrowserAccess}}...{{end}}` block describes the required learnings shape for browser steps. - **`controller_learn_code.go:reviewMainPyScript` Check 10** — regex-rejects ephemeral refs in saved main.py when the script imports browser tool calls. ## What's intentionally NOT done - **No new `browser_inspect` MCP tool.** The probe is JS the agent passes to existing `browser_evaluate`. Rationale: keeps tool surface lean; probe logic stays in one prompt file. Promote to a real tool only if prompt-embedded proves unreliable in practice. - **No self-healing runtime.** When a saved main.py selector rots, the current flow is: run fails → execution agent sees error → patches main.py. A future "self-healing" pass could read the intent from learnings, re-derive the selector from the current snapshot, and patch automatically without invoking the execution agent. Designed but deferred until we have real failure-pattern data. - **No iframe probe traversal.** The probe runs in the top document only. Sites with iframes (TRACES embedded forms, Stripe checkout) need per-frame probe invocation. Deferred. ## Related docs - [learning_architecture.md](learning_architecture.md) — how learnings get written and manually locked. - [step_config_format_specification.md](step_config_format_specification.md) — `learnings_access` field governing read/write of global SKILL.md. - [learn_code_flow.md](learn_code_flow.md) — the full learn-code lifecycle (fast path, fix loop, save-back). ## Cost And Log Measurement Canonical HTML: https://agentworkshq.com/docs/workflow/cost_and_log_measurement/ Raw Markdown: https://agentworkshq.com/docs-content/workflow/cost_and_log_measurement.md # Cost And Log Measurement This doc explains how workflow costs and logs are measured now. It is not the same as [workflow_monitoring.md](./workflow_monitoring.md). - `workflow_monitoring.md` is about the user-facing observability surfaces - this doc is about the storage and measurement architecture underneath those surfaces ## Two Different Systems There are two separate data planes: ### 1. Cost measurement - driven by `token_usage` events from LLM calls - persisted into the workflow `costs/` ledger - aggregated by run folder, phase, step, and model ### 2. Workflow logs - written explicitly by workflow controllers as files under `runs/.../logs/` - store execution results, conversations, validation results, learning traces, orchestration traces, and markers Costs are not reconstructed from log files. Logs are not reconstructed from cost files. ## Cost Measurement Source Of Truth The source of truth for workflow cost measurement is the `token_usage` event stream. Current flow: 1. An agent emits a `token_usage` event. 2. The context-aware bridge intercepts that event. 3. It extracts: - provider - model ID - prompt/input tokens - completion/output tokens - cache read and cache write tokens - reasoning tokens - LLM call count 4. It persists those values directly to the `costs/` ledger. This happens in [context_aware_bridge.go](../../agent_go/pkg/orchestrator/context_aware_bridge.go). ## What Gets Counted The cost ledger tracks per-model usage with pricing fields: - input tokens - output tokens - cache tokens - cache read tokens - cache write tokens - reasoning tokens - LLM call count - input cost - output cost - reasoning cost - cache read cost - cache write cost - total cost - context window usage The main types are defined in: - [base_orchestrator_types.go](../../agent_go/pkg/orchestrator/base_orchestrator_types.go) - [cost_storage.go](../../agent_go/pkg/orchestrator/cost_storage.go) Important nuance: - cache reads and cache writes are tracked separately - pricing is normalized or recomputed from model pricing helpers when files are read or merged - the ledger stores both per-model totals and per-step-per-model totals ## Cost Storage Layout The current storage layout is under `costs/`. ### Phase-only costs Used for phase agents such as planning and builder-style workflow-phase sessions. Files: - `costs/phase/token_usage.json` - `costs/phase/daily/YYYY-MM-DD.json` These are written by `PersistPhaseTokenUsage(...)` in [base_orchestrator_tokens.go](../../agent_go/pkg/orchestrator/base_orchestrator_tokens.go). Builder workflow-phase sessions also write phase cost data directly in [server.go](../../agent_go/cmd/server/server.go). ### Execution run costs Used for normal workflow execution runs. Files: - `costs/execution//YYYY-MM-DD.json` Each daily file stores a map of run folders: - `run_folders[runFolder] = TokenUsageFile` So one daily bucket can contain multiple runs for the same group on the same UTC date. ### Evaluation run costs Used for evaluation execution. Files: - `costs/evaluation//YYYY-MM-DD.json` Evaluation uses the same ledger shape as execution, but under the `evaluation` scope. ## Aggregation Shape Run-level cost data is aggregated into two main maps: - `by_model` - `by_step_and_model` `by_model`: - total usage across the whole run for each model `by_step_and_model`: - nested usage keyed by step and model - current key format is usually `phase:stepID` - older fallback format may use `phase:stepIndex` This is what lets the UI render: - stage totals - step-wise totals - model-wise totals - per-run totals ## Legacy Compatibility Older layouts still exist in some workspaces: - `runs//token_usage.json` - `evaluation/runs//token_usage.json` - workspace-root `token_usage.json` for phase costs Current behavior: - these are migrated into the `costs/` ledger when read or written - migration is handled automatically by the cost storage helpers This logic is in [cost_storage.go](../../agent_go/cmd/server/cost_storage.go) and [token_usage_store.go](../../agent_go/pkg/orchestrator/token_usage_store.go). ## How `/api/workflow/costs` Works The workflow cost API does not read a single per-run JSON file anymore. Current behavior: - reads `costs/phase/token_usage.json` - reads `costs/phase/daily/*` - reads all execution daily ledgers under `costs/execution/*` - reads all evaluation daily ledgers under `costs/evaluation/*` - merges run-folder totals across daily files - returns execution and evaluation cost data side by side for each run folder This happens in: - [workflow.go](../../agent_go/cmd/server/workflow.go) - [cost_storage.go](../../agent_go/cmd/server/cost_storage.go) ## Step Token Usage Events `step_token_usage` is a derived summary event, not the primary source of truth. Current behavior: - the system reads the persisted cost ledger for the current run - aggregates the requested step across all models - emits a summary event for UI display This means: - `token_usage` events drive persistence - `step_token_usage` events summarize persisted step totals See [base_orchestrator_tokens.go](../../agent_go/pkg/orchestrator/base_orchestrator_tokens.go). ## Workflow Log Source Of Truth Workflow logs are explicit artifact files written by workflow controllers. They are not reconstructed from the database event stream. The main log families are: - execution result JSON - execution conversation JSON - execution prompts JSON - validation JSON - pre-validation JSON - learning execution JSONL - learning conversation JSON - conditional evaluation JSON - orchestration execution JSONL - todo-task execution JSONL ## Log File Layout For a normal run folder, logs live under: - `runs/{runFolder}/logs/...` Common files: - `logs/{step-folder}/execution/execution-attempt-{N}-iteration-{M}.json` - `logs/{step-folder}/execution/execution-attempt-{N}-iteration-{M}-conversation.json` - `logs/{step-folder}/execution/execution-attempt-{N}-iteration-{M}-prompts.json` - `logs/{step-folder}/validation.json` - `logs/{step-folder}/validation-{N}.json` - `logs/{step-folder}/pre_validation.json` - `logs/{step-folder}/learning-execution.json` - `logs/{step-folder}/learning-conversation.json` - `logs/{step-folder}/conditional-evaluation.json` - `logs/{step-folder}/orchestration-execution.json` - `logs/{step-folder}/todo-task-execution.json` There are also builder/session logs under: - `builder/conversation/YYYY-MM-DD/session-{sessionId}-conversation.json` ## Who Writes The Logs Main writers: - execution result and conversation logs: [controller_execution.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_execution.go) - learning logs: [controller_learning.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_learning.go) - pre-validation logs: [pre_validation.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/pre_validation.go) - workflow log API scanner: [workflow.go](../../agent_go/cmd/server/workflow.go) ## Archived Logs When a step is re-executed, older log files can be archived instead of being discarded. Current behavior: - selected log files are moved into `archived/{timestamp}` under the step log folder - this preserves older attempts for debugging while keeping the active log folder readable This is handled in [controller_progress.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_progress.go). ## How `/api/workflow/logs` Works The workflow log API scans the run folder and reconstructs a structured view for the UI. Current behavior: - scans `runs/{runFolder}/logs/` - scans `runs/{runFolder}/execution/` - maps step folders back to plan metadata from `planning/plan.json` - parses known file families into typed response sections - exposes conversation file paths for lazy loading This is why the execution log popup can show: - per-step execution attempts - validation history - learning traces - conditional results - orchestration and todo-task traces The implementation is in [workflow.go](../../agent_go/cmd/server/workflow.go). ## Practical Summary Use this mental model: - costs come from `token_usage` events and live in `costs/` - logs come from controller-written files and live in `runs/.../logs/` - `step_token_usage` is a derived UI summary from persisted costs - `/api/workflow/costs` merges the cost ledger - `/api/workflow/logs` scans and structures the run artifacts ## Related Docs - [workflow_monitoring.md](./workflow_monitoring.md) - [iteration_run_folder_architecture.md](./iteration_run_folder_architecture.md) - [evaluation_system.md](./evaluation_system.md) ## Evaluation System Canonical HTML: https://agentworkshq.com/docs/workflow/evaluation_system/ Raw Markdown: https://agentworkshq.com/docs-content/workflow/evaluation_system.md # Evaluation System This doc describes how workflow evaluation works now. The current system is file-backed and execution-oriented: - the evaluation plan lives in `evaluation/evaluation_plan.json` - evaluation step config lives in `evaluation/step_config.json` - evaluation execution runs in an internal sandbox under `evaluation/runs/iteration-0[/group]` - the final report is published to `evaluation/runs/{targetRunFolder}/evaluation_report.json` ## What Evaluation Is For Evaluation is the workflow's goal-measurement layer for completed execution runs. It answers: - did a target run actually satisfy the success criteria in `soul/soul.md`? - what did each evaluation step conclude, with what evidence? It is separate from: - execution-time pre-validation (mechanical run-shape checks) - Pulse per-run triage (operational breakage: errors, skipped steps, empty artifacts, hallucinated successes) - step learning - workflow execution itself Division of labor: **Pulse owns "did it run right"; evals own "did it achieve the goal."** Eval steps should map one-to-one to success criteria and never duplicate operational checks — see the `evaluation-plan` guidance template for the authoring contract. Pulse reads the eval report each run and maps verdicts onto the per-criterion goal card in `builder/improve.html`, which is the durable goal signal (there is no separate numeric metrics layer). ## Current Mental Model Use this mental model: 1. Build or edit the eval plan in `evaluation/evaluation_plan.json`. 2. Point evaluation at a completed execution run folder such as `iteration-8/production`. 3. Evaluation steps execute in an internal eval sandbox. 4. Those eval steps read the original execution artifacts through `{{TARGET_RUN_PATH}}`. 5. A scoring phase produces `evaluation_report.json`. 6. The report is published back under the requested target run folder path inside `evaluation/runs/`. ## Eval Plan Files The core evaluation files are: - `evaluation/evaluation_plan.json` - `evaluation/step_config.json` - `evaluation/eval_layout.json` - `learnings/` — shared with execution-plan learnings (see "Learnings And Step Config In Eval Mode" below) Frontend eval mode loads these files directly. The eval plan is not embedded in the main workflow manifest structure. Eval steps can be scoped to the route or artifact they apply to: ```json { "id": "eval-bid-submission", "title": "Evaluate Bid Submission", "description": "Verify bid submission artifacts...", "applies_to_routes": [ { "routing_step_id": "workflow-mode-router", "route_ids": ["route-bid"] } ] } ``` When this field is present, the runtime checks the target run's `routing-evaluation.json` before launching the eval step. Non-applicable checks are skipped and recorded in the final report with `max_score: 0`, so a run is not penalized for route paths it did not take. Current frontend behavior is implemented in [useEvaluationPlanData.ts](../../frontend/src/components/workflow/hooks/useEvaluationPlanData.ts). ## Eval Mode In The UI The UI still has a separate eval mode. Current behavior: - plan mode shows the main workflow - eval mode shows `evaluation/evaluation_plan.json` - eval steps use `evaluation/step_config.json` - eval layout is separate from the main workflow layout So evaluation is still a first-class workspace view, not just a hidden background feature. ## Running Evaluation The runtime entry point is `ExecuteEvaluationOnly(...)`. Current behavior: - reads `evaluation/evaluation_plan.json` - requires a target run folder - computes an internal eval run folder using the workshop-style `iteration-0` sandbox - sets `isEvaluationMode = true` - injects `TARGET_RUN_PATH` - applies `evaluation/step_config.json` - runs the evaluation steps - runs a final scoring phase This is implemented in [evaluation_execution.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_execution.go). ## Internal Eval Sandbox One of the biggest architecture changes is the eval run folder model. Evaluation does **not** primarily execute inside: - `evaluation/runs/{targetRunFolder}/...` Instead it executes inside an internal sandbox: - `evaluation/runs/iteration-0` - or `evaluation/runs/iteration-0/` The target run folder is still important, but it is the thing being evaluated, not the main place eval steps execute. This is why the code uses: - internal eval run folder for execution - published target run folder for the final report ## TARGET_RUN_PATH `TARGET_RUN_PATH` is the main bridge between evaluation and the original workflow run. Current behavior: - the runtime injects `TARGET_RUN_PATH` as the absolute path to the original execution folder - eval steps are expected to read the original run artifacts through that variable For example, the original run might be: - `runs/iteration-12/production/execution` The eval step runs in the eval sandbox, but reads the original artifacts through: - `{{TARGET_RUN_PATH}}` This is the correct way to reference original execution outputs in eval steps. ## Report Phase After eval steps finish, the runtime builds a single `evaluation_report.json` covering every eval step — directly in Go (`runEvaluationReportPhase` in `evaluation_execution.go`). **There is no scoring agent, no combined LLM scoring call, and no learn_code scoring fast path** (all removed); each eval step's own structured output is its verdict. ### Report shape The on-disk report contains: - `target_run_folder`, `generated_at` - `step_scores[]`, one entry per eval step: - `step_id` - placeholder `score: 0` / `max_score: 0` with fixed `reasoning`/`evidence` text pointing readers at `output_content` ("Final scoring is disabled; this report preserves the eval step output for review") - `output_content` — the eval step's own structured output, attached from the first of: `output_content.json`, the step's `context_output` file, a `pre_validation` file, or `context_output.json` in the eval sandbox (`enrichEvaluationReportWithStepOutputs`) - eval steps skipped by `applies_to_routes` appear as skipped entries with `max_score: 0`, so a run is not penalized for route paths it did not take `output_content` is the source of truth for each eval step's verdict: the step should emit its own `score`, `max_score`, `reasoning`, and `evidence` (plus any domain-specific judgment dimensions) as structured output, enforced by the step's validation schema. Consumers — Pulse triage, the goal card in `builder/improve.html`, the scheduled improve loop, the UI — read the per-step verdicts; nothing numeric is aggregated downstream. ### Where the report lands Whichever path produces it, the runtime writes the report to: - internal path: `evaluation/runs/{internalEvalRunFolder}/evaluation_report.json` - published path: `evaluation/runs/{targetRunFolder}/evaluation_report.json` The publish step is what makes evaluation reports line up with the execution run the user asked about. ## Auto-Evaluation Evaluation can also run automatically after workflow execution. Current behavior: - after a successful batch group execution, the workflow checks whether `evaluation/evaluation_plan.json` exists - if it exists, auto-evaluation runs for that target run folder - evaluation failures do not fail the original group execution This behavior is implemented in: - [controller_batch_execution.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_batch_execution.go) - [evaluation_execution.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_execution.go) ## Evaluation Costs The old mental model of evaluation writing a standalone `evaluation/runs/.../token_usage.json` as the main source of truth is no longer the right model. Current cost architecture: - evaluation cost data is stored in the `costs/` ledger under the `evaluation` scope - `/api/workflow/costs` returns eval cost data as `evaluation_token_usage` - the UI merges execution and evaluation costs when showing workflow run cost summaries So: - execution costs -> `costs/execution/...` - evaluation costs -> `costs/evaluation/...` For the full cost architecture, see [cost_and_log_measurement.md](./cost_and_log_measurement.md). ## Evaluation Reports API The evaluation report UI is driven by `/api/workflow/evaluation-reports`. Current behavior: - scans `evaluation/runs/*` - finds `evaluation_report.json` - supports both bare iteration folders and nested group folders - returns aggregate statistics across reports - can filter to a specific run folder This is implemented in [workflow.go](../../agent_go/cmd/server/workflow.go). ## Learnings And Step Config In Eval Mode Evaluation mode has its own step config but shares the learnings namespace with execution steps. Current behavior: - step config comes from `evaluation/step_config.json` (separate from `planning/step_config.json`) - learnings go under `learnings/{stepID}/` — the same folder execution steps write to - cross-plan step-ID uniqueness between `planning/plan.json` and `evaluation/evaluation_plan.json` is enforced at write time by `validateCrossPlanStepIDUniqueness`, so the shared namespace cannot collide Eval steps therefore reuse workflow learnings (e.g. a `_global` SKILL.md written by execution steps is visible to eval steps), but keep an independent `step_config.json`. ## Validation Of The Eval Plan The system includes `validate_evaluation_plan` for checking the file after edits. Current validation checks include: - JSON parse validity - non-empty step list - unique IDs - required title and description - validation of `pre_validation` regex and JSONPath rules where present This is implemented in [evaluation_helpers.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_helpers.go). ## What Changed From Older Docs The older evaluation doc described a different architecture in a few places. Current corrections: - evaluation is file-backed, not centered on a separate designer manager architecture - eval execution uses an internal `evaluation/runs/iteration-0[/group]` sandbox - the published report is copied to the target run folder path - evaluation costs come from the `costs/` ledger, not primarily from legacy per-run token files - the eval plan is edited and loaded directly from `evaluation/evaluation_plan.json` - the combined LLM scoring agent and the `__evaluation_scoring__` learn_code fast path were removed — the report is assembled in Go from per-step `output_content` - the numeric metrics layer (`planning/metrics.json`, `db/metrics_history.jsonl`, `propose_metric`/`retire_metric`) was removed (2026-07-01) — the goal signal is the per-criterion goal card in `builder/improve.html`, maintained agentically by Pulse from eval reports + `soul.md` ## Practical Summary Use this mental model: - edit the eval plan in `evaluation/evaluation_plan.json` - run evaluation against a completed execution run - eval steps execute in the internal eval sandbox - eval steps read original execution artifacts via `{{TARGET_RUN_PATH}}` - scoring publishes `evaluation_report.json` back to the requested target run folder - costs for eval runs are tracked in the `costs/evaluation/` ledger ## Related Docs - [cost_and_log_measurement.md](./cost_and_log_measurement.md) - [iteration_run_folder_architecture.md](./iteration_run_folder_architecture.md) - [workflow_builder_interactive.md](./workflow_builder_interactive.md) - [workflow_monitoring.md](./workflow_monitoring.md) ## Human Feedback System Canonical HTML: https://agentworkshq.com/docs/workflow/human_feedback_system/ Raw Markdown: https://agentworkshq.com/docs-content/workflow/human_feedback_system.md # Human Feedback System ## 📋 Overview The human feedback system is an interactive virtual tool that pauses LLM execution to request real-time input from users, enabling human-in-the-loop workflows. It provides browser notifications, real-time UI updates, and optional Slack integration for seamless feedback collection. The system supports 2FA/OTP input, confirmations, and clarifying questions through a thread-safe request/response coordination system. **Key Benefits:** - Enables human-in-the-loop workflows for critical decisions - Supports 2FA/OTP input, confirmations, and clarifying questions - Real-time notifications via browser push notifications - Optional Slack integration with smart delayed notifications (2-minute delay) - Thread-safe request/response coordination with timeout handling - Respond via Slack threads or UI (no UI required for Slack responses) - Integrates seamlessly with event-driven architecture --- ## 📁 Key Files & Locations | Component | File | Key Functions | |-----------|------|---------------| | **Virtual Tool** | [`human_tools.go`](../../agent_go/cmd/server/virtual-tools/human_tools.go) | `CreateHumanTools()`, `handleHumanFeedback()` | | **Backend Store** | [`human_feedback_store.go`](../../agent_go/cmd/server/virtual-tools/human_feedback_store.go) | `CreateRequest()`, `CreateRequestWithSlack()`, `SubmitResponse()`, `WaitForResponse()`, `Cleanup()` | | **API Endpoint** | [`server.go`](../../agent_go/cmd/server/server.go) | `handleSubmitHumanFeedback()` (POST `/api/human-feedback/submit`) | | **Slack Service** | [`slack_service.go`](../../agent_go/cmd/server/services/slack_service.go) | `SendFeedbackNotification()`, `GetUniqueIDFromThread()`, `TestConnection()` | | **Slack API Routes** | [`slack_feedback_routes.go`](../../agent_go/cmd/server/slack_feedback_routes.go) | Configuration and test endpoints | | **Frontend UI** | [`HumanFeedbackToolCallDisplay.tsx`](../../frontend/src/components/events/tools/ToolCallSpecialRender/HumanFeedbackToolCallDisplay.tsx) | `HumanFeedbackToolCallDisplay` component | | **Slack Config UI** | [`SlackFeedbackConfig.tsx`](../../frontend/src/components/settings/SlackFeedbackConfig.tsx) | Configuration component | | **API Service** | [`api.ts`](../../frontend/src/services/api.ts) | `submitHumanFeedback()`, `getSlackFeedbackConfig()`, `updateSlackFeedbackConfig()`, `testSlackConnection()` | | **Event Data Structures** | [`data.go`](../../agent_go/pkg/orchestrator/events/data.go) | `BlockingHumanFeedbackEvent`, `RequestHumanFeedbackEvent` | | **Orchestrator Helpers** | [`base_orchestrator.go`](../../agent_go/pkg/orchestrator/base_orchestrator.go) | `RequestHumanFeedback()`, `RequestYesNoFeedback()`, `RequestMultipleChoiceFeedback()` | --- ## 🔄 How It Works ### Core System Lifecycle 1. **Tool Registration** - `CreateHumanTools()` defines the `human_feedback` virtual tool - Tool requires: `{"message_for_user": string, "unique_id": string (UUID)}` - Executor: `handleHumanFeedback()` function 2. **LLM Invokes Tool** - LLM calls `human_feedback` with message and unique UUID - `handleHumanFeedback()` receives the call 3. **Backend Request Creation** - Global `HumanFeedbackStore` singleton creates request entry - Store maps `unique_id` → `HumanFeedbackRequest` struct - Creates channel (`chan string`) for response coordination - If Slack enabled, starts 2-minute delayed notification timer 4. **Frontend Notification** - Event system automatically notifies frontend - `HumanFeedbackToolCallDisplay` component renders - Browser push notification shown (if permissions granted) - UI displays message + text input + submit button 5. **Slack Notification (Optional, Delayed)** - If Slack enabled and user hasn't responded within 2 minutes - System checks if user already responded - If no response, sends Slack notification to configured channel - Message includes question, context, and unique request ID - Maps Slack message timestamp to unique ID for thread replies 6. **User Response** - User responds via UI (types response, clicks "Submit Feedback" or Ctrl/Cmd+Enter) - OR user replies in Slack thread - Frontend sends POST to `/api/human-feedback/submit` (UI path) - OR Slack Events API sends webhook to backend (Slack path) - Backend verifies Slack webhook signature and extracts unique ID from thread 7. **Backend Coordination** - Response written to channel - `WaitForResponse()` unblocks with user's response - `handleHumanFeedback()` returns response to LLM 8. **LLM Continuation** - LLM receives user response as tool output - Execution continues in same turn with user input ### Timeout Handling - Default timeout: **5 minutes** (tool executor) - Orchestrator helpers: **10 minutes** - On timeout: Returns error, LLM handles gracefully ### Delayed Notification Logic The Slack integration uses smart delayed notifications: 1. **Immediate**: Feedback request appears in UI right away 2. **2-Minute Delay**: System waits 2 minutes before sending Slack notification 3. **Response Check**: Before sending, system checks if user already responded 4. **Conditional Send**: Only sends Slack notification if user hasn't responded 5. **Non-Blocking**: All notification logic runs asynchronously --- ## 🏗️ Architecture ```mermaid sequenceDiagram participant LLM participant Tool as handleHumanFeedback participant Store as HumanFeedbackStore participant Slack as SlackService participant API as /api/human-feedback/submit participant Frontend as React UI participant User LLM->>Tool: Call human_feedback(message, unique_id) Tool->>Store: CreateRequestWithSlack(unique_id, message) Store-->>Tool: Request created Store->>Frontend: Event: tool_call_start Frontend->>User: Show notification + UI Store->>Store: Start 2-minute timer (async) Tool->>Store: WaitForResponse(unique_id, 5min) Note over Tool,Store: Tool blocks here alt User responds via UI (< 2 min) User->>Frontend: Type response + submit Frontend->>API: POST /api/human-feedback/submit API->>Store: SubmitResponse(unique_id, response) Store->>Tool: Channel receives response Tool-->>LLM: Return user response else User doesn't respond (< 2 min) Store->>Store: Check if user responded Store->>Slack: SendFeedbackNotification() (if enabled) Slack->>User: Slack message in channel User->>Slack: Reply in thread Slack->>Store: Extract unique_id, SubmitResponse() Store->>Tool: Channel receives response Tool-->>LLM: Return user response end ``` --- ## 🧩 Example Usage ### LLM Tool Call ```json { "tool_name": "human_feedback", "arguments": { "message_for_user": "Please enter the 2FA code sent to your email", "unique_id": "550e8400-e29b-41d4-a716-446655440000" } } ``` ### Backend Tool Handler **File:** [`human_tools.go`](../../agent_go/cmd/server/virtual-tools/human_tools.go) ```go func handleHumanFeedback(ctx context.Context, args map[string]interface{}) (string, error) { messageForUser := args["message_for_user"].(string) uniqueID := args["unique_id"].(string) feedbackStore := GetHumanFeedbackStore() if err := feedbackStore.CreateRequestWithSlack(ctx, uniqueID, messageForUser, "", nil); err != nil { return "", fmt.Errorf("failed to create feedback request: %w", err) } response, err := feedbackStore.WaitForResponse(uniqueID, 5*time.Minute) if err != nil { return "", fmt.Errorf("failed to get user feedback: %w", err) } return response, nil } ``` ### Orchestrator Helper Functions **File:** [`base_orchestrator.go`](../../agent_go/pkg/orchestrator/base_orchestrator.go) ```go // Request human feedback with text input approved, feedback, err := orchestrator.RequestHumanFeedback( ctx, "approval_123", "Please approve this plan", "Additional context", sessionID, workflowID, ) // Request yes/no feedback approved, err := orchestrator.RequestYesNoFeedback( ctx, "yesno_456", "Do you want to proceed?", "Approve", "Reject", "", sessionID, workflowID, ) // Request multiple choice feedback choice, err := orchestrator.RequestMultipleChoiceFeedback( ctx, "choice_789", "Select deployment environment", []string{"Development", "Staging", "Production"}, "", sessionID, workflowID, ) ``` --- ## ⚙️ Configuration ### Tool Parameters | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `message_for_user` | string | Yes | Message displayed to the user requesting feedback | | `unique_id` | string | Yes | Unique UUID identifying this request | ### Timeout Configuration | Component | Timeout | Location | |-----------|---------|----------| | Tool executor | `5 * time.Minute` | [`human_tools.go`](../../agent_go/cmd/server/virtual-tools/human_tools.go) | | Orchestrator helpers | `10 * time.Minute` | [`base_orchestrator.go`](../../agent_go/pkg/orchestrator/base_orchestrator.go) | | Browser notification | 30 seconds (auto-close) | Frontend component | ### Slack Configuration | Setting | Type | Description | |---------|------|-------------| | `enabled` | boolean | Whether Slack notifications are enabled | | `bot_token` | string | Slack bot token (starts with `xoxb-`) | | `app_token` | string | App-level token for Socket Mode (starts with `xapp-`) | | `channel_id` | string | Target Slack channel ID (starts with `C`) | | Notification delay | 2 minutes | Fixed delay before sending Slack notification | **Configuration Storage:** - Stored in `slack_feedback_config` database table - Managed via UI: Sidebar → Slack icon - API endpoints: `GET/POST /api/human-feedback/slack/config` ### Browser Notifications **Permissions:** - Requested on component mount - Shown when `Notification.permission === 'granted'` - Properties: Title "Human Feedback Required", body from `message_for_user`, `requireInteraction: true`, `silent: false` --- ## 🛠️ Common Issues & Solutions | Issue | Cause | Solution | |-------|-------|----------| | `timeout waiting for feedback` | User didn't respond within timeout period | Increase timeout in `handleHumanFeedback()` or orchestrator helper functions | | `feedback request already exists` | Duplicate `unique_id` used | Always generate fresh UUID: `fmt.Sprintf("feedback_%d", time.Now().UnixNano())` | | Browser notification not showing | Permission denied or not granted | Request permission via button in UI or browser settings | | Response not received | Frontend submit failed | Check browser console for errors; verify `/api/human-feedback/submit` endpoint accessible | | Tool blocks forever | Backend store channel deadlock | Check that `SubmitResponse()` is called; review channel handling in `WaitForResponse()` | | Slack connection test fails | Invalid credentials or scopes | Verify bot token (`xoxb-`), app token (`xapp-`), channel ID (`C`), bot has `chat:write` scope, bot is member of channel | | Slack notifications not sent | User responded quickly | Expected behavior - notifications only sent if no response after 2 minutes | | Thread replies not captured | Socket Mode misconfigured | Verify Socket Mode enabled, Events API enabled, `message.channels` event subscribed, App-Level Token has `connections:write` scope, bot has `channels:history` scope | --- ## 🔍 For LLMs: Quick Reference ### Constraints ✅ **Allowed:** - Any user-facing message (questions, requests for OTP/2FA, confirmations) - UUID generation for `unique_id` parameter - Waiting synchronously for user response (blocks LLM execution) - Multiple feedback requests in sequence (different unique_ids) ❌ **Forbidden:** - Reusing same `unique_id` for multiple requests - Not providing `unique_id` (required parameter) - Empty or missing `message_for_user` - Expecting instant response (users may take time) ### Example Pattern **Simple feedback request:** ```json { "tool": "human_feedback", "arguments": { "message_for_user": "Please confirm the database migration plan before I proceed", "unique_id": "feedback_1701234567890" } } ``` **2FA/OTP use case:** ```json { "tool": "human_feedback", "arguments": { "message_for_user": "Enter the 6-digit verification code sent to your email", "unique_id": "otp_1701234567891" } } ``` **LLM receives:** ``` User response: "582491" ``` ### Integration with Workflow Agents **Planning agent pattern:** ```go // 1. human_feedback - ask user for approval // 2. Receive yes/no response // 3. If approved: call update_plan_steps / add_plan_steps / delete_plan_steps ``` **Variable extraction agent pattern:** ```go // 1. human_feedback - "I detected variable {{API_KEY}}. Should I extract it?" // 2. User responds: "Yes" or "No" // 3. If yes: call update_variable tool // 4. If no: don't modify variables ``` --- ## 🔌 API Endpoints ### Submit Human Feedback **POST** `/api/human-feedback/submit` **Request:** ```json { "unique_id": "550e8400-e29b-41d4-a716-446655440000", "response": "User's feedback text" } ``` **Response:** ```json { "success": true } ``` ### Slack Configuration **GET** `/api/human-feedback/slack/config` **Response:** ```json { "enabled": true, "channel_id": "C1234567890", "bot_token": "xoxb-...1234", "app_token": "xapp-...5678" } ``` **POST** `/api/human-feedback/slack/config` **Request:** ```json { "enabled": true, "bot_token": "xoxb-...", "app_token": "xapp-...", "channel_id": "C1234567890" } ``` **POST** `/api/human-feedback/slack/test` **Response:** ```json { "success": true, "message": "Slack connection test successful!" } ``` --- ## 🔒 Security Model ### Thread Safety **Global Singleton:** ```go var ( globalHumanFeedbackStore *HumanFeedbackStore humanFeedbackStoreOnce sync.Once ) ``` **Mutex Protection:** - `sync.RWMutex` protects concurrent access to request map and waiters - Write operations (CreateRequest, SubmitResponse) use `Lock()` - Read operations (WaitForResponse lookup) use `RLock()` ### Socket Mode Security - WebSocket connection (no public webhook required) - App-Level Token separate from Bot Token - Automatic reconnection handling - Webhook signature verification for thread replies ### Token Storage - Tokens stored in database (`slack_feedback_config` table) - Tokens masked in API responses (last 4 characters only) - **Recommendation:** Encrypt tokens in production environments --- ## 📊 Database Schema ### `slack_feedback_config` Table | Column | Type | Description | |--------|------|-------------| | `id` | TEXT | Primary key (always 'slack_config') | | `enabled` | BOOLEAN | Whether Slack notifications are enabled | | `bot_token` | TEXT | Slack bot token (encrypted in production) | | `app_token` | TEXT | App-level token for Socket Mode | | `channel_id` | TEXT | Target Slack channel ID | | `created_at` | DATETIME | Creation timestamp | | `updated_at` | DATETIME | Last update timestamp | ### `slack_feedback_messages` Table | Column | Type | Description | |--------|------|-------------| | `id` | TEXT | Primary key | | `unique_id` | TEXT | Maps to `HumanFeedbackRequest.UniqueID` | | `slack_message_ts` | TEXT | Slack message timestamp | | `slack_channel_id` | TEXT | Slack channel ID | | `slack_thread_ts` | TEXT | Thread timestamp | | `created_at` | DATETIME | Creation timestamp | ### Data Structures ```go type HumanFeedbackRequest struct { UniqueID string MessageForUser string UserResponse string IsCompleted bool CreatedAt time.Time } type HumanFeedbackStore struct { requests map[string]*HumanFeedbackRequest waiters map[string]chan string mu sync.RWMutex } ``` --- ## 🚀 Slack Setup Instructions 1. **Create Slack App** at https://api.slack.com/apps 2. **Configure Bot Token Scopes**: `chat:write`, `channels:read`, `channels:history` 3. **Enable Socket Mode**: Generate App-Level Token with `connections:write` scope 4. **Enable Events API**: Subscribe to `message.channels` event 5. **Get Channel ID**: Right-click channel → View channel details 6. **Configure in UI**: Sidebar → Slack icon → Enter tokens and channel ID → Test connection → Save --- ## 📖 Related Documentation - [Workflow Monitoring](workflow_monitoring.md) - Reviews runs that may require human approval - [Todo Task Step Type](todo-task-step-type.md) - Uses human feedback for plan approval and variable confirmation - Virtual tools - See the backend tool handlers referenced above - [Event System](../core/event_system.md) - How events coordinate frontend/backend ## Learning Architecture Canonical HTML: https://agentworkshq.com/docs/workflow/learning_architecture/ Raw Markdown: https://agentworkshq.com/docs-content/workflow/learning_architecture.md # Learning Architecture This document describes the current workflow architecture as implemented now. The older model in this repo treated learning as part of a larger learning-plus-validation architecture with LLM validation modes, per-step learning files, and exploration/exploitation prompt strategies. That is no longer the right mental model. ## Current Reality Today the learning side of workflow runtime is built around two simpler ideas: - **Learning writes into a shared global skill** at `learnings/_global/`. - **Scripted code steps can also persist step-specific scripts** under their own learnings folder. If you remember the older architecture, these are the most important updates: - Learning is no longer primarily about per-step prose learnings. - The learning agent now writes domain knowledge into a global skill folder, usually centered on `learnings/_global/SKILL.md`. - For `learn_code` steps, `main.py` is the executable truth; `SKILL.md` is secondary guidance. ## Learning ### What learning writes now The main learning destination is the **global skill**: - `learnings/_global/SKILL.md` - optional supporting files under: - `learnings/_global/references/` - `learnings/_global/scripts/` - other skill-structured files `SKILL.md` should stay lean. Treat it as the index and overview for the workflow runbook, not the place for detailed accumulated guidance. Keep it under roughly 80-100 lines, with links to focused `references/.md` files. Detailed selectors, auth flows, API quirks, timing/wait rules, file-format notes, retry patterns, and step-specific HOW guidance should live in those reference files. The learning agent prompt in [`learning_agent.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/learning_agent.go) is explicit: - accumulate **domain knowledge across all workflow steps** - keep it focused on the target system - merge findings into one shared skill - follow skill structure, not old flat learning-note files - keep `SKILL.md` as a short index and put detailed HOW knowledge in reference files The controller also hardwires global learning mode in [`controller_learning.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_learning.go): - `UseGlobalLearning = "true"` - `ContributingStepID` - `ContributingStepTitle` - optional `GlobalSkillObjective` ### Step-specific learnings still exist, but differently There are still step-specific artifacts, but they are no longer the main prose-learning model: - `learn_code` / scripted steps save reusable scripts under `learnings/{step-id}/` - especially `learnings/{step-id}/main.py` - scripted steps may also keep `SKILL.md` notes for edge cases and repair hints - metadata remains per step in `learnings/{step-id}/.learning_metadata.json` So the current split is: - **global domain knowledge** → `learnings/_global/` - **step-specific executable artifacts** → `learnings/{step-id}/` ### Learning objective The current system expects a workflow-level objective for the global skill: - `global_skill_objective` This tells the learning agent what kind of reusable knowledge should be accumulated, for example: - auth flow patterns - selectors - API patterns - common failure modes - target-system structure This is a better description of the current design than the older “extract learnings per step until stable” framing. ## Learning lifecycle ### Success learning After a successful step: - runtime can launch **success learning** in the background - it reads recent execution history and validation result - it updates the global skill - it updates step metadata This happens in [controller_learning.go](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_learning.go). Important current details: - success learning is the real active learning path - learning detection via a separate LLM-based “did we learn something new?” phase has been removed - metadata is updated using a rule-based path instead ### Learning metadata Runtime learning metadata is observational. It gives the workflow builder and review tools evidence for deciding whether a step's learning writes are still useful, but it does not mutate `lock_learnings`. Current metadata logic in [`controller_learning_detection.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_learning_detection.go): - the step description is hashed (SHA256 of trimmed `step.GetDescription()`) on every successful run - if the hash matches the previously-stored `last_description_hash`, `description_hash_runs` increments - if the hash differs, `description_hash_runs` resets to 1 and the stored hash is updated Editing the step description resets the description-hash run counter. This is review evidence only; runtime execution does not auto-lock or auto-unlock learning. ### Manual lock lifecycle `lock_learnings` is owned by the builder/user. Set it when the workflow should keep reading the current global skill but stop accepting automatic SKILL.md writes from that step. Clear it explicitly after a material description change or when the builder/user wants that step to learn again. ### Locking and disabling learning The important current controls on `AgentConfigs`: - `learnings_access` (string enum: `"read" | "read-write" | "none"`) — primary gate. Mirrors `knowledgebase_access`. - `"read"` (default) — step sees `_global/SKILL.md` in its prompt; does NOT contribute. - `"read-write"` — step reads AND contributes. Requires `learning_objective` to be non-empty. - `"none"` — step neither reads nor contributes. The true disable. - `learning_objective` (string) — the **extraction instruction** for the post-step learning agent. Required when access is `"read-write"`. No longer a gate. - `lock_learnings` (bool) — freezes the learning agent for this step even while access is `"read-write"`. Existing `SKILL.md` still flows into execution prompts. Runtime never auto-sets or auto-clears this; it is a builder/user decision. - `global_skill_objective` (workflow-level, not per-step) — describes what domain knowledge the global skill should accumulate. Auto-migration for legacy configs (runtime-only, no file rewrites): if `learnings_access` is unset, `learning_objective` non-empty infers `"read-write"`; empty infers `"read"`. Recommended usage: - leave `learnings_access` unset (defaults to `"read"`) for most steps — they benefit from cross-step context. - set `learnings_access: "read-write"` + a non-empty `learning_objective` on steps that produce durable HOW-knowledge about the target system. - set `learnings_access: "none"` for steps that are truly throwaway or whose context would pollute the global skill (e.g. pure file moves, human-input steps — the latter is forced to `"none"` automatically). - set `lock_learnings: true` only when the builder/user intentionally decides the step should stop writing SKILL.md; include `review_notes` explaining why. ### Failure learning The older docs described a full failure-learning architecture. That is not a good description of the current codebase. What still exists: - some comments, metadata fields, and workshop text still reference failure learning What matters operationally now: - the active, clearly implemented learning path is success learning into the global skill Until failure-learning behavior is re-established as a first-class runtime path, docs should not present it as a central architecture pillar. ## Scripted code steps For `learn_code` steps, learning and execution are intentionally split: - `main.py` is the executable artifact - `SKILL.md` is supporting knowledge - global skill captures reusable domain knowledge - step folder captures the reusable script and related metadata That means learning for scripted steps is not just “write prose notes.” It is: - maintain reusable code in `learnings/{step-id}/main.py` - maintain reusable domain knowledge in `learnings/_global/` See [learn_code_flow.md](learn_code_flow.md). ## Current file layout ### Global ```text learnings/ _global/ SKILL.md references/ scripts/ ``` ### Step-specific ```text learnings/ / .learning_metadata.json SKILL.md # optional supporting notes main.py # scripted steps scripts/ diffs/ ``` Not every step uses every file. The important distinction is: - `_global/` is the shared workflow skill - `/` is the step-specific artifact area ## What to update in other docs When editing related workflow docs, keep these rules consistent: - describe learning as global-skill-first - gate read access and write contribution through `learnings_access` — `lock_learnings` is a freeze switch, not the enable/disable mechanism - for scripted steps, describe `main.py` as the executable source of truth - description-hash metadata is review evidence only; runtime does not auto-lock or auto-unlock learnings - leave validation details to the dedicated pre-validation docs ## Code references - [`controller_execution.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_execution.go): learning triggers and post-execution flow - [`controller_learning.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_learning.go): success learning and global-skill write path - [`learning_agent.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/learning_agent.go): global skill prompt and skill-structured output - [`controller_learning_detection.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/controller_learning_detection.go): learning metadata updates - [`interactive_workshop_manager.go`](../../agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/interactive_workshop_manager.go): current user-facing guidance for global learning ## Related docs - [pre_validation_guide.md](pre_validation_guide.md) - [step_config_format_specification.md](step_config_format_specification.md) - [learn_code_flow.md](learn_code_flow.md) ## Pulse: post-run pipeline consolidation Canonical HTML: https://agentworkshq.com/docs/workflow/pulse_consolidation/ Raw Markdown: https://agentworkshq.com/docs-content/workflow/pulse_consolidation.md # Pulse: post-run pipeline consolidation Status: **Implemented** (2026-06-21) — Phase 1 (rename) + Phase 2 (backup-always + Pulse-does-low-risk-fixes). Auto-fix is intentionally scoped to low-risk reversible harden; bigger `replan` changes stay with the scheduled auto-improve loop. ## Pulse vs the auto-improve loop (division of labor) These are two tiers over the same Pulse log / Bug-Goal vocabulary — they compose, they don't overlap: - **Pulse** — runs after **every** run (when enabled). Cheap + immediate: **back up → triage → apply a low-risk reversible Bug harden → record a Goal replan *proposal* → notify.** Never runs `replan` or a full improvement pass itself. - **Auto-improve loop** — a **separate schedule** (`/auto-improve`, optimizer mode). Owns the **bigger changes**: batched harden and the **`replan`** tool for structural plan rewrites, acting on the proposals Pulse recorded. Pulse is skipped after these optimizer-mode runs (`scheduler.go`: `WorkshopMode != "optimizer"`), so the two never fight over the same run. ## Problem After a run there are **four disconnected mechanisms**, with different triggers, gating, and reliability: 1. **Post-run monitor** (`runPostRunMonitor`, `scheduler.go:1164`) — opt-in (`post_run_monitor`), a dedicated agent pass that writes the Pulse log (`builder/improve.html` verdict pills + goal card). Its final notify/summary step writes the `builder/card.health.html` dashboard card after harden, artifact review, cost/time, backup, and publish are known. Auto-improve **cadence #1**. 2. **Scheduled harden** — auto-improve **cadence #2**, applies low-risk Bug fixes on its own schedule. 3. **Scheduled replan-proposal** — auto-improve **cadence #3**, proposes Goal changes. 4. **Backup** (`workflowRunBackupDirective`, `background_agents.go:1607`) — a directive appended to the builder's AUTO-NOTIFICATION after a `run_full_workflow` completion. Best-effort steering, **not** a guaranteed step; scheduled runs don't back up themselves (`scheduler.go:1268`). Naming reality: the monitor reference doc already calls `builder/improve.html` "**the Pulse log**". So "monitor" (the pass) and "Pulse" (the log) are two halves of one feature that was never unified in the UI. ## Decisions (user, 2026-06-21) 1. **Pulse everywhere.** The feature/toggle is named **Pulse**. The right-panel tab is **Pulse** (reverts the Phase-3 "History" rename). Internally the monitor pass becomes the "Pulse pass." 2. **Full auto.** Enabling **Pulse** runs the complete post-run loop **every run**: **back up → triage → apply fixes (harden for Bug, replan for Goal)**. The four mechanisms above collapse into this one toggle. ## Target model ``` Pulse (one toggle) → after every run: 1. Back up — always, guaranteed (local-git default = zero-config). Skipped only when source_hash is unchanged (no empty commits). 2. Triage — the current monitor: Bug + Goal verdicts, Pulse log, verdict signal. 3. Fix — Bug → harden (low-risk reliability/contract fixes); Goal → replan (now applied, was propose-only). 4. Notify — one transition notification (unchanged policy). Pulse tab (UI) → the durable record: Timeline (improve.html) + Plan edits (changelog). ``` Backup is no longer a separate steering directive; it's step 1 of the Pulse pass and runs in the scheduler post-run block where the monitor already lives. ## Changes required ### UI (rename — safe, mechanical) - `WorkflowCanvas.tsx`: tab label + titles "History" → **"Pulse"** (revert Phase 3). - `HistoryView.tsx` → `PulseView.tsx` (component rename; keeps Timeline + Plan edits sub-tabs). `PlanChangelogFeed.tsx` unchanged. - `WorkflowToolbar.tsx`: the monitor toggle (`monitorOn`) relabels to **"Pulse"**. ### Behavior (the real change — needs care) - `runPostRunMonitor` → `runPulse` (or keep name, expand prompt). Add **backup as step 1** (guaranteed, source-hash gated) and **fix as step 3** (harden + replan). - Rewrite `guidance/templates/system/post-run-monitor.md`: it is no longer read-only — it backs up, then triages, then applies the safe fixes. Reconcile with the existing harden / replan reference docs so there's one fix contract, not three. - `scheduler.go`: backup runs here for scheduled runs (always). Remove / demote the `workflowRunBackupDirective` steering path so backup isn't double-driven. - `PostRunMonitor` manifest flag stays the gate but is surfaced as "Pulse enabled". ### Risk / open - **Auto-fix every run changes the safety model.** Cadences #1–#3 were deliberately separated (cheap read-only triage vs riskier fixes on a slower schedule). Folding them means a fix can land on every run. Mitigation: keep harden's "low-risk only" contract; keep replan as the heavier action and decide whether it truly applies or still proposes for high-risk plan rewrites. **To confirm before the behavioral rewrite ships.** - Cost: every run now does backup + triage + (sometimes) fix instead of a cheap triage. Acceptable per the "full auto" decision, but worth a source-hash / no-op fast path. ## Backup visibility (2026-06-21) Backup surfaces in three places now: the toolbar status dot + the Backup popup (existing), and — added here — the **Pulse log Run row**. Since Pulse owns the backup, its step-3 Run row records the backup result (`backed up ✓ ` / `unchanged — already backed up` / `backup ✗ `). Doc-only change to `post-run-monitor.md` (step 3) and `review-improve-log.md` (Run kind) — agent-driven, no Go. ## Phasing 1. **Rename to Pulse** ✅ Done (2026-06-21, UI only). `WorkflowCanvas` tab + titles "History" → "Pulse"; `HistoryView.tsx` → `PulseView.tsx`; toolbar "Monitor" button + help popup → "Pulse" (internal vars `monitorOn`/`post_run_monitor` unchanged). Verified: tsc 0, lints clean. 2. **Backup always + Pulse does low-risk fixes** ✅ Done (2026-06-21). Rewrote the Pulse pass prompt in `scheduler.go` (`runPostRunMonitor`) to the 4-step contract (back up → triage → low-risk harden / replan-proposal → notify) and rewrote `guidance/templates/system/post-run-monitor.md` to match (new "0. Back up first" + "3b. Apply the fix" sections; dropped the strict read-only framing; kept step 5 = Notify so the prompt's reference still resolves). Backend build + vet OK. Safety rails kept: **low-risk reversible fixes only** (bigger work → auto-improve loop), and **source-hash gate** so steady runs skip the push. 3. **Unify the backup directive** ✅ Done (2026-06-21). `workflowRunBackupDirective` (the interactive arm) now shares **one backup contract** with Pulse's step 1: same zero-config local-git default, same source-hash skip. So the two arms (Pulse = scheduled, directive = interactive + Pulse-off fallback) can't double-push — whichever runs second sees the source already backed up and skips. Stale `scheduler.go:1268` comment corrected. Build + vet OK; no test pinned the old wording. **Non-goal (decided 2026-06-21):** do NOT move the source-hash skip into deterministic Go code. The whole post-run/backup loop stays agent-driven — the agent reads `backup/status.json` and decides. No Go-side gating coupling. ## Phase 4 — Publish folded into the Pulse loop (2026-06-24) Publish (the public-URL twin of Backup — see `docs/workflow/publish_design.md`) is now a step of the Pulse pass, so a workflow's public dashboard stays current automatically: ``` Pulse → after every run: 1. Back up — guaranteed, source-hash gated (unchanged). 2. Triage — Bug + Goal verdicts, Pulse log (unchanged). 3. Fix — low-risk harden / replan proposal (unchanged). 4. Re-publish — ONLY if publish is configured + enabled; rebuilds from source and redeploys both artifacts. Skipped when publish is off. 5. Notify — one transition notification (unchanged). ``` `post-run-monitor.md` gained a "### 4b. Re-publish (only if publish is on)" section. Like backup, publish is **agent-driven and read-only in the UI**: setup/run/restore/publish all happen in the builder chat via the **`/backup`** and **`/publish`** slash commands; the toolbar popups are status-only. The dead write endpoints (`/workflow/{backup,publish}/{config,run}`) were removed. ### Publish output contract (what `/publish` ships) - **Both artifacts, always** — the baked report **dashboard** (`dashboard.html`) AND the **Pulse log** (`pulse.html`, from `builder/improve.html`), joined by a top-nav `index.html` wrapper (Dashboard | Pulse tabs). Publishing only one is a bug. - **Dark only, matching the app** — the published pages must set **both** theming hooks on ``: `class="dark"` (the app's Tailwind mechanism, `ThemeContext`) **and** `data-theme="dark"` (what report widgets `HtmlWidgetFrame` and the Pulse-log skeleton key on). Setting only `data-theme` left the dashboard light, because report widgets key primarily off the `.dark` class. No toggle, no `prefers-color-scheme` (that follows the viewer's OS). See `[[project_published_page_theme_contract]]`. - **Stage outside the workspace** — deploy CLIs (`netlify`, `vercel`, …) write state (`.netlify/`, `.vercel/`) to their CWD. The workflow folder is writable EXCEPT `planning/`, but the CLI's CWD is often the docs root (above the folder, outside the write allow-list), so it gets rejected. Copy the finished static files to `/tmp/publish-/` and run the deploy from there. ### Plan-edits consolidation (2026-06-24) The toolbar **Plan edits** popup (the granular `planning/changelog/*.json` audit feed) got a **Consolidate** control — drop edits older than 7/30/90/180 days — backed by `POST /workflow/plan-changelog/prune`. The server prunes (deletes whole `changelog-*.json` files older than the cutoff) because `planning/` is shell-guarded from the agent. ## Workflow self-improvement & reporting — system overview Canonical HTML: https://agentworkshq.com/docs/workflow/self_improvement_and_reporting/ Raw Markdown: https://agentworkshq.com/docs-content/workflow/self_improvement_and_reporting.md # Workflow self-improvement & reporting — system overview **Start here.** This is the map of how a workflow keeps itself healthy and moving toward its goals, and how that work is made visible and steerable for the user. It ties together five parts that are otherwise documented separately: **Pulse**, **Auto-improve**, the **scheduled step messages**, **notifications**, and the **Org dashboard**. Each section links to the detailed doc. ## Why this is the critical layer — managing at scale This is the most important subsystem in the product: it's what makes **100+ agents and automations manageable** by a small team. Without it, every workflow needs a human watching it — which caps you at a handful. With it: - each workflow **self-heals** (Pulse fixes operational breakage) and **self-improves toward its goal** (Auto-improve), so routine operation needs no human; - the reporting **rolls everything up** so a human manages **by exception** — the dashboard's triage bar surfaces only what needs attention, notifications fire only on real transitions (broke / recovered / new finding), and big changes wait as **proposals to approve**. You take in a hundred automations at a glance and act only where the system flags it. Span of control is the whole point: the system handles the routine fixing and improving and surfaces only the **exceptions + decisions**, so human effort scales with the number of *exceptions*, not the number of *workflows*. That is the difference between running 5 automations and running 100+. ## 1. Purpose — two jobs 1. **Fix/improve workflows toward their goals.** Keep them *working*, and move them toward *winning*. 2. **Make that legible and steerable.** Report what's happening so the user can *see*, get *alerted*, and *decide* — because the system proposes big changes rather than applying them silently. Everything below is one of those two jobs, or the substrate that connects them. ## 2. The two loops Two loops run per workflow. They answer different questions and own different halves of a workflow's state. | | **Pulse** | **Auto-improve** | |---|---|---| | Job | **FIX** — keep it *working* | **IMPROVE** — make it *win* | | Axis | 🩺 operational + 💵 spend/time ("does it run right, and did it spend sanely?") | 🎯 goal ("is it achieving its goal?") | | Trigger | **after every run** (reactive) | **scheduled** (proactive) | | Autonomy | applies low-risk fixes itself (`harden_workflow`) | **proposal-only** for big changes (replan); user/builder approve | | Statuses | healthy / bug / critical; normal / elevated / missing cost | on-track / at-risk / off-goal | | Code | `runPostRunMonitor` / `postRunMonitorSteps` (`scheduler.go`) | `optimizerScheduleMessages` (`scheduler.go`) | | Guidance | `post-run-monitor.md`, `optimize-playbook.md` | `optimize-playbook.md` + `get_workflow_command_guidance(kind="improve-workflow")` | | Detailed doc | `pulse_consolidation.md` | `auto_improvement_framework.md` | **Pulse step sequence** (one focused turn per step): triage → fix/harden → artifact review → LLM/cost report → backup → publish → notify. **Auto-improve step sequence:** pre-backup → improve → final backup → publish → notify. The improve turn can also **adjust its own cadence** (run more often while actively improving, back off when stable) via `update_workflow_schedule` — see `workflow_scheduling.md`. "Working but off-goal" is a normal, important state: Pulse says it runs fine, Auto-improve says it isn't moving the goal yet. ## 3. The shared substrate (the loops' memory) Both loops read and write **`builder/improve.html`** — the per-workflow Pulse/improve log, newest-first. It *is* the loop's memory: - **Verdict pills** (Bug, Goal), stamped with the run they're as-of. - **Goal card** — each success criterion's Met/Short/At-risk + evidence. - **Decision cards** — each fix (harden/replan) the loop applied. - **Auto-improve major decision cards** — visually distinct decision entries with `Why now`, evidence, change, expected impact, files touched, and remaining risk/gap, so material replans/report/eval/cadence changes do not look like routine Pulse notes. - **Self-verification** — on a later run the loop *confirms the last unconfirmed Decision*: `ok` (cite before→after), `bad` (regressed → reopen a finding), or `flat` (path not hit → stays pending). So "I fixed X last run → re-check X" is built in. - **Open findings** — anchored ids that persist across runs until a fix closes them. - **Human input requests** — structured question cards with status, options/default, and evidence; notifications may point to them, but email is not the source of truth. See `review-improve-log.md` for the log's structure and the confirm-Decision rules. Each loop also writes a compact **dashboard card** in the workflow's own workspace, every run (overwrite), via current workspace write paths such as `diff_patch_workspace_file`: - Pulse final notify/summary step → `builder/card.health.html` (🩺 final post-Pulse status + compact named fields for state/fix/evidence/next, not the full email narrative) - Pulse report step → `builder/card.cost.html` (💵 cost/time status + headline/metric) - Auto-improve → `builder/card.progress.html` (🎯 status + goal + headline) These are served to the UI by `getBuilderDoc(workspace, "card-health"|"card-progress"|"card-cost")` (`auto_improvement_endpoints.go`). ## 4. The reporting / steering surfaces The same verdicts/Decisions/cards the loops produce while fixing are what surface here — *#2 is #1 seen again*, never a separate analysis. - **Notifications — `notify_user`** (active, "you need to know this"). Fans out to connected channels: **Gmail** (`email_html` rich body + `email_body` plain fallback — must be **inline-styled**, Gmail strips `