Nova CoPilot for Screens is an agentic automation system that learns real user workflows directly from UI interactions and converts them into reliable Nova Act agents with human-in-the-loop safety.
Unlike traditional RPA, Nova CoPilot combines reasoning, multimodal embeddings, and adaptive execution to automate legacy portals and SaaS tools without APIs.
Most automation tools require manual workflow design.
Nova CoPilot introduces a Watch-First Automation Model:
- Observe real user sessions
- Infer repeatable workflows using Nova reasoning
- Generate agent policies
- Execute with explainability and confidence scoring
This shifts automation from scripting → learning.
Browser Recorder Extension │ ▼ Session Normalization Service (DOM + Screen Snapshot Encoding) │ ▼ Nova Multimodal Embeddings (Task Similarity + UI Understanding) │ ▼ Planner Agent (Nova 2 Lite)
Workflow abstraction
Constraint generation
Error path reasoning │ ▼ Executor Agent (Nova Act)
Stateful browser actions
Adaptive selectors
Retry logic │ ▼ Observability Layer Action logs Confidence scores Human approval signals
Captures structured session traces:
- DOM tree context
- Screen regions
- Action sequences
Transforms sessions into generalized workflows:
Example abstraction:
IF user navigates CRM → searches email → updates status THEN create RefundApprovalWorkflow
Executes workflows with:
- semantic element targeting
- layout-shift tolerance
- state recovery
- Reasoning over session clusters
- Workflow generalization
- Dynamic planning
- DOM + screenshot semantic alignment
- UI change robustness
- Reliable browser automation
- Stateful execution across pages
- Voice-triggered agent execution
- Spoken explanations
Automation maturity levels:
-
Ghost Mode
- Shows predicted actions
- Confidence scoring
-
Supervised Mode
- Human approval checkpoints
-
Auto Mode
- Low-risk execution with audit logs
🌐 App Link (Live Demo)
https://lovable-agent-scribe.lovable.app
🎬 Demo Video
https://youtu.be/GlU6srYAs-o
📝 Builder.aws Blog Post
https://builder.aws.com/content/39t0fl7Me50yX1wwFNm55ggNgNA/from-click-fatigue-to-community-impact-building-nova-copilot-for-screens-with-amazon-nova-by-syessasvini
Scenario: Refund approval automation
Steps:
- Record workflow once
- Nova infers reusable pattern
- Agent generated
- Ghost mode previews actions
- Auto mode executes batch refunds
Metrics dashboard displays:
- Time saved/run
- Error rate
- Automation confidence trend
- Screen-native agent learning (not prompt scripting)
- Multimodal UI understanding instead of static selectors
- Agent planning separated from execution
- Human-aligned trust model via ghost mode
Frontend:
- React + Extension API
Backend:
- Python / Node.js
- AWS Lambda
- API Gateway
AI:
- Amazon Nova 2 Lite
- Nova Act
- Nova Multimodal Embeddings
- Adaptive selector embeddings
- Self-healing agent policies
- Federated workflow learning
MIT