The Current Approach for the problem comprises of server-client architecture, the extension resides on client side and the API acts as the server. The system comprises of one single agent that resides on server side. The agent analyses the carefully curated DOM structure, web UI’s screenshot along with the user’s query to select its next course of action.

Tools:

The agent responds only in terms of tools. Agent has a total of 19 tools to use. Refer the below document for tools.

🛠️ Browser Automation Tools: Implementation Guide

Working:

System Architecture

The browser automation system operates through a sophisticated client-server architecture that seamlessly bridges natural language understanding with precise browser control:

Client Side (Chrome Extension):

• 13 Modular Components: Each handling specific aspects of browser automation
• Real-time State Capture: Continuous monitoring of browser state and DOM changes
• Multi-modal Data Collection: Screenshots, DOM snapshots, and metadata extraction
• Tool Execution Engine: Direct browser API interaction for automation tasks

Server Side (FastAPI Backend):

• Single AI Agent: Gemini-powered decision-making engine
• Conversation Memory: Maintains context across multi-step workflows
• Usage Analytics: Token tracking and cost optimization
• Error Recovery: Intelligent fallback and retry mechanisms

Core Working Principles