doc updates

README.md

@@ -2,67 +2,209 @@
 
 A self-hosted **Agentic Engineering Platform (AEP)** — an IDE that drives autonomous AI agents to perform software engineering work on your behalf, running in your own environment.
 
+## Quick Start
+
+```bash
+# Install dependencies
+pnpm install
+
+# Start backend (Terminal 1)
+cd gadget-code
+pnpm dev:backend
+
+# Start frontend (Terminal 2)
+cd gadget-code/frontend
+pnpm dev
+
+# Open browser to https://localhost:5174
+```
+
 ## Projects
 
 | Package        | Role                                                                     |
 | -------------- | ------------------------------------------------------------------------ |
 | `gadget-code`  | Web service — agentic IDE, browser UI, API server                        |
 | `gadget-drone` | Worker process — runs the agentic workflow loop in workspace directories |
-| `@gadget/ai`   | Shared AI API abstraction — Ollama and OpenAI, called by both            |
+| `@gadget/ai`   | Shared AI API abstraction — Ollama and OpenAI                            |
-
+| `@gadget/api`  | Shared TypeScript interfaces — common types across all packages          |
-## AI API Abstraction (`@gadget/ai`)
-
-All AI API calls throughout the Gadget Code ecosystem route through `@gadget/ai`. No consumer code imports Ollama or OpenAI SDKs directly. No consumer code checks `provider.sdk` after the factory call. The shared module translates Gadget Code's internal API contract into whatever provider is configured, and translates responses back to Gadget Code's internal types.
-
-See `packages/ai/README.md` for the full API reference.
-
-## Setup
-
-```bash
-pnpm install
-```
-
-## Build
-
-```bash
-pnpm -r build              # build all packages
-pnpm --filter @gadget/ai build
-pnpm --filter gadget-drone build
-pnpm --filter gadget-code build:backend
-```
-
-## Run
-
-```bash
-# Backend
-pnpm --filter gadget-code dev
-
-# Drone worker (in a project workspace directory)
-pnpm --filter gadget-drone dev
-```
 
 ## Architecture
 
-### @gadget/ai
+```
+┌──────────────┐     ┌──────────────┐     ┌──────────────┐
+│  Browser IDE │────▶│  gadget-code │────▶│ gadget-drone │
+│  (React 19)  │◀────│ (Express 5)  │◀────│  (Worker)    │
+└──────────────┘     └──────────────┘     └──────────────┘
+       │                    │                      │
+       │ Socket.IO          │ MongoDB              │ Files
+       │ JWT Auth           │ Redis                │ Git
+```
 
-AI API calls are handled by `@gadget/ai`, which both projects depend on. This keeps all AI SDK knowledge in one place, and currently implements:
+### How It Works
 
-- [AiApi](./packages/ai/src/api.ts) - abstract base class for all AI APIs/SDKs
+1. **User creates a project** in the browser IDE
-- [OllamaAiApi](./packages/ai/src/ollama.ts) - Ollama API implementation
+2. **User selects a drone** instance to work on the project
-- [OpenAiApi](./packages/ai/src/openai.ts) - OpenAI API implementation
+3. **User enters a prompt** in the Chat Session view
+4. **Prompt flows** IDE → Web → Drone via Socket.IO
+5. **Drone executes** the Agentic Workflow Loop (AWL)
+6. **Streaming response** flows back: thinking → response → tool calls
+7. **Results persist** in MongoDB as ChatTurn records
 
-### gadget-drone
+## AI Provider Setup
 
-gadget-drone is a headless process runs on end-user machines, connecting via Socket.IO to gadget-code, to receive and execute work orders for the agentic workflow loop.
+Before using AI features, add a provider via CLI:
 
-At startup, gadget-drone examines `process.cwd()` to determine if it's a workspace directory, and if so, starts a worker process that connects to gadget-code and waits for work orders. It processes work orders in project directories in the gadget-drone workspace directory, and communicates events, status, and results to the IDE via gadget-code's web services and Socket.IO.
+```bash
+# Add Ollama provider (no API key needed)
+cd gadget-code
+pnpm cli provider add "Local Ollama" ollama http://localhost:11434
 
-gadget-drone never connects directly to MongoDB or Redis — it communicates entirely through the Gadget Code API.
+# Add OpenAI provider
+pnpm cli provider add "OpenAI" openai https://api.openai.com $OPENAI_API_KEY
 
-### gadget-code
+# List providers
+pnpm cli provider ls
 
-gadget-code runs on server infrastructure (MongoDB, Redis, etc.) and serves the browser-based IDE. The IDE connects to gadget-code via Socket.IO to send and receive commands in chat sessions to gadget-drone.
+# Probe provider for models (discovers available models)
+pnpm cli provider probe <provider-id>
+```
 
-gadget-code can be stacked on a single host for local development, and can achieve significant scale on a single host. It can also be deployed in tiers, potentially made of clusters, and the web tier can be horizontally scaled for production use with high availability.
+## User Management
 
-Libraries such as [redis-adapter](https://github.com/socketio/socket.io-redis-adapter) and [redis-emitter](https://github.com/socketio/socket.io-redis-emitter) are used for message routing and distribution in the Socket.IO library, handling the real-time message routing among gadget-code, gadget-drone, and the IDE running in the browser.
+```bash
+# Add a user
+pnpm cli user add user@example.com password123 "Display Name"
+
+# Grant admin rights
+pnpm cli admin grant user@example.com
+
+# Reset password
+pnpm cli user password user@example.com newpassword
+```
+
+## Development
+
+### Prerequisites
+
+- Node.js 22+
+- pnpm 10+
+- MongoDB on `localhost:27017`
+- Redis on `localhost:6379`
+- SSL certificates in `ssl/` directory (for dev servers)
+
+### Build
+
+```bash
+# Build all packages
+pnpm -r build
+
+# Build specific package
+pnpm --filter @gadget/ai build
+pnpm --filter gadget-drone build
+pnpm --filter gadget-code build
+```
+
+### Run Development Servers
+
+```bash
+# Backend (gadget-code directory)
+cd gadget-code
+pnpm dev:backend
+# Runs on https://localhost:3443
+
+# Frontend (gadget-code/frontend directory)
+cd gadget-code/frontend
+pnpm dev
+# Runs on https://localhost:5174
+
+# Drone worker (in a workspace directory)
+cd ~/my-gadget-workspace
+pnpm --filter @gadget/drone dev
+```
+
+### Testing
+
+```bash
+# Unit tests (gadget-code directory)
+cd gadget-code
+pnpm test
+
+# E2E tests with Playwright (requires running dev servers)
+npx playwright test tests/e2e/project-manager.test.ts
+npx playwright test tests/e2e/chat-session.test.ts
+
+# Seed test data for E2E tests
+npx tsx scripts/seed-test-drones.ts
+```
+
+## Key Features
+
+### Project Manager
+
+- Create and manage projects
+- Select available drones (filters out offline)
+- View chat session history
+- Create new chat sessions with provider/model selection
+
+### Chat Session View
+
+- Real-time streaming responses
+- Collapsible thinking content
+- Tool call summaries
+- Model/mode selection per session
+- Session statistics (tool calls, file ops, subagents)
+
+### Workspace Management
+
+- Each drone manages a workspace directory
+- Projects cloned into workspace by slug
+- Crash recovery via `.gadget/` directory
+- Workspace persistence across restarts
+
+## AI API Abstraction
+
+All AI calls route through `@gadget/ai`. No consumer code imports Ollama or OpenAI SDKs directly.
+
+```typescript
+// Correct: Use the factory
+import { createAiApi } from '@gadget/ai';
+const ai = createAiApi(providerConfig);
+
+// Incorrect: Don't import SDKs directly
+import { Ollama } from 'ollama'; // ❌
+```
+
+See [`packages/ai/README.md`](./packages/ai/README.md) for the full API reference.
+
+## Documentation
+
+- [Architecture Overview](./docs/architecture-stats.md)
+- [Socket Protocol](./docs/socket-protocol.md)
+- [Workspace Management](./docs/gadget-workspace.md)
+- [Drone Documentation](./gadget-drone/docs/gadget-drone.md)
+- [UI Design Guide](./gadget-code/docs/ui-design-guide.md)
+
+## Monorepo Structure
+
+```
+gadget/
+├── packages/ai/           # @gadget/ai — AI API abstraction
+├── packages/api/          # @gadget/api — Shared interfaces
+├── gadget-code/           # Web service + browser IDE
+│   ├── src/               # Backend (Express, Socket.IO, Mongoose)
+│   ├── frontend/          # Frontend (React, Vite, Tailwind)
+│   └── tests/             # Unit + E2E tests
+├── gadget-drone/          # Worker process
+│   ├── src/               # Drone implementation
+│   └── docs/              # Drone documentation
+└── docs/                  # Architecture & protocol docs
+```
+
+## License
+
+Apache 2.0 — See [LICENSE](./LICENSE) for details.
+
+---
+
+**Status:** Production-ready foundation with complete Chat Session UI  
+**Last Updated:** April 29, 2026