gadget/docs/architecture.md
2026-05-07 00:10:57 -04:00

731 lines
25 KiB
Markdown

# Gadget Code Architecture
**Date:** April 29, 2026
**Status:****CHAT SESSION UI COMPLETE** - Full end-to-end prompt flow operational
## Executive Summary
Gadget Code is a fully functional Agentic Engineering Platform with complete Chat Session UI, AI provider management, and end-to-end prompt processing. Users can create projects, select drones, configure AI providers/models, and submit prompts that flow through the complete agentic workflow loop.
**Key Capabilities:**
- ✅ Complete Socket.IO communication (IDE↔Web↔Drone)
- ✅ AI Provider CLI management (add/list/probe/remove)
- ✅ Chat Session REST API with full CRUD
- ✅ Project Manager with drone selection & chat session lists
- ✅ Chat Session View with streaming responses
- ✅ Workspace persistence for crash recovery
- ✅ 21 unit tests passing
---
## System Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Gadget Code Ecosystem │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Browser IDE │────▶│ gadget-code │────▶│ gadget-drone │ │
│ │ (React 19) │◀────│ (Express 5) │◀────│ (Worker) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ │ Socket.IO │ MongoDB │ Files │
│ │ JWT Auth │ Redis Sessions │ Git │
│ │ │ │ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Chat View │ │ REST API │ │ AWL Loop │ │
│ │ Streaming │ │ /projects │ │ Tool Calls │ │
│ │ Tool Calls │ │ /sessions │ │ File Ops │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
### Core Components
| Component | Technology | Role |
| ------------------- | -------------------------------- | ----------------------------------------------- |
| **Browser IDE** | React 19 + Vite 8 + Tailwind 4 | Agentic development environment |
| **gadget-code:web** | Express 5 + Socket.IO + Mongoose | API server, session management, routing |
| **gadget-code:ide** | TypeScript + React Router | Chat sessions, project management, file editing |
| **gadget-drone** | TypeScript + Socket.IO client | Agentic Workflow Loop executor |
| **@gadget/ai** | TypeScript abstraction | Unified AI API (Ollama/OpenAI) |
| **@gadget/api** | TypeScript interfaces | Shared type definitions |
---
## Message Flow
### Complete Prompt Journey
```
User Types Prompt
┌──────────────────┐
│ Chat Session View│
│ - Prompt Input │
│ - submitPrompt │
└────────┬─────────┘
│ Socket: submitPrompt(content)
┌──────────────────┐
│ CodeSession │
│ - Create ChatTurn│
│ - Track drone │
└────────┬─────────┘
│ Socket: processWorkOrder(project, session, turn)
┌──────────────────┐
│ DroneSession │
│ - Route to drone│
└────────┬─────────┘
│ Socket: processWorkOrder(...)
┌──────────────────┐
│ gadget-drone │
│ - AgentService │
│ - AWL Loop │
└────────┬─────────┘
│ Processing...
├─► thinking(content) ────┐
├─► response(content) ────┤
├─► toolCall(...) ────────┤
└─► workOrderComplete() ──┤
┌──────────────────┐
│ DroneSession │
│ - Update Turn │
│ - Route to IDE │
└────────┬─────────┘
│ Socket: thinking/response/toolCall
┌──────────────────┐
│ Chat Session View│
│ - Stream display│
│ - Update UI │
└──────────────────┘
```
---
## AI Provider Architecture
### Provider Management Flow
```
Admin CLI Database Runtime
│ │ │
├─ provider add ───────▶│ │
│ (name, sdk, url) │ │
│ │ │
├─ provider probe ─────▶│ AiProvider model │
│ │ - Fetch models │
│ │ - Discover caps │
│ │ - Update in-place │
│ │ │
│ ├────────────────────▶│ createAiApi()
│ │ │ - SDK factory
│ │ │ - Ollama/OpenAI
│ │ │
│ │◀────────────────────┤
│ │ Runtime config │
│ │ - _id, name, sdk │
│ │ - baseUrl, apiKey │
│ │ │
└─ provider ls ────────▶│ List providers │
│ (hide apiKey) │
```
### Provider Data Model
```typescript
interface IAiProvider {
_id: ObjectId;
name: string;
apiType: "ollama" | "openai";
baseUrl: string;
apiKey: string; // Encrypted, select: false
enabled: boolean;
models: IAiModel[];
lastModelRefresh: Date;
}
interface IAiModel {
id: string;
name: string;
parameterCount?: number;
parameterLabel?: string; // "7b", "70b"
contextWindow?: number;
capabilities: {
canCallTools: boolean;
hasVision: boolean;
hasEmbedding: boolean;
hasThinking: boolean;
isInstructTuned: boolean;
};
settings?: {
temperature?: number;
topP?: number;
topK?: number;
numCtx?: number;
};
}
```
---
## Chat Session Architecture
### Session Lifecycle
```
User Creates Session
┌────────────────────────────┐
│ POST /api/v1/chat-sessions │
│ - projectId │
│ - providerId │
│ - selectedModel │
│ - mode (plan/build/..) │
└───────────┬────────────────┘
┌────────────────────────────┐
│ ChatSession Document │
│ - _id │
│ - user, project │
│ - provider, model │
│ - mode, stats │
│ - pins[] │
└───────────┬────────────────┘
┌────────────────────────────┐
│ Chat Session View │
│ - Messages list │
│ - Prompt input │
│ - Session sidebar │
│ - Model selector │
└───────────┬────────────────┘
┌────────────────────────────┐
│ User Submits Prompt │
│ - submitPrompt event │
│ - Creates ChatTurn │
│ - Routes to drone │
└───────────┬────────────────┘
┌────────────────────────────┐
│ Streaming Response │
│ - thinking (collapsible) │
│ - response (streaming) │
│ - toolCall (summaries) │
│ - workOrderComplete │
└────────────────────────────┘
```
### Chat Turn Structure
```typescript
interface IChatTurn {
_id: ObjectId;
createdAt: Date;
user: IUser | ObjectId;
project: IProject | ObjectId;
session: IChatSession | ObjectId;
provider: IAiProvider | ObjectId;
llm: string; // Model ID
mode: ChatSessionMode;
status: "processing" | "finished" | "error";
prompts: {
user: string;
system?: string;
};
thinking?: string;
response?: string;
toolCalls: IChatToolCall[];
subagents: IChatSubagentProcess[];
stats: IChatTurnStats;
}
interface IChatToolCall {
callId: string;
name: string;
parameters?: string;
response?: string;
}
interface IChatTurnStats {
toolCallCount: number;
inputTokens: number;
thinkingTokenCount: number;
responseTokens: number;
durationMs: number;
durationLabel: string; // "hh:mm:ss"
}
```
---
## Workspace Architecture
### Project Manager Layout
```
┌──────────────────────────────────────────────────────────────┐
│ Header Bar │
├──────────┬───────────────────────┬───────────────────────────┤
│ │ │ │
│ Projects │ Project Inspector │ Right Sidebar │
│ │ │ │
│ List │ - Name, Slug │ ┌─────────────────────┐ │
│ │ - Git URL │ │ Available Drones │ │
│ [proj-1] │ - Status, Created │ │ (40% of space) │ │
│ [proj-2] │ - Delete button │ │ │ │
│ │ │ │ drone-alpha ● │ │
│ │ │ │ [Select] │ │
│ │ │ │ │ │
│ │ │ │ drone-beta ○ │ │
│ │ │ │ [Select] │ │
│ │ │ │ │ │
│ │ │ └─────────────────────┘ │
│ │ │ ┌─────────────────────┐ │
│ │ │ │ Chat Sessions │ │
│ │ │ │ (60% of space) │ │
│ │ │ │ │ │
│ │ │ │ Session 1 │ │
│ │ │ │ [Open] │ │
│ │ │ │ │ │
│ │ │ │ Session 2 │ │
│ │ │ │ [Open] │ │
│ │ │ │ │ │
│ │ │ └─────────────────────┘ │
│ │ │ ┌─────────────────────┐ │
│ │ │ │ [New Chat Session] │ │
│ │ │ │ (fixed at bottom) │ │
│ │ │ └─────────────────────┘ │
├──────────┴───────────────────────┴───────────────────────────┤
│ Status Bar │
└──────────────────────────────────────────────────────────────┘
```
### Drone Filtering Rules
- **Shown in list:** `status !== 'offline'` (available + busy only)
- **Selectable:** `status === 'available'` only
- **Busy drones:** Visible but Select button disabled
- **Offline drones:** Completely filtered out
### Chat Session Creation Flow
1. User selects project → loads Project Manager
2. System fetches available drones (filters offline)
3. User clicks "Select" on a drone → stores in state
4. User clicks "[New Chat Session]" → opens modal
5. Modal shows:
- Provider dropdown (from `/api/v1/providers`)
- Model dropdown (from selected provider's models)
- Mode selector (Plan/Build/Test/Ship/Dev)
- Optional session name
6. User submits → `POST /api/v1/chat-sessions`
7. System creates session → navigates to `/projects/:projectId/chat-session/:sessionId`
---
## Chat Session View Design
### Layout Specification
```
┌─────────────────────────────────────────────────────────────┐
│ Work Area (flex-1) │ Session Sidebar (w-80) │
├──────────────────────────────┤ │
│ │ Chat: Session Name │
│ Chat Messages │ ID: abc123... │
│ - User prompts (right) │ Model: llama3.2 (Ollama) ▼ │
│ - Assistant responses │ Mode: BUILD ▼ │
│ - Thinking (collapsible) │ │
│ - Tool calls (1-line) ├──────────────────────────────┤
│ │ TC │ FO │ SA │
│ │ 12 │ 0 │ 0 │
├──────────────────────────────┤ │
│ [Prompt input ][Send] │ Project: my-project │
└──────────────────────────────┴──────────────────────────────┘
```
### Message Display Rules
- **User messages:** Right-aligned, brand color background
- **Assistant responses:** Left-aligned, dark background
- **Thinking content:** Collapsible with toggle, shows char count
- **Tool calls:** One-line summary in chat (`⚡ toolName ✓`)
- **Tool call details:** Click to open inspector (future)
- **Auto-scroll:** Always scroll to bottom on new message
### Session Sidebar Features
- **Model selector:** Dropdown showing `Model Name (Provider Name)`
- **Mode selector:** Plan/Build/Test/Ship/Dev
- **Stats panel:**
- TC = Tool Calls
- FO = File Operations (future)
- SA = Subagents (future)
- **Session settings:** Cog icon → edit name, provider, mode
---
## CLI Commands
### Provider Management
```bash
# Add a provider (auto-probes for models)
pnpm cli provider add "My Ollama" ollama http://localhost:11434
pnpm cli provider add "OpenAI" openai https://api.openai.com $OPENAI_API_KEY
# List all providers
pnpm cli provider ls
# Enable/disable a provider
pnpm cli provider status <provider-id> active
pnpm cli provider status <provider-id> inactive
# Remove a provider
pnpm cli provider remove <provider-id>
# Probe provider for models (updates in-place)
pnpm cli provider probe <provider-id>
```
### User Management
```bash
# Add user
pnpm cli user add user@example.com password123 "Display Name"
# Reset password
pnpm cli user password user@example.com newpassword
# Ban/unban user
pnpm cli user ban user@example.com
pnpm cli user unban user@example.com
# Remove user
pnpm cli user remove user@example.com
```
### Admin Commands
```bash
# Grant admin rights
pnpm cli admin grant user@example.com
# Revoke admin rights
pnpm cli admin revoke user@example.com
```
---
## Testing Strategy
### Unit Tests
**Location:** `gadget-code/tests/**/*.test.ts`
**Coverage:**
- `code-session.test.ts` - Prompt submission flow (9 tests)
- `drone-session.test.ts` - Event routing (12 tests)
- Total: 21 tests passing
**Run:**
```bash
cd gadget-code
pnpm test
```
### E2E Tests (Playwright)
**Location:** `gadget-code/tests/e2e/**/*.test.ts`
**Test Scenarios:**
1. **Project Manager Layout** - Three-column layout verification
2. **Drone Filtering** - Only available/busy drones shown
3. **Drone Selection** - Select buttons functional
4. **New Session Button** - Visible, disabled until drone selected
5. **Chat Session Creation** - Modal flow with provider/model selection
6. **Prompt Submission** - End-to-end prompt flow
7. **Response Streaming** - Thinking, response, tool calls displayed
**Run:**
```bash
# Prerequisites: MongoDB, Redis, dev servers running
cd gadget-code
# Start backend (in gadget-code directory)
pnpm dev:backend
# Start frontend (in gadget-code/frontend directory)
pnpm dev
# Run tests (in separate terminal)
npx playwright test tests/e2e/project-manager.test.ts
npx playwright test tests/e2e/chat-session.test.ts
```
**Test Data Seeding:**
```bash
# Seed mock drones for testing
cd gadget-code
npx tsx scripts/seed-test-drones.ts
```
This creates:
- 2 available drones
- 1 busy drone
- 2 offline drones (filtered from UI)
---
## Development Setup
### Prerequisites
- Node.js 22+
- pnpm 10+
- MongoDB on `localhost:27017`
- Redis on `localhost:6379`
- SSL certificates in `ssl/` directory
### Installation
```bash
# Install all workspace packages
pnpm install
# Build all packages
pnpm -r build
```
### Running Development Servers
```bash
# Terminal 1: Backend (gadget-code directory)
cd gadget-code
pnpm dev:backend
# Runs on https://localhost:3443
# Terminal 2: Frontend (gadget-code/frontend directory)
cd gadget-code/frontend
pnpm dev
# Runs on https://localhost:5174
# Terminal 3: Drone (separate workspace directory)
cd ~/my-gadget-workspace
pnpm --filter gadget-drone dev
# Registers with platform, waits for work orders
```
### Building for Production
```bash
# Build all packages
pnpm -r build
# Build backend only
cd gadget-code
pnpm build:backend # Outputs to dist/
# Build frontend only
cd gadget-code
pnpm build:frontend # Outputs to frontend/dist/
```
---
## Key Architectural Decisions
### 1. Socket.IO Only (No Bull Queue)
**Decision:** All message routing uses Socket.IO with directed delivery.
**Rationale:**
- Better performance for real-time agentic workflows
- Eliminates Redis dependency for end users
- Simpler deployment model
**Recovery:** Workspace persistence via `.gadget/` directory
### 2. CLI-Only Administration
**Decision:** No admin UI - all administrative functions via CLI.
**Rationale:**
- Reduces attack surface
- Simpler codebase
- Admins comfortable with CLI
**Commands:** User management, provider management, admin rights
### 3. AI API Abstraction
**Decision:** All AI calls route through `@gadget/ai`. No direct SDK imports.
**Rationale:**
- Single source of truth for AI interfaces
- Easy to add new providers
- Consistent error handling
### 4. Workspace Persistence
**Decision:** Drones maintain `.gadget/` directory for crash recovery.
**Rationale:**
- No external dependencies for drones
- Survives restarts
- Enables retry routing
### 5. Model Selection in UI
**Decision:** Users select provider + model per chat session, changeable anytime.
**Rationale:**
- Flexibility for different tasks
- Cost optimization
- Model experimentation
---
## File Structure
```
gadget/
├── packages/
│ ├── ai/ # @gadget/ai - AI API abstraction
│ │ ├── src/
│ │ │ ├── api.ts # Core interfaces
│ │ │ ├── ollama.ts # Ollama implementation
│ │ │ ├── openai.ts # OpenAI implementation
│ │ │ └── index.ts # Factory function
│ │ └── README.md
│ └── api/ # @gadget/api - Shared types
│ └── src/
│ └── interfaces/ # TypeScript interfaces
├── gadget-code/ # Web service + IDE
│ ├── src/
│ │ ├── web-app.ts # Express app entry
│ │ ├── web-cli.ts # CLI entry
│ │ ├── controllers/
│ │ │ └── api/v1/
│ │ │ ├── chat-session.ts
│ │ │ ├── provider.ts
│ │ │ ├── project.ts
│ │ │ └── drone.ts
│ │ ├── services/
│ │ │ ├── chat-session.ts
│ │ │ ├── socket.ts
│ │ │ └── drone.ts
│ │ ├── models/
│ │ │ ├── chat-session.ts
│ │ │ ├── chat-turn.ts
│ │ │ └── ai-provider.ts
│ │ └── lib/
│ │ ├── code-session.ts
│ │ └── drone-session.ts
│ ├── frontend/
│ │ └── src/
│ │ ├── App.tsx
│ │ ├── pages/
│ │ │ ├── ProjectManager.tsx
│ │ │ ├── ChatSessionView.tsx
│ │ │ └── Home.tsx
│ │ ├── components/
│ │ │ ├── Header.tsx
│ │ │ └── StatusBar.tsx
│ │ └── lib/
│ │ ├── api.ts
│ │ └── socket.ts
│ ├── tests/
│ │ ├── code-session.test.ts
│ │ ├── drone-session.test.ts
│ │ └── e2e/
│ │ ├── project-manager.test.ts
│ │ └── chat-session.test.ts
│ └── scripts/
│ └── seed-test-drones.ts
├── gadget-drone/ # Worker process
│ ├── src/
│ │ ├── gadget-drone.ts
│ │ └── services/
│ │ ├── agent.ts # AWL implementation
│ │ ├── ai.ts # AI service
│ │ ├── platform.ts # Platform registration
│ │ └── workspace.ts # Workspace persistence
│ └── docs/
│ └── gadget-drone.md
└── docs/
├── architecture-stats.md # This document
├── socket-protocol.md
└── gadget-workspace.md
```
---
## Current Status
### ✅ Complete
- [x] Socket.IO communication (IDE↔Web↔Drone)
- [x] AI Provider CLI management
- [x] Chat Session REST API
- [x] Project Manager UI (3-column layout)
- [x] Chat Session View UI
- [x] Drone selection & filtering
- [x] Model selection in sidebar
- [x] Workspace persistence
- [x] Crash recovery
- [x] Unit tests (21 passing)
- [x] E2E test framework
### 🔄 In Progress
- [ ] E2E tests for complete flow
- [ ] Tool call inspectors
- [ ] File browser integration
- [ ] Session settings modal
### 📋 Future
- [ ] File operations (FO stats)
- [ ] Subagent support (SA stats)
- [ ] Model capability discovery
- [ ] Streaming token counts
- [ ] Session pins/bookmarks
---
**Last Updated:** April 29, 2026
**Branch:** `feature/socket-protocol`
**Status:** Production-ready foundation with complete Chat Session UI