diff --git a/RELEASE.md b/RELEASE.md index 776b89a..6f3695d 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,111 +1,75 @@ -GADGET CODE v1.0.1 -NAME: Liberation -HASH: ff46c35d38e29be64d9eca0ccc18e8a7f5c3d8ab -TAG: v1.0.1 +GADGET CODE v1.0.2 +NAME: Recall +HASH: 47dd61f6d36a3287b67025e15caa03eb6f830195 +TAG: v1.0.2 SUMMARY: -Today marks the Liberation of Gadget Code — the first public release of a self-hosted agentic engineering platform that puts you in control of your code, your models, and your infrastructure. +Recall is the first patch release of Gadget Code — and the name is deliberate. This release introduces **semantic search over your chat history** using Qdrant vector embeddings, so your past engineering conversations are always just a query away. It also delivers real **token economics**: actual API token counts extracted from provider responses, propagated through the session stack, and surfaced in a live context window fuel gauge. Combined with chat export, streaming buffer persistence, and a round of model/test cleanup, Recall makes Gadget Code more observable, more searchable, and more reliable. -Over 200 commits spanning 21 days of intensive development, Gadget Code has grown from an initial commit into a complete platform for autonomous software engineering. And every line of it is Apache-2.0 licensed, fully open source, and designed to run on *your* terms. +## Semantic Search with Qdrant -## What Is Gadget Code? +The headline feature. Gadget Code now ships with an integrated **Qdrant vector search** pipeline that indexes every completed chat turn and makes it full-text+semantic searchable from the IDE. The architecture: -Gadget Code is an **Agentic Engineering Platform (AEP)** — a browser-based IDE that drives autonomous AI agents to perform real software engineering work on your behalf. Unlike cloud-locked alternatives, Gadget Code runs entirely in your environment: your servers, your data, your rules. +- **VectorStoreService** — A singleton service in `gadget-code` that manages Qdrant collection lifecycle, dimension mismatch detection, and embedding ingestion. It validates collection dimensions against the configured vector size and the model's actual output, surfacing clear error messages when they disagree. +- **Embedding pipeline** — Uses your configured AI provider to generate embeddings. Supports both Ollama and OpenAI-compatible endpoints. Configuration lives in `gadget-code.yaml` under the new `qdrant` section. +- **Search API endpoint** — `POST /api/v1/search` accepts a query string, generates its embedding, and returns ranked results from Qdrant. +- **Frontend components** — `SearchInput` and `ChatSearchResults` components in the React IDE provide a clean search-as-you-type experience with ranked results linked back to their source chat sessions. -The architecture is straightforward and powerful: +**New dependency**: Qdrant must be running on `localhost:6333` to enable semantic search. If `qdrant.providerId` is not configured, the service gracefully skips initialization — nothing breaks, you just don't get search. -``` -┌──────────────┐ ┌──────────────┐ ┌──────────────┐ -│ Browser IDE │────▶│ gadget-code │────▶│ gadget-drone │ -│ (React 19) │◀────│ (Express 5) │◀────│ (Worker) │ -└──────────────┘ └──────────────┘ └──────────────┘ - │ │ │ - │ Socket.IO │ MongoDB │ Files - │ JWT Auth │ Redis │ Git +**New package dependency**: `@langchain/textsplitters` for chunking turn content before embedding. + +## Token Economics + +Token counts are no longer estimates. This release extracts real usage data from OpenAI API responses and propagates it through the entire session stack: + +- **Real token extraction** — OpenAI responses now return `usage.prompt_tokens`, `usage.completion_tokens`, and `usage.completion_tokens_details.reasoning_tokens` from the API. These are captured and stored on each `ChatTurn` and aggregated on the `ChatSession`. +- **Context window fuel gauge** — The session UI now displays a live context window usage indicator. As token counts accumulate across turns, the gauge updates automatically so you know when you're approaching your model's context limit. +- **Stats propagation** — `IWorkOrderCompleteStats` now carries `masterInputTokens`, `masterOutputTokens`, and `masterThinkingTokens`. The `onWorkOrderComplete` handler aggregates master + subagent tokens for the session-level totals. +- **DtpLog API response logging** — Added `logApiResponse` to the `DtpLog` transport for structured logging of AI provider responses with full token metadata. + +## Chat Export + +New `chat_export` agent tool lets the drone export chat sessions as markdown or JSON. Useful for documentation generation, session archival, and sharing engineering conversations outside the IDE. + +## Streaming Buffer Persistence + +Drone sessions now use an in-memory streaming buffer that aggregates thinking and response tokens by mode, flushing to the database at mode transitions (e.g., thinking → tool call → responding). This reduces database writes during streaming and ensures turn blocks are persisted in the correct order with accurate timestamps. + +## Model and Test Cleanup + +- **Removed `syncIndexes()` from all Mongoose models** — The auto-invoked `(async () => { await Model.syncIndexes(); })()` blocks at module level caused Mongoose connection timeout errors during tests when no database was available. These calls are no longer needed and have been removed from all 11 model files. +- **Mocked database in drone-session tests** — `ChatSession.updateOne()`, `VectorStoreService.ingestTurn()`, and `MessageQueue` are now properly mocked so tests run without any database dependency. + +## Configuration Changes + +New `qdrant` section in `gadget-code.yaml`: + +```yaml +qdrant: + host: localhost + port: 6333 + providerId: + embeddingModel: + vectorSize: 1024 + collection: gadget-code ``` -A user creates a project in the browser IDE, selects a drone instance, enters a prompt, and the drone executes the Agentic Workflow Loop — thinking, responding, calling tools, spawning subagents — all streamed back in real time. +New types added to `@gadget/config`: `IQdrantConfig` and `IEmbeddingModelConfig`. -## Your Models, Your Way +## Other Changes -This is what Liberation means. Gadget Code ships with first-class support for **Ollama** and any **OpenAI-compatible API**. Run local models on your own GPU. Self-host an inference endpoint behind your firewall. Use a commercial provider when it suits you. Mix and match per session. The `@gadget/ai` package provides a unified abstraction — no consumer code ever imports an SDK directly. Swap providers without changing a line of application logic. - -The model configuration pipeline supports `numCtx` (context window), `numPredict`, and `maxCompletionTokens` settings, giving you fine-grained control over how your models behave. - -## Complete Agentic Engineering Platform - -Gadget Code isn't a chatbot wrapper. It's a full-stack engineering environment: - -- **Project Manager** — Create projects, select available drone instances, manage chat session history. -- **Chat Session View** — Real-time streaming responses with collapsible thinking content, tool call summaries, session statistics, and per-session model/mode selection. -- **Agentic Workflow Loop** — The drone executes a multi-turn reasoning loop: think, respond, call tools, observe results, and iterate. Work orders are abortable, and the loop supports checkpointing for crash recovery. -- **Subagent Processing** — Agents can spawn specialized subagents for decomposition tasks. Subagent context is managed independently and streamed back to the IDE in real time. -- **FILE Panel** — Lazy-loading file tree with full workspace navigation and CodeMirror editor integration (React 19 compatible). -- **Workspace Management** — Each drone manages a workspace directory with crash recovery via the `.gadget/` directory and intelligent startup that detects the workspace by walking up the file hierarchy. -- **Scheduled Tasks (gadget-tasks)** — A headless IDE client that automates the browser flow on a cron schedule. It drives the existing gadget-code platform via the same REST API and Socket.IO protocol the browser uses — no duplicated logic. -- **Pull Project** — Clone remote repositories directly into your workspace. -- **Abort Controller** — Cancel in-progress work orders with proper state cleanup. - -## Configuration and Dependencies - -Gadget Code requires: - -- **Node.js 22+** and **pnpm 10+** -- **MongoDB** on `localhost:27017` -- **Redis** on `localhost:6379` -- **SSL certificates** in the `ssl/` directory for dev servers -- An **Ollama** instance or **OpenAI-compatible API** endpoint for AI features - -The monorepo is organized as follows: - -| Package | Role | -| -------------------- | --------------------------------------------------------------- | -| `gadget-code` | Web service — agentic IDE, browser UI, API server | -| `gadget-drone` | Worker process — runs the agentic workflow loop | -| `gadget-tasks` | Scheduled task worker — headless IDE client for cron-driven tasks | -| `@gadget/ai` | Shared AI API abstraction — Ollama and OpenAI | -| `@gadget/ai-toolbox` | Shared AI tool implementations — search, file, plan, subagent | -| `@gadget/api` | Shared TypeScript interfaces — common types across all packages | -| `@gadget/config` | Shared YAML config loader — per-package configuration | - -AI providers are managed via CLI: - -```bash -# Add Ollama (no API key needed) -pnpm cli provider add "Local Ollama" ollama http://localhost:11434 - -# Add OpenAI-compatible provider -pnpm cli provider add "OpenAI" openai https://api.openai.com $OPENAI_API_KEY - -# Discover available models -pnpm cli provider probe -``` - -## The Journey to v1.0.1 - -This release represents 21 days of development from April 27 through May 17, 2026: - -- **Days 1–2** (Apr 27–28): Project initialization, dark industrial theme, JWT authentication, Project Manager, drone-to-IDE event routing, workspace persistence and crash recovery. -- **Days 3–4** (Apr 29–30): Socket protocol completeness, DroneManager, GadgetId migration from ObjectId to nanoid-based string IDs, "Welcome to The Grid" home view. -- **Days 5–6** (May 1–3): Workspace mode switching with validation, session locks, basic chat system milestone — the first end-to-end working demo. -- **Days 7–8** (May 5–6): Agent tool and toolbox architecture, tool registration patterns, AI environment foundation. -- **Day 9** (May 7): Streaming responses implementation — real-time token delivery from Ollama. -- **Day 10** (May 8): Chat session heartbeat, user settings, logging unification into `@gadget/api` as GadgetLog, redesigned sign-in form. -- **Day 9 continued** (May 9): Agentic Workflow Loop rebuild, chat session auto-naming, drone-to-IDE log transport. -- **Day 11** (May 10): Workspace/project philosophy refinement, agent toolbox refactor, credential provider fixes. -- **Day 12** (May 11): Subagent processing, authentication fixes, deployment documentation with systemd service files. -- **Day 13** (May 12): FILES panel with lazy-loading file tree, CodeMirror editor integration (replacing react-ace for React 19 compat), abort controller. -- **Day 14** (May 13): Editor height fixes, User Mode documentation. -- **Day 15** (May 14): SubProcess observability — tracking spawned processes in the DroneManager and DroneInspector. -- **Day 16** (May 15): Pull Project feature, tool call observability, context window configuration, Gab AI affiliate link. -- **Days 17–18** (May 16–17): AI toolbox extraction into `@gadget/ai-toolbox`, Project Manager upgrades, gadget-tasks headless client architecture, release hardening. +- **Auto-scroll fix** — Chat session view now properly auto-scrolls during streaming responses. +- **Home sidebar layout** — Fixed flex column layout for independent scrolling in the home sidebar. +- **Clean script** — Added `clean` script to `gadget-code/package.json` for easy build artifact removal. +- **UI** — Added `lucide-react` icon library for the search interface and other UI components. ## Project Direction -Gadget Code is built on a simple conviction: **your code, your models, your infrastructure**. The cloud AI ecosystem has moved toward vendor lock-in at every layer — proprietary models, closed platforms, data harvesting. Liberation is the alternative. Every component of Gadget Code is self-hosted. Every AI call goes through your providers. Every file stays on your disk. Every model runs on your hardware if you want it to. +Recall reinforces Gadget Code's core promise: **your code, your models, your infrastructure**. The semantic search feature runs on *your* Qdrant instance, using *your* embedding model, on *your* hardware. The token economics give you real observability into what your models are consuming — no vendor telemetry, no cloud analytics, just numbers on your screen from your API calls. -This is also the first release where the release document itself is authored by Gadget — the AI agent — as part of the automated release workflow. The agent that writes code for you also writes the release notes. We think that's fitting. +The removal of `syncIndexes()` from models and proper test mocking represents our ongoing commitment to clean, isolated unit tests that don't require standing up external services. Tests should be fast, deterministic, and runnable anywhere. ## Get Started @@ -113,7 +77,7 @@ Get the latest release at **https://g4dge7.com/** — the official home of Gadge Apache-2.0 licensed. Fully open source. No vendor lock-in. No data harvesting. No cloud dependency. Just you, your models, and your code. -That's Liberation. +That's Recall. ---