Compare commits

..

29 Commits

Author SHA1 Message Date
Rob Colbert
8ca6881661 release: 1.0.2 2026-05-19 16:37:01 -04:00
Rob Colbert
4fb0191460 release: generate RELEASE.md for v1.0.2 Recall 2026-05-19 16:36:25 -04:00
Rob Colbert
47dd61f6d3 docs: update CHANGE.md and README.md for upcoming patch release
- CHANGE.md: Added entries for May 18 (token economics, chat export, UI fixes)
  and May 19 (Qdrant semantic search, vector embeddings, model/test fixes)
- README.md: Added Qdrant to prerequisites and architecture diagram,
  documented token economics, semantic search, and chat export features
2026-05-19 16:35:01 -04:00
Rob Colbert
e0fd6ece54 fix: remove syncIndexes from models, mock DB in drone-session tests
- Removed auto-invoked syncIndexes() blocks from all 11 model files
  (api-client, api-client-log, csrf-token, drone-monitor,
  drone-registration, email-log, email-verification, ide-session,
  user, web-token, web-visit) — no longer needed
- Fixed drone-session.test.ts by mocking ChatSession.updateOne(),
  VectorStoreService.ingestTurn(), and MessageQueue to prevent
  real DB connections during tests
- Removed unused mongoose Types import from drone-session test
- All 129 tests now pass across all packages
2026-05-19 16:32:58 -04:00
Rob Colbert
a7a6a91a13 fix: pass requested vector dimension to embedding API (Ollama + OpenAI)
The qwen3-embedding:4b model defaults to 2560-d vectors. Both
Ollama (client.embed()) and OpenAI support a dimensions parameter
to request a specific output size. This change threads the configured
qdrant.vectorSize through the AI provider layer so the model returns
vectors matching the Qdrant collection dimensions.

- AiApi.embeddings() now accepts optional dimensions parameter
- Ollama provider: switched from client.embeddings() to client.embed()
- OpenAI provider: passes dimensions to embeddings.create()
- VectorStoreService.getEmbedding() passes env.qdrant.vectorSize
- Added unit tests for dimension mismatch detection, collection
  creation, and search guards
2026-05-19 16:21:04 -04:00
Rob Colbert
3c076fc01a Merge branch 'develop' of git.digitaltelepresence.com:rob/gadget into develop 2026-05-19 15:46:16 -04:00
Rob Colbert
66774d94f9 search input and backend error fixes 2026-05-19 15:43:24 -04:00
Rob Colbert
00e2fdfa4f a little logging and formatting cleanup 2026-05-19 15:42:48 -04:00
Rob Colbert
70ad578425 Merge branch 'develop' of git.digitaltelepresence.com:rob/gadget into develop 2026-05-19 14:30:04 -04:00
Rob Colbert
9aaacead42 quick auto-scroll fix 2026-05-19 14:29:58 -04:00
Rob Colbert
56f35a2fdf feat: Qdrant semantic search over chat history
Adds vector-based semantic search across all chat sessions using Qdrant.
When a ChatTurn finishes, its content is chunked, embedded, and upserted
to a Qdrant collection. A search API and UI components enable searching
at user, project, and session scope.

Phase 1 — Configuration & Dependencies
- Add port/apiKey to GadgetCodeConfig.qdrant type
- Uncomment and update qdrant section in YAML config example
- Add qdrant config passthrough in env.ts
- Add @qdrant/js-client-rest dependency

Phase 2 — AI Embedding API (@gadget/ai)
- Add IAiEmbeddingResponse interface and abstract embeddings() to AiApi
- Implement embeddings() in OllamaAiApi (client.embeddings)
- Implement embeddings() in OpenAiApi (client.embeddings.create)
- Export IAiEmbeddingResponse from package index

Phase 3 — Backend Vector Store Service
- Create VectorStoreService (ingestTurn, search, removeTurnPoints)
- Hook fire-and-forget ingest after turn.save() in drone-session
- Register VectorStoreService in service startup/shutdown

Phase 4 — Backend Search API
- Create POST /api/v1/search controller with userId enforcement
- Batch-hydrate results from MongoDB (user, project, session, turn)
- Register search route in v1 API router

Phase 5 — Frontend Search Components
- SearchInput: debounced input with lucide-react icons
- ChatSearchResults: modal with score badges, metadata, loading states
- DroneSelectionModal: drone picker for sessions without a drone
- Add searchApi and ISearchResult to API client
- Add search to Home (global), ProjectManager (project), ChatSessionView (session)
- Add id=turn-{turnId} to ChatTurn for scroll targeting
- Scroll-to-turn from search result selection and router state
- Show DroneSelectionModal when no drone available
- Add Select Drone button in ChatSessionView sidebar
2026-05-19 14:28:30 -04:00
Rob Colbert
d47ef96916 remove empty content continue; prettier then reformatted many things 2026-05-19 11:48:56 -04:00
Rob Colbert
1758bb2db7 add @langchain/textsplitters for text vector processing 2026-05-19 11:08:56 -04:00
Rob Colbert
aeaefc53dd cleanup 2026-05-19 09:54:36 -04:00
Rob Colbert
faf815d66d added Qdrant embedding model config 2026-05-19 09:34:35 -04:00
Rob Colbert
e2cefc2617 added qdrant config 2026-05-19 03:53:21 -04:00
Rob Colbert
f539f91018 Merge branch 'develop' of git.digitaltelepresence.com:rob/gadget into develop 2026-05-19 02:22:49 -04:00
Rob Colbert
cf31aff8db added a clean script 2026-05-19 02:22:39 -04:00
Rob Colbert
6696d79a0d repairs for OpenAI token stats/economics 2026-05-18 16:40:19 -04:00
Rob Colbert
ef43fa2a00 yet another attempt from yet another agent on trying to just capture some simple stats 2026-05-18 16:08:12 -04:00
Rob Colbert
241f3b5b69 feat(ai): add raw API response logging at OpenAI client call sites
- Log the SDK response object immediately after client.chat.completions.create()
  in both generate() and readStreamingChatCompletion()
- These debug-level logs capture the response at the call site before any
  iteration or processing
- All downstream info-level logs continue to dump raw chunk/response objects
2026-05-18 15:45:27 -04:00
Rob Colbert
73b0b3149d fix(ai): log raw OpenAI API response objects, not cooked usage
- Add rawUsage field to OpenAiChatIterationResult to carry the
  original OpenAI SDK usage object through the call chain
- All downstream chat() logs now dump rawUsage via
  JSON.parse(JSON.stringify()) instead of our cooked
  { promptTokens, completionTokens }
- This exposes completion_tokens_details.reasoning_tokens and
  any other fields OpenAI returns that we weren't capturing
2026-05-18 15:29:59 -04:00
Rob Colbert
51b3013659 feat(ai): add DtpLog logging of all OpenAI API responses
- Fix broken chunkCount reference in generate() usage chunk log
- Add generate() complete log with usage, response/thinking lengths
- Add readStreamingChatCompletion() usage chunk log (full chunk dump)
- Add readStreamingChatCompletion() iteration complete log
- Add readNonStreamingChatCompletion() full response dump
- Add chat() streaming iteration usage log
- Add chat() non-streaming fallback usage log
- Add chat() final response stats log

All logs use info level and 'OpenAI {method}: {event}' naming for
easy grep/filtering. Streaming chunks use JSON.parse(JSON.stringify())
to serialize the full OpenAI SDK response objects.
2026-05-18 15:20:53 -04:00
Rob Colbert
2483161663 context window fuel gauge auto update fix 2026-05-18 15:00:40 -04:00
Rob Colbert
00969cf9bc feat: token economics — real API token extraction, stats propagation, and context window fuel gauge
Phase 1: OpenAI API Token Extraction
- Add stream_options: { include_usage: true } to all streaming API calls
- Capture chunk.usage from final streaming chunks and response.usage from non-streaming
- Extend OpenAiChatIterationResult with optional usage field
- Update buildStats() to accept and return real token counts from usage data
- Wire iteration.usage through chat() to both buildStats() call sites

Phase 2: Agent Loop Stats Propagation
- Rename inputTokens/outputTokens to masterInputTokens/masterOutputTokens, add masterThinkingTokens
- Accumulate response.stats.tokenCounts after each master AI call
- Delete all Math.ceil(length/4) crude approximations (master and subagent loops)
- Track startTime/durationMs and emit IWorkOrderCompleteStats with workOrderComplete
- Subagent loop uses response.stats?.tokenCounts instead of Math.ceil

Phase 3: Database Model Changes
- Add contextWindowUsage field to IChatSession, ChatSessionSchema, and frontend ChatSession
- Initialize contextWindowUsage: 0 on session creation

Phase 4: Persist Stats on Turn Completion
- drone-session: accept IWorkOrderCompleteStats, persist turn stats, walk subagent records for aggregate
- drone-session: $inc session stats and contextWindowUsage, add formatDurationLabel() helper
- code-session: accept and forward stats, update in-memory session stats
- message-queue: Redis replay handles 4th stats arg
- Update WorkOrderCompleteMessage type in @gadget/api to accept stats parameter

Phase 5: UI — Context Window Fuel Gauge
- Add contextWindowUsage prop to SessionPanel
- Add fuel gauge bar with E→F labels, green/yellow/red zones, token count display
- Visible for ALL provider types (not gated by apiType)

Phase 6: Frontend Streaming State
- Add IWorkOrderCompleteStats interface to frontend api.ts
- handleWorkOrderComplete accepts stats, updates turn stats and session contextWindowUsage
- Pass contextWindowUsage prop to SessionPanel
2026-05-18 14:36:36 -04:00
Rob Colbert
9c1e65785d feat: add chat_export tool for session export (markdown/json)
New chat tool: chat_export

- Export current ChatSession and all finished ChatTurn records to disk
- Two formats: markdown (human-readable) and json (machine-readable)
- JSON exports always include a companion -readme.md with author credits,
  Gadget Code version/link, AI provider/model/parameters
- Output to .gadget/exports/ with sanitized session name + timestamp
- Security-first: excludes emails, API keys, baseUrls, system prompts,
  gitUrls, and other PII/infrastructure details from all exports
- Only finished/aborted/error turns included (processing excluded)
- Follows SubagentTool setter-injection pattern for session data
- Available in all chat session modes (plan/build/test/ship/dev)

Files:
- packages/ai-toolbox/src/chat/export.ts (NEW - 729 lines)
- packages/ai-toolbox/src/chat/index.ts (barrel update)
- gadget-drone/src/tools/index.ts (re-export update)
- gadget-drone/src/services/agent.ts (import, register, getExportData)
2026-05-18 11:44:53 -04:00
Rob Colbert
0437ca66d8 Merge branch 'develop' of git.digitaltelepresence.com:rob/gadget into develop 2026-05-18 09:11:16 -04:00
Rob Colbert
ca0cbf8e71 added lucide-react 2026-05-18 09:11:08 -04:00
Rob Colbert
802184a487 fix: home sidebar flex column layout with independent scrolling sections
- Convert sidebar <aside> to flex column container (flex flex-col overflow-hidden min-h-0)
- Remove sidebar-level scroll (overflow-y-auto → overflow-hidden)
- Add min-h-0 to parent flex containers in both layout paths for proper height containment
- Reorder sidebar sections: Clock → Projects → Drones → Recent Chats
- Add shrink-0 to Clock component to keep it at intrinsic content size
- Convert Projects, Drones, and Recent Chats sections to flex-1 min-h-0 flex-col with scrollable inner content
- Pin section headers with shrink-0; list content scrolls independently via overflow-y-auto
- Pin Recent Chats hint/footer with shrink-0 below its scrollable list
- No logic changes — purely CSS/flexbox layout correction
2026-05-18 09:04:25 -04:00
53 changed files with 3159 additions and 404 deletions

View File

@ -4,6 +4,37 @@ This document tracks all significant changes to Gadget Code, organized by date.
---
## 2026-05-19 (Monday)
**Qdrant Semantic Search and Vector Embeddings**
Implemented Qdrant-powered semantic search over chat history with full embedding pipeline, including Qdrant and LangChain configuration, vector dimension passthrough for both Ollama and OpenAI, and a new search API endpoint with frontend search input component. Also fixed Mongoose syncIndexes errors across all models and improved drone-session test isolation.
### Changes:
- **Semantic Search**: Implemented Qdrant vector search over chat history with embedding pipeline and search API endpoint.
- **Embeddings**: Added Qdrant and embedding model configuration; added `@langchain/textsplitters` for text vector processing; fixed vector dimension passthrough for Ollama and OpenAI.
- **Frontend**: Added SearchInput and ChatSearchResults components; integrated search into ChatSessionView.
- **Models**: Removed `syncIndexes` from all Mongoose models (caused duplicate index errors); mocked DB in drone-session tests.
- **Fixes**: Fixed auto-scroll, search input errors, and backend error handling; logging and formatting cleanup.
- **Build**: Added clean script.
---
## 2026-05-18 (Sunday)
**Token Economics, Chat Export, and UI Layout Fixes**
Implemented token economics with real API token extraction from OpenAI responses, stats propagation through the session pipeline, and a context window fuel gauge. Added chat_export agent tool for markdown/JSON session export. Fixed home sidebar layout for independent scrolling sections.
### Changes:
- **Token Economics**: Implemented real API token extraction from OpenAI responses, stats propagation, and context window fuel gauge with auto-update.
- **Logging**: Added DtpLog logging of all OpenAI API responses with raw response object capture.
- **Chat Export**: Added `chat_export` agent tool for session export in markdown and JSON formats.
- **UI**: Fixed home sidebar flex column layout with independent scrolling sections; added `lucide-react` icons.
- **Config**: Added Qdrant and embedding model config types.
---
## 2026-05-17 (Sunday)
**Release Hardening, Subagent Improvements, and Workspace Intelligence**

View File

@ -41,6 +41,8 @@ pnpm dev
│ │ │
│ Socket.IO │ MongoDB │ Files
│ JWT Auth │ Redis │ Git
│ │ Qdrant │
│ │ │
```
### How It Works
@ -117,6 +119,7 @@ pnpm cli user password user@example.com newpassword
- pnpm 10+
- MongoDB on `localhost:27017`
- Redis on `localhost:6379`
- Qdrant on `localhost:6333` (for semantic search)
- SSL certificates in `ssl/` directory (for dev servers)
### Build
@ -196,6 +199,23 @@ npx tsx scripts/seed-test-drones.ts
- Model/mode selection per session
- Session statistics (tool calls, file ops, subagents)
### Token Economics
- Real API token extraction from OpenAI and Ollama responses
- Token usage stats propagated through session pipeline
- Context window fuel gauge with auto-update
### Semantic Search
- Qdrant-powered vector search over chat history
- Embedding pipeline with Ollama and OpenAI support
- Configurable embedding model and vector dimensions
### Chat Export
- Export chat sessions as markdown or JSON
- Agent tool for automated session export
### Workspace Management
- Each drone manages a workspace directory
@ -253,5 +273,5 @@ Apache 2.0 — See [LICENSE](./LICENSE) for details.
---
**Status:** v1.0.1 — First release. Self-hosted agentic engineering platform with local/self-hosted AI model support, complete Chat Session UI, subagent processing, and workspace management.
**Last Updated:** May 17, 2026
**Status:** v1.0.1 — Self-hosted agentic engineering platform with local/self-hosted AI model support, complete Chat Session UI, subagent processing, workspace management, token economics, and Qdrant semantic search.
**Last Updated:** May 19, 2026

View File

@ -1,111 +1,75 @@
GADGET CODE v1.0.1
NAME: Liberation
HASH: ff46c35d38e29be64d9eca0ccc18e8a7f5c3d8ab
TAG: v1.0.1
GADGET CODE v1.0.2
NAME: Recall
HASH: 47dd61f6d36a3287b67025e15caa03eb6f830195
TAG: v1.0.2
SUMMARY:
Today marks the Liberation of Gadget Code — the first public release of a self-hosted agentic engineering platform that puts you in control of your code, your models, and your infrastructure.
Recall is the first patch release of Gadget Code — and the name is deliberate. This release introduces **semantic search over your chat history** using Qdrant vector embeddings, so your past engineering conversations are always just a query away. It also delivers real **token economics**: actual API token counts extracted from provider responses, propagated through the session stack, and surfaced in a live context window fuel gauge. Combined with chat export, streaming buffer persistence, and a round of model/test cleanup, Recall makes Gadget Code more observable, more searchable, and more reliable.
Over 200 commits spanning 21 days of intensive development, Gadget Code has grown from an initial commit into a complete platform for autonomous software engineering. And every line of it is Apache-2.0 licensed, fully open source, and designed to run on *your* terms.
## Semantic Search with Qdrant
## What Is Gadget Code?
The headline feature. Gadget Code now ships with an integrated **Qdrant vector search** pipeline that indexes every completed chat turn and makes it full-text+semantic searchable from the IDE. The architecture:
Gadget Code is an **Agentic Engineering Platform (AEP)** — a browser-based IDE that drives autonomous AI agents to perform real software engineering work on your behalf. Unlike cloud-locked alternatives, Gadget Code runs entirely in your environment: your servers, your data, your rules.
- **VectorStoreService** — A singleton service in `gadget-code` that manages Qdrant collection lifecycle, dimension mismatch detection, and embedding ingestion. It validates collection dimensions against the configured vector size and the model's actual output, surfacing clear error messages when they disagree.
- **Embedding pipeline** — Uses your configured AI provider to generate embeddings. Supports both Ollama and OpenAI-compatible endpoints. Configuration lives in `gadget-code.yaml` under the new `qdrant` section.
- **Search API endpoint**`POST /api/v1/search` accepts a query string, generates its embedding, and returns ranked results from Qdrant.
- **Frontend components**`SearchInput` and `ChatSearchResults` components in the React IDE provide a clean search-as-you-type experience with ranked results linked back to their source chat sessions.
The architecture is straightforward and powerful:
**New dependency**: Qdrant must be running on `localhost:6333` to enable semantic search. If `qdrant.providerId` is not configured, the service gracefully skips initialization — nothing breaks, you just don't get search.
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Browser IDE │────▶│ gadget-code │────▶│ gadget-drone │
│ (React 19) │◀────│ (Express 5) │◀────│ (Worker) │
└──────────────┘ └──────────────┘ └──────────────┘
│ │ │
│ Socket.IO │ MongoDB │ Files
│ JWT Auth │ Redis │ Git
**New package dependency**: `@langchain/textsplitters` for chunking turn content before embedding.
## Token Economics
Token counts are no longer estimates. This release extracts real usage data from OpenAI API responses and propagates it through the entire session stack:
- **Real token extraction** — OpenAI responses now return `usage.prompt_tokens`, `usage.completion_tokens`, and `usage.completion_tokens_details.reasoning_tokens` from the API. These are captured and stored on each `ChatTurn` and aggregated on the `ChatSession`.
- **Context window fuel gauge** — The session UI now displays a live context window usage indicator. As token counts accumulate across turns, the gauge updates automatically so you know when you're approaching your model's context limit.
- **Stats propagation**`IWorkOrderCompleteStats` now carries `masterInputTokens`, `masterOutputTokens`, and `masterThinkingTokens`. The `onWorkOrderComplete` handler aggregates master + subagent tokens for the session-level totals.
- **DtpLog API response logging** — Added `logApiResponse` to the `DtpLog` transport for structured logging of AI provider responses with full token metadata.
## Chat Export
New `chat_export` agent tool lets the drone export chat sessions as markdown or JSON. Useful for documentation generation, session archival, and sharing engineering conversations outside the IDE.
## Streaming Buffer Persistence
Drone sessions now use an in-memory streaming buffer that aggregates thinking and response tokens by mode, flushing to the database at mode transitions (e.g., thinking → tool call → responding). This reduces database writes during streaming and ensures turn blocks are persisted in the correct order with accurate timestamps.
## Model and Test Cleanup
- **Removed `syncIndexes()` from all Mongoose models** — The auto-invoked `(async () => { await Model.syncIndexes(); })()` blocks at module level caused Mongoose connection timeout errors during tests when no database was available. These calls are no longer needed and have been removed from all 11 model files.
- **Mocked database in drone-session tests**`ChatSession.updateOne()`, `VectorStoreService.ingestTurn()`, and `MessageQueue` are now properly mocked so tests run without any database dependency.
## Configuration Changes
New `qdrant` section in `gadget-code.yaml`:
```yaml
qdrant:
host: localhost
port: 6333
providerId: <your-ai-provider-id>
embeddingModel: <model-name>
vectorSize: 1024
collection: gadget-code
```
A user creates a project in the browser IDE, selects a drone instance, enters a prompt, and the drone executes the Agentic Workflow Loop — thinking, responding, calling tools, spawning subagents — all streamed back in real time.
New types added to `@gadget/config`: `IQdrantConfig` and `IEmbeddingModelConfig`.
## Your Models, Your Way
## Other Changes
This is what Liberation means. Gadget Code ships with first-class support for **Ollama** and any **OpenAI-compatible API**. Run local models on your own GPU. Self-host an inference endpoint behind your firewall. Use a commercial provider when it suits you. Mix and match per session. The `@gadget/ai` package provides a unified abstraction — no consumer code ever imports an SDK directly. Swap providers without changing a line of application logic.
The model configuration pipeline supports `numCtx` (context window), `numPredict`, and `maxCompletionTokens` settings, giving you fine-grained control over how your models behave.
## Complete Agentic Engineering Platform
Gadget Code isn't a chatbot wrapper. It's a full-stack engineering environment:
- **Project Manager** — Create projects, select available drone instances, manage chat session history.
- **Chat Session View** — Real-time streaming responses with collapsible thinking content, tool call summaries, session statistics, and per-session model/mode selection.
- **Agentic Workflow Loop** — The drone executes a multi-turn reasoning loop: think, respond, call tools, observe results, and iterate. Work orders are abortable, and the loop supports checkpointing for crash recovery.
- **Subagent Processing** — Agents can spawn specialized subagents for decomposition tasks. Subagent context is managed independently and streamed back to the IDE in real time.
- **FILE Panel** — Lazy-loading file tree with full workspace navigation and CodeMirror editor integration (React 19 compatible).
- **Workspace Management** — Each drone manages a workspace directory with crash recovery via the `.gadget/` directory and intelligent startup that detects the workspace by walking up the file hierarchy.
- **Scheduled Tasks (gadget-tasks)** — A headless IDE client that automates the browser flow on a cron schedule. It drives the existing gadget-code platform via the same REST API and Socket.IO protocol the browser uses — no duplicated logic.
- **Pull Project** — Clone remote repositories directly into your workspace.
- **Abort Controller** — Cancel in-progress work orders with proper state cleanup.
## Configuration and Dependencies
Gadget Code requires:
- **Node.js 22+** and **pnpm 10+**
- **MongoDB** on `localhost:27017`
- **Redis** on `localhost:6379`
- **SSL certificates** in the `ssl/` directory for dev servers
- An **Ollama** instance or **OpenAI-compatible API** endpoint for AI features
The monorepo is organized as follows:
| Package | Role |
| -------------------- | --------------------------------------------------------------- |
| `gadget-code` | Web service — agentic IDE, browser UI, API server |
| `gadget-drone` | Worker process — runs the agentic workflow loop |
| `gadget-tasks` | Scheduled task worker — headless IDE client for cron-driven tasks |
| `@gadget/ai` | Shared AI API abstraction — Ollama and OpenAI |
| `@gadget/ai-toolbox` | Shared AI tool implementations — search, file, plan, subagent |
| `@gadget/api` | Shared TypeScript interfaces — common types across all packages |
| `@gadget/config` | Shared YAML config loader — per-package configuration |
AI providers are managed via CLI:
```bash
# Add Ollama (no API key needed)
pnpm cli provider add "Local Ollama" ollama http://localhost:11434
# Add OpenAI-compatible provider
pnpm cli provider add "OpenAI" openai https://api.openai.com $OPENAI_API_KEY
# Discover available models
pnpm cli provider probe <provider-id>
```
## The Journey to v1.0.1
This release represents 21 days of development from April 27 through May 17, 2026:
- **Days 12** (Apr 2728): Project initialization, dark industrial theme, JWT authentication, Project Manager, drone-to-IDE event routing, workspace persistence and crash recovery.
- **Days 34** (Apr 2930): Socket protocol completeness, DroneManager, GadgetId migration from ObjectId to nanoid-based string IDs, "Welcome to The Grid" home view.
- **Days 56** (May 13): Workspace mode switching with validation, session locks, basic chat system milestone — the first end-to-end working demo.
- **Days 78** (May 56): Agent tool and toolbox architecture, tool registration patterns, AI environment foundation.
- **Day 9** (May 7): Streaming responses implementation — real-time token delivery from Ollama.
- **Day 10** (May 8): Chat session heartbeat, user settings, logging unification into `@gadget/api` as GadgetLog, redesigned sign-in form.
- **Day 9 continued** (May 9): Agentic Workflow Loop rebuild, chat session auto-naming, drone-to-IDE log transport.
- **Day 11** (May 10): Workspace/project philosophy refinement, agent toolbox refactor, credential provider fixes.
- **Day 12** (May 11): Subagent processing, authentication fixes, deployment documentation with systemd service files.
- **Day 13** (May 12): FILES panel with lazy-loading file tree, CodeMirror editor integration (replacing react-ace for React 19 compat), abort controller.
- **Day 14** (May 13): Editor height fixes, User Mode documentation.
- **Day 15** (May 14): SubProcess observability — tracking spawned processes in the DroneManager and DroneInspector.
- **Day 16** (May 15): Pull Project feature, tool call observability, context window configuration, Gab AI affiliate link.
- **Days 1718** (May 1617): AI toolbox extraction into `@gadget/ai-toolbox`, Project Manager upgrades, gadget-tasks headless client architecture, release hardening.
- **Auto-scroll fix** — Chat session view now properly auto-scrolls during streaming responses.
- **Home sidebar layout** — Fixed flex column layout for independent scrolling in the home sidebar.
- **Clean script** — Added `clean` script to `gadget-code/package.json` for easy build artifact removal.
- **UI** — Added `lucide-react` icon library for the search interface and other UI components.
## Project Direction
Gadget Code is built on a simple conviction: **your code, your models, your infrastructure**. The cloud AI ecosystem has moved toward vendor lock-in at every layer — proprietary models, closed platforms, data harvesting. Liberation is the alternative. Every component of Gadget Code is self-hosted. Every AI call goes through your providers. Every file stays on your disk. Every model runs on your hardware if you want it to.
Recall reinforces Gadget Code's core promise: **your code, your models, your infrastructure**. The semantic search feature runs on *your* Qdrant instance, using *your* embedding model, on *your* hardware. The token economics give you real observability into what your models are consuming — no vendor telemetry, no cloud analytics, just numbers on your screen from your API calls.
This is also the first release where the release document itself is authored by Gadget — the AI agent — as part of the automated release workflow. The agent that writes code for you also writes the release notes. We think that's fitting.
The removal of `syncIndexes()` from models and proper test mocking represents our ongoing commitment to clean, isolated unit tests that don't require standing up external services. Tests should be fast, deterministic, and runnable anywhere.
## Get Started
@ -113,7 +77,7 @@ Get the latest release at **https://g4dge7.com/** — the official home of Gadge
Apache-2.0 licensed. Fully open source. No vendor lock-in. No data harvesting. No cloud dependency. Just you, your models, and your code.
That's Liberation.
That's Recall.
---

View File

@ -132,12 +132,14 @@ minio:
videos: "gadget-videos"
audios: "gadget-audios"
# ── Qdrant Vector Database (COMING SOON) ────────────────────────────────────
# ── Qdrant Vector Database ──────────────────────────────────────────────────────
# Used for semantic search over chat history and conversation context.
# Enable when the Qdrant integration is released.
#
# qdrant:
# host: "localhost"
# port: 6333
# apiKey: "${QDRANT_API_KEY}"
# collection: "gadget-chat-embeddings"
qdrant:
host: "localhost"
port: 6333
apiKey: "${QDRANT_API_KEY}"
collection: "gadget-chat-embeddings"
providerId: "${QDRANT_PROVIDER_ID}" # AiProvider._id from the database
embeddingModel: "nomic-embed-text" # model name on the provider
vectorSize: 768 # must match the embedding model output

View File

@ -0,0 +1,129 @@
// src/components/ChatSearchResults.tsx
// Copyright (C) 2026 Robert Colbert <rob.colbert@openplatform.us>
// All Rights Reserved
import { useState, useEffect, useCallback } from "react";
import { Search, X, Loader2 } from "lucide-react";
import { searchApi, ISearchResult } from "../lib/api";
interface ChatSearchResultsProps {
query: string;
filters?: { projectId?: string; sessionId?: string; turnId?: string };
onSelect: (result: ISearchResult) => void;
onClose: () => void;
}
export default function ChatSearchResults({
query,
filters,
onSelect,
onClose,
}: ChatSearchResultsProps) {
const [editQuery, setEditQuery] = useState(query);
const [results, setResults] = useState<ISearchResult[]>([]);
const [loading, setLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const performSearch = useCallback(async () => {
if (!editQuery.trim()) return;
setLoading(true);
setError(null);
try {
const data = await searchApi.search(editQuery, filters, 20);
setResults(data);
} catch (err) {
setError((err as Error).message || "Search failed");
} finally {
setLoading(false);
}
}, [editQuery, filters]);
useEffect(() => { performSearch(); }, [performSearch]);
const handleSearchKeyDown = (e: React.KeyboardEvent<HTMLInputElement>) => {
if (e.key === "Enter") {
e.preventDefault();
performSearch();
}
};
const formatScore = (score: number): string => `${Math.round(score * 100)}%`;
const formatTimestamp = (iso: string): string => {
if (!iso) return "";
try { return new Date(iso).toLocaleDateString(undefined, { month: "short", day: "numeric", year: "numeric", hour: "2-digit", minute: "2-digit" }); }
catch { return iso; }
};
return (
<div className="fixed inset-0 bg-black/80 flex items-center justify-center z-50"
onClick={(e) => { if (e.target === e.currentTarget) onClose(); }}
onKeyDown={(e) => { if (e.key === "Escape") onClose(); }}
>
<div className="bg-bg-secondary border border-border-default rounded-lg w-full max-w-3xl max-h-[80vh] flex flex-col mx-4">
{/* Header with editable search input */}
<div className="flex items-center gap-3 px-6 py-4 border-b border-border-default">
<Search className="text-text-tertiary shrink-0" size={16} />
<input
type="text"
value={editQuery}
onChange={(e) => setEditQuery(e.target.value)}
onKeyDown={handleSearchKeyDown}
className="flex-1 bg-transparent text-text-primary text-sm focus:outline-none placeholder:text-text-tertiary"
placeholder="Edit search query…"
autoFocus
/>
<button
onClick={performSearch}
className="text-text-tertiary hover:text-brand transition-colors shrink-0"
aria-label="Submit search"
>
<Search size={16} />
</button>
<button onClick={onClose} className="text-text-tertiary hover:text-text-primary transition-colors p-1 shrink-0" aria-label="Close">
<X size={18} />
</button>
</div>
{/* Result count bar */}
{!loading && (
<div className="px-6 py-2 text-sm text-text-tertiary border-b border-border-default">
{results.length} result{results.length !== 1 ? "s" : ""} for "{query === editQuery ? query : editQuery}"
</div>
)}
<div className="overflow-y-auto flex-1 p-4">
{loading && (
<div className="flex items-center justify-center py-12">
<Loader2 className="text-brand text-2xl animate-spin" />
<span className="ml-3 text-text-secondary">Searching</span>
</div>
)}
{error && <div className="text-center py-12"><p className="text-red-400">{error}</p></div>}
{!loading && !error && results.length === 0 && <div className="text-center py-12"><p className="text-text-tertiary">No results found</p></div>}
{!loading && !error && results.length > 0 && (
<div className="space-y-2">
{results.map((result) => (
<button key={result.id} onClick={() => onSelect(result)}
className="w-full text-left p-4 rounded-lg bg-bg-primary hover:bg-bg-tertiary border border-border-default hover:border-brand transition-colors group"
>
<div className="flex items-start justify-between gap-3">
<span className="shrink-0 inline-flex items-center px-2 py-0.5 rounded text-xs font-mono bg-brand/20 text-brand">{formatScore(result.score)}</span>
<div className="flex-1 min-w-0"><p className="text-sm text-text-primary line-clamp-3 whitespace-pre-wrap">{result.content}</p></div>
</div>
<div className="flex items-center gap-2 mt-2 text-xs text-text-tertiary">
{result.project?.name && <span>{result.project.name}</span>}
{result.session?.name && <><span></span><span>{result.session.name}</span></>}
<span></span>
<span className="font-mono">{result.turnId.slice(0, 8)}</span>
{result.createdAt && <><span className="mx-1">·</span><span>{formatTimestamp(result.createdAt)}</span></>}
</div>
</button>
))}
</div>
)}
</div>
</div>
</div>
);
}

View File

@ -34,7 +34,7 @@ const ChatTurn = memo(function ChatTurn({ turn }: ChatTurnProps) {
const startedAt = new Date(turn.createdAt);
return (
<div className="border-b border-border-subtle pb-4 last:border-b-0">
<div id={`turn-${turn._id}`} className="border-b border-border-subtle pb-4 last:border-b-0">
{/* Turn Header */}
<div className="flex items-center gap-4 text-xs text-text-muted mb-3 px-4">
<div>{startedAt.toLocaleTimeString()}</div>

View File

@ -28,7 +28,7 @@ export default function Clock() {
}, []);
return (
<div className="p-3">
<div className="p-3 shrink-0">
<div className="text-xs font-semibold text-text-muted uppercase tracking-wider mb-2">
Clock
</div>

View File

@ -0,0 +1,84 @@
// src/components/DroneSelectionModal.tsx
// Copyright (C) 2026 Robert Colbert <rob.colbert@openplatform.us>
// All Rights Reserved
import { useState, useEffect } from "react";
import { X, Loader2 } from "lucide-react";
import { droneApi, type DroneRegistration } from "../lib/api";
interface DroneSelectionModalProps {
onSelect: (drone: DroneRegistration) => void;
onClose: () => void;
}
export default function DroneSelectionModal({ onSelect, onClose }: DroneSelectionModalProps) {
const [drones, setDrones] = useState<DroneRegistration[]>([]);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
const loadDrones = async () => {
try { setDrones(await droneApi.getAll()); }
catch (err) { setError((err as Error).message || "Failed to load drones"); }
finally { setLoading(false); }
};
loadDrones();
}, []);
const statusColor = (status: string) => {
switch (status) {
case "available": return "bg-green-500";
case "busy": return "bg-yellow-500";
default: return "bg-gray-500";
}
};
const availableDrones = drones.filter(d => d.status === "available" || d.status === "busy");
return (
<div className="fixed inset-0 bg-black/80 flex items-center justify-center z-50"
onClick={(e) => { if (e.target === e.currentTarget) onClose(); }}
onKeyDown={(e) => { if (e.key === "Escape") onClose(); }}
>
<div className="bg-bg-secondary border border-border-default rounded-lg w-full max-w-xl mx-4">
<div className="flex items-center justify-between px-6 py-4 border-b border-border-default">
<h2 className="text-lg font-semibold text-text-primary">Select a Drone</h2>
<button onClick={onClose} className="text-text-tertiary hover:text-text-primary transition-colors p-1" aria-label="Close"><X size={18} /></button>
</div>
<div className="p-4 max-h-[60vh] overflow-y-auto">
{loading && (
<div className="flex items-center justify-center py-8">
<Loader2 className="text-brand text-xl animate-spin" />
<span className="ml-3 text-text-secondary">Loading drones</span>
</div>
)}
{error && <div className="text-center py-8"><p className="text-red-400">{error}</p></div>}
{!loading && !error && availableDrones.length === 0 && (
<div className="text-center py-8">
<p className="text-text-tertiary">No drones are currently available.</p>
<p className="text-text-muted text-sm mt-2">Start a drone agent to begin working.</p>
</div>
)}
{!loading && !error && availableDrones.length > 0 && (
<div className="space-y-2">
{availableDrones.map((drone) => (
<button key={drone._id} onClick={() => onSelect(drone)}
className="w-full text-left p-3 rounded-lg bg-bg-primary hover:bg-bg-tertiary border border-border-default hover:border-brand transition-colors group"
>
<div className="flex items-center gap-3">
<span className={`shrink-0 w-2.5 h-2.5 rounded-full ${statusColor(drone.status)}`} />
<div className="flex-1 min-w-0">
<div className="font-mono text-sm text-text-primary group-hover:text-brand transition-colors">{drone.hostname}</div>
<div className="text-xs text-text-tertiary truncate">{drone.workspaceDir}</div>
</div>
<span className="text-xs text-text-muted capitalize">{drone.status}</span>
</div>
</button>
))}
</div>
)}
</div>
</div>
</div>
);
}

View File

@ -0,0 +1,68 @@
// src/components/SearchInput.tsx
// Copyright (C) 2026 Robert Colbert <rob.colbert@openplatform.us>
// All Rights Reserved
import { useState } from "react";
import { Search, X } from "lucide-react";
interface SearchInputProps {
placeholder?: string;
onSearch: (query: string) => void;
onClear?: () => void;
className?: string;
}
export default function SearchInput({
placeholder = "Search…",
onSearch,
onClear,
className = "",
}: SearchInputProps) {
const [value, setValue] = useState("");
const handleChange = (e: React.ChangeEvent<HTMLInputElement>) => {
const v = e.target.value;
setValue(v);
if (!v.trim()) {
onClear?.();
}
};
const handleClear = () => {
setValue("");
onClear?.();
};
const handleSubmit = () => {
if (value.trim()) onSearch(value.trim());
};
const handleKeyDown = (e: React.KeyboardEvent<HTMLInputElement>) => {
if (e.key === "Enter") handleSubmit();
if (e.key === "Escape") handleClear();
};
return (
<div className={`relative flex items-center ${className}`}>
<Search className="absolute left-3 text-text-tertiary pointer-events-none" size={14} />
<input
type="text"
value={value}
onChange={handleChange}
onKeyDown={handleKeyDown}
placeholder={placeholder}
className="w-full bg-bg-secondary border border-border-default rounded pl-8 pr-16 py-2 text-sm text-text-primary placeholder:text-text-tertiary focus:outline-none focus:border-brand transition-colors"
/>
{value && (
<>
<button onClick={handleClear} className="absolute right-8 text-text-tertiary hover:text-text-primary transition-colors" aria-label="Clear search">
<X size={14} />
</button>
<button onClick={handleSubmit} className="absolute right-2 text-text-tertiary hover:text-brand transition-colors" aria-label="Submit search">
<Search size={14} />
</button>
</>
)}
</div>
);
}

View File

@ -41,6 +41,7 @@ interface SessionPanelProps {
truncateModelName: (name: string, maxLength?: number) => string;
getSelectedModelCapabilities: () => { hasThinking: boolean } | null;
getSelectedModel: () => AiModel | null;
contextWindowUsage: number;
}
export default function SessionPanel({
@ -82,7 +83,9 @@ export default function SessionPanel({
truncateModelName,
getSelectedModelCapabilities,
getSelectedModel,
contextWindowUsage,
}: SessionPanelProps) {
const usagePercent = sessionNumCtx > 0 ? (contextWindowUsage / sessionNumCtx) * 100 : 0;
return (
<div className="border-b border-border-subtle shrink-0">
<div className="flex items-center justify-between px-4 py-2 bg-bg-tertiary">
@ -292,6 +295,33 @@ export default function SessionPanel({
</div>
</div>
)}
<div>
<div className="text-xs text-text-muted">Context Window Usage</div>
<div className="mt-1">
<div className="flex items-center gap-1.5">
<span className="text-[10px] font-mono text-text-muted w-3">E</span>
<div className="flex-1 h-3 bg-bg-tertiary rounded-full overflow-hidden border border-border-subtle">
<div
className={`h-full rounded-full transition-all duration-300 ${
usagePercent >= 80 ? 'bg-red-500' :
usagePercent >= 50 ? 'bg-yellow-500' :
'bg-green-500'
}`}
style={{ width: `${Math.min(usagePercent, 100)}%` }}
/>
</div>
<span className="text-[10px] font-mono text-text-muted w-3">F</span>
</div>
<div className="flex justify-between mt-0.5">
<span className="text-[10px] text-text-muted tabular-nums">
{contextWindowUsage.toLocaleString()} tokens
</span>
<span className="text-[10px] text-text-muted tabular-nums">
{sessionNumCtx.toLocaleString()} max
</span>
</div>
</div>
</div>
</div>
</div>
);

View File

@ -388,6 +388,7 @@ export interface ChatSession {
selectedModel: string;
reasoningEffort?: string;
numCtx?: number;
contextWindowUsage: number;
stats: ChatSessionStats;
pins: Array<{ _id?: string; content: string }>;
}
@ -401,6 +402,18 @@ export interface ChatTurnStats {
durationLabel: string;
}
export interface IWorkOrderCompleteStats {
masterInputTokens: number;
masterOutputTokens: number;
masterThinkingTokens: number;
toolCallCount: number;
durationMs: number;
/** Aggregate (master + subagent) totals for session-level accounting */
totalInputTokens?: number;
totalOutputTokens?: number;
totalToolCallCount?: number;
}
export interface ChatTurnPrompts {
user: string;
system?: string;
@ -487,3 +500,32 @@ export const chatSessionApi = {
getTurns: (id: string) =>
api.get<ChatTurn[]>(`/api/v1/chat-sessions/${id}/turns`),
};
export interface ISearchResult {
id: string;
content: string;
score: number;
userId: string;
projectId: string;
sessionId: string;
turnId: string;
role: string;
createdAt: string;
user?: { _id: string; displayName: string; username: string };
project?: { _id: string; name: string; slug: string };
session?: { _id: string; name: string };
turn?: { _id: string; status: string; mode: string; llm: string };
}
export const searchApi = {
search: (
query: string,
filters?: { projectId?: string; sessionId?: string; turnId?: string },
topK?: number,
) =>
api.post<ISearchResult[]>("/api/v1/search", {
query,
...filters,
topK,
}),
};

View File

@ -1,6 +1,6 @@
import { createContext } from "react";
import { io, Socket } from "socket.io-client";
import type { ChatSession } from "./api";
import type { ChatSession, IWorkOrderCompleteStats } from "./api";
/**
* Web Worker for heartbeat timing avoids browser tab throttling.
@ -30,6 +30,7 @@ export interface ServerToClientEvents {
turnId: string,
success: boolean,
message?: string,
stats?: IWorkOrderCompleteStats,
) => void;
log: (
timestamp: string,
@ -90,6 +91,7 @@ export interface SocketEvents {
turnId: string,
success: boolean,
message?: string,
stats?: IWorkOrderCompleteStats,
) => void;
"agent:thinking": (data: { agentId: string; thinking: string }) => void;
"agent:response": (data: { agentId: string; chunk: string }) => void;
@ -228,8 +230,8 @@ class SocketClient {
this.socket.on(
"workOrderComplete",
(turnId: string, success: boolean, message?: string) => {
this.emit("workOrderComplete", turnId, success, message);
(turnId: string, success: boolean, message?: string, stats?: IWorkOrderCompleteStats) => {
this.emit("workOrderComplete", turnId, success, message, stats);
},
);

View File

@ -2,7 +2,7 @@ import { useState, useEffect, useRef, useContext, useCallback } from 'react';
import { useParams, useNavigate, useLocation } from 'react-router-dom';
import { BrushCleaning, Maximize2 } from 'lucide-react';
import { socketClient } from '../lib/socket';
import { chatSessionApi, projectApi, providerApi, type ChatSession, type ChatTurn, type ChatTurnBlock, ChatTurnStats, ChatSessionMode, type AiProvider, type Project } from '../lib/api';
import { chatSessionApi, projectApi, providerApi, type ChatSession, type ChatTurn, type ChatTurnBlock, ChatTurnStats, ChatSessionMode, type AiProvider, type Project, type IWorkOrderCompleteStats, type ISearchResult } from '../lib/api';
import { WorkspaceMode } from '../lib/types';
import SessionPanel from '../components/SessionPanel';
import ProjectPanel from '../components/ProjectPanel';
@ -12,6 +12,9 @@ import ChatTurnComponent from '../components/ChatTurn';
import EditorPanel from '../components/EditorPanel';
import ErrorBoundary from '../components/ErrorBoundary';
import ExpandedPromptEditor from '../components/ExpandedPromptEditor';
import SearchInput from '../components/SearchInput';
import ChatSearchResults from '../components/ChatSearchResults';
import DroneSelectionModal from '../components/DroneSelectionModal';
import { AppContext } from '../App';
interface LogEntry {
@ -56,6 +59,7 @@ enum SessionStartupState {
interface RouterState {
drone?: any; // DroneRegistration passed via router state
project?: Project; // Project passed via router state (optional — can be fetched)
scrollToTurnId?: string; // Turn to scroll to after search result selection
}
export default function ChatSessionView() {
@ -104,6 +108,9 @@ export default function ChatSessionView() {
const [reconnectAttempts, setReconnectAttempts] = useState(0);
const [editorFilePath, setEditorFilePath] = useState<string | undefined>(undefined);
const [isExpandedEditor, setIsExpandedEditor] = useState(false);
const [searchQuery, setSearchQuery] = useState("");
const [showSearchResults, setShowSearchResults] = useState(false);
const [showDroneModal, setShowDroneModal] = useState(false);
const messagesEndRef = useRef<HTMLDivElement>(null);
const inputRef = useRef<HTMLTextAreaElement>(null);
@ -123,6 +130,23 @@ export default function ChatSessionView() {
loadSessionData();
}, [projectId, sessionId]);
// Scroll to turn if navigated from search results
useEffect(() => {
const routerState = location.state as RouterState | null;
const targetTurnId = routerState?.scrollToTurnId;
if (targetTurnId && turns.length > 0) {
// Clear the router state so we don't re-scroll on re-renders
window.history.replaceState({}, '');
// Use requestAnimationFrame to ensure DOM is painted
requestAnimationFrame(() => {
const el = document.getElementById(`turn-${targetTurnId}`);
if (el) {
el.scrollIntoView({ behavior: 'smooth', block: 'center' });
}
});
}
}, [turns, location.state]);
useEffect(() => {
setupSocketListeners();
return () => cleanupSocketListeners();
@ -310,9 +334,9 @@ export default function ChatSessionView() {
return; // Don't proceed to ready state
}
} else {
// No drone provided — skip lock step, go directly to Ready
// (This handles the case where the view is loaded directly via URL without a drone,
// e.g., for viewing completed sessions or task runs)
// No drone provided — show drone selection modal
// The user must select a drone to interact with the session
setShowDroneModal(true);
setStartupState(SessionStartupState.Ready);
}
@ -686,7 +710,7 @@ export default function ChatSessionView() {
}, 4000);
}, []);
const handleWorkOrderComplete = useCallback((turnId: string, success: boolean, message?: string) => {
const handleWorkOrderComplete = useCallback((turnId: string, success: boolean, message?: string, stats?: IWorkOrderCompleteStats) => {
// Backend has already flushed and persisted all streaming content
// Just clean up frontend streaming state and update status
if (streamingStateRef.current.has(turnId)) {
@ -734,6 +758,40 @@ export default function ChatSessionView() {
setIsAborting(false);
setIsProcessing(false);
currentTurnIdRef.current = null;
// Update turn stats from completion event
if (stats) {
setTurns(prev => prev.map(t => {
if (t._id !== turnId) return t;
return {
...t,
stats: {
...t.stats,
inputTokens: stats.masterInputTokens,
thinkingTokenCount: stats.masterThinkingTokens,
responseTokens: stats.masterOutputTokens,
toolCallCount: stats.toolCallCount,
durationMs: stats.durationMs,
durationLabel: stats.durationMs < 1000
? `${stats.durationMs}ms`
: `${(stats.durationMs / 1000).toFixed(1)}s`,
},
};
}));
// Update session contextWindowUsage for fuel gauge
// Uses aggregate totals (master+subagent) when available, falls back to master-only
setSession(prev => prev ? {
...prev,
contextWindowUsage: (prev.contextWindowUsage ?? 0) + stats.masterInputTokens,
stats: {
...prev.stats,
inputTokens: prev.stats.inputTokens + (stats.totalInputTokens ?? stats.masterInputTokens),
outputTokens: prev.stats.outputTokens + (stats.totalOutputTokens ?? stats.masterOutputTokens),
toolCallCount: prev.stats.toolCallCount + (stats.totalToolCallCount ?? stats.toolCallCount),
},
} : prev);
}
}, [showToast]);
const handleWorkspaceModeChanged = useCallback((mode: string) => {
@ -1186,9 +1244,57 @@ export default function ChatSessionView() {
}, [isProcessing, showToast, handleCancel]);
const scrollToBottom = () => {
messagesEndRef.current?.scrollIntoView({ behavior: 'smooth' });
messagesEndRef.current?.scrollIntoView({ behavior: 'instant' });
};
// ── Search handlers ──
const handleSearch = useCallback((query: string) => {
setSearchQuery(query);
setShowSearchResults(true);
}, []);
const handleSearchSelect = useCallback((result: ISearchResult) => {
setShowSearchResults(false);
setSearchQuery("");
// Scroll to the turn within this session view
const el = document.getElementById(`turn-${result.turnId}`);
if (el) {
el.scrollIntoView({ behavior: 'smooth', block: 'center' });
}
}, []);
const handleSearchClear = useCallback(() => {
setSearchQuery("");
setShowSearchResults(false);
}, []);
// ── Drone selection handler ──
const handleDroneSelect = useCallback(async (drone: any) => {
setDroneRegistration(drone);
droneRegistrationRef.current = drone;
setShowDroneModal(false);
// Persist and request lock
localStorage.setItem('dtp_drone_registration', JSON.stringify(drone));
try {
const success = await socketClient.requestSessionLock(
drone,
projectRef.current,
sessionRef.current,
);
if (success) {
setStartupState(SessionStartupState.Ready);
appContext?.setStatusMessage('Session ready');
} else {
setStartupState(SessionStartupState.LockFailed);
appContext?.setStatusMessage('Failed to lock drone — drone may be busy');
}
} catch (err) {
setStartupState(SessionStartupState.LockFailed);
appContext?.setStatusMessage('Failed to lock drone');
}
}, [session, appContext]);
const sessionLocked = startupState === SessionStartupState.Ready;
const promptDisabled = isProcessing || workspaceMode !== WorkspaceMode.Agent || !sessionLocked || !session?.selectedModel || isEditingProvider;
@ -1332,7 +1438,16 @@ export default function ChatSessionView() {
</div>
{/* Prompt Input */}
<div className="border-t border-border-subtle p-4 bg-bg-secondary shrink-0">
<div className="border-t border-border-subtle bg-bg-secondary shrink-0">
{/* Search bar */}
<div className="px-4 pt-3 pb-1">
<SearchInput
placeholder="Search in session"
onSearch={handleSearch}
onClear={handleSearchClear}
/>
</div>
<div className="p-4 pt-2">
<form onSubmit={handleSubmitPrompt} className="flex gap-2">
<div className="flex flex-col self-end">
<button
@ -1390,6 +1505,7 @@ export default function ChatSessionView() {
</form>
</div>
</div>
</div>
)}
{/* Log Panel (25%) */}
@ -1402,6 +1518,20 @@ export default function ChatSessionView() {
{/* Sidebar */}
<div className="w-80 border-l border-border-subtle bg-bg-secondary flex flex-col">
{/* Drone Selection Button (when no drone is registered) */}
{!droneRegistration && (
<div className="px-4 py-3 border-b border-border-subtle">
<button
onClick={() => setShowDroneModal(true)}
className="w-full px-3 py-2 bg-brand text-white rounded hover:bg-red-700 transition-colors text-sm font-medium"
>
Select Drone
</button>
<p className="text-xs text-text-muted mt-1.5">
A drone is required to send prompts and interact with the session.
</p>
</div>
)}
<SessionPanel
session={session}
workspaceMode={workspaceMode}
@ -1441,6 +1571,7 @@ export default function ChatSessionView() {
truncateModelName={truncateModelName}
getSelectedModelCapabilities={getSelectedModelCapabilities}
getSelectedModel={getSelectedModel}
contextWindowUsage={session?.contextWindowUsage ?? 0}
/>
<ProjectPanel
@ -1455,6 +1586,20 @@ export default function ChatSessionView() {
}}
/>
</div>
{showSearchResults && searchQuery && session && (
<ChatSearchResults
query={searchQuery}
filters={{ sessionId: session._id }}
onSelect={handleSearchSelect}
onClose={handleSearchClear}
/>
)}
{showDroneModal && (
<DroneSelectionModal
onSelect={handleDroneSelect}
onClose={() => setShowDroneModal(false)}
/>
)}
</div>
);
}

View File

@ -1,12 +1,14 @@
import { useState, useEffect, useRef } from "react";
import { Link, useNavigate } from "react-router-dom";
import type { User, Project, DroneRegistration, ChatSession, SubProcessStat } from "../lib/api";
import type { User, Project, DroneRegistration, ChatSession, SubProcessStat, ISearchResult } from "../lib/api";
import { projectApi, droneApi, chatSessionApi } from "../lib/api";
import { socketClient } from "../lib/socket";
import Clock from "../components/Clock";
import GadgetGrid from "../components/GadgetGrid";
import LogRenderer from "../components/LogRenderer";
import type { LogRendererEntry } from "../components/LogRenderer";
import SearchInput from "../components/SearchInput";
import ChatSearchResults from "../components/ChatSearchResults";
function SystemReady() {
return (
@ -269,53 +271,12 @@ function DashboardSidebar({
};
return (
<aside className="w-64 border-l border-border-subtle bg-bg-secondary overflow-y-auto">
<aside className="w-64 border-l border-border-subtle bg-bg-secondary flex flex-col overflow-hidden min-h-0">
<Clock />
<div className="p-3 border-t border-border-subtle">
<div className="text-xs font-semibold text-text-muted uppercase tracking-wider mb-2">
Recent Chats
</div>
{loading ? (
<p className="text-sm text-text-muted px-2">Loading...</p>
) : recentChats.length === 0 ? (
<p className="text-sm text-text-muted px-2">No recent chats.</p>
) : (
<div className="space-y-1">
{recentChats.map((session) => (
<button
key={session._id}
onClick={() => handleOpenChat(session)}
className="w-full text-left px-2 py-1.5 text-sm text-text-secondary hover:bg-bg-tertiary hover:text-text-primary rounded transition-colors"
>
<div className="flex items-center justify-between gap-1">
<span className="truncate font-medium text-xs">{session.name || 'Untitled'}</span>
<span className={`text-[10px] px-1.5 py-0.5 rounded-full font-medium shrink-0 ${modeBadgeColors[session.mode] || 'bg-gray-500/20 text-gray-400'}`}>
{session.mode}
</span>
</div>
<div className="flex items-center justify-between gap-1 mt-0.5">
<span className="text-[11px] text-text-muted truncate">{getProjectName(session)}</span>
<span className="text-[10px] text-text-muted shrink-0">
{formatRelativeTime(session.lastMessageAt || session.createdAt)}
</span>
</div>
</button>
))}
</div>
)}
{noDroneMessage && (
<p className="text-xs text-yellow-400 mt-2 px-2">{noDroneMessage}</p>
)}
{selectedDrone && (
<p className="text-[10px] text-text-muted mt-2 px-2 italic">
Select a chat to resume with {selectedDrone.hostname}
</p>
)}
</div>
<div className="p-3 border-t border-border-subtle">
<div className="flex items-center justify-between text-xs font-semibold text-text-muted uppercase tracking-wider mb-2">
{/* Projects */}
<div className="flex-1 min-h-0 flex flex-col border-t border-border-subtle">
<div className="shrink-0 px-3 pt-3 pb-2 flex items-center justify-between text-xs font-semibold text-text-muted uppercase tracking-wider">
<span>Projects</span>
<Link
to="/projects"
@ -324,6 +285,7 @@ function DashboardSidebar({
[+]
</Link>
</div>
<div className="flex-1 min-h-0 overflow-y-auto px-3 pb-3">
{loading ? (
<p className="text-sm text-text-muted px-2">Loading...</p>
) : projects.length === 0 ? (
@ -342,9 +304,11 @@ function DashboardSidebar({
</div>
)}
</div>
</div>
<div className="p-3 border-t border-border-subtle">
<div className="flex items-center justify-between text-xs font-semibold text-text-muted uppercase tracking-wider mb-2">
{/* Drones */}
<div className="flex-1 min-h-0 flex flex-col border-t border-border-subtle">
<div className="shrink-0 px-3 pt-3 pb-2 flex items-center justify-between text-xs font-semibold text-text-muted uppercase tracking-wider">
<span>Drones</span>
<Link
to="/drones"
@ -372,6 +336,7 @@ function DashboardSidebar({
</svg>
</Link>
</div>
<div className="flex-1 min-h-0 overflow-y-auto px-3 pb-3">
{loading ? (
<p className="text-sm text-text-muted px-2">Loading...</p>
) : drones.length === 0 ? (
@ -418,6 +383,54 @@ function DashboardSidebar({
</div>
)}
</div>
</div>
{/* Recent Chats */}
<div className="flex-1 min-h-0 flex flex-col border-t border-border-subtle">
<div className="shrink-0 px-3 pt-3 pb-2 text-xs font-semibold text-text-muted uppercase tracking-wider">
Recent Chats
</div>
<div className="flex-1 min-h-0 overflow-y-auto px-3 pb-3">
{loading ? (
<p className="text-sm text-text-muted px-2">Loading...</p>
) : recentChats.length === 0 ? (
<p className="text-sm text-text-muted px-2">No recent chats.</p>
) : (
<div className="space-y-1">
{recentChats.map((session) => (
<button
key={session._id}
onClick={() => handleOpenChat(session)}
className="w-full text-left px-2 py-1.5 text-sm text-text-secondary hover:bg-bg-tertiary hover:text-text-primary rounded transition-colors"
>
<div className="flex items-center justify-between gap-1">
<span className="truncate font-medium text-xs">{session.name || 'Untitled'}</span>
<span className={`text-[10px] px-1.5 py-0.5 rounded-full font-medium shrink-0 ${modeBadgeColors[session.mode] || 'bg-gray-500/20 text-gray-400'}`}>
{session.mode}
</span>
</div>
<div className="flex items-center justify-between gap-1 mt-0.5">
<span className="text-[11px] text-text-muted truncate">{getProjectName(session)}</span>
<span className="text-[10px] text-text-muted shrink-0">
{formatRelativeTime(session.lastMessageAt || session.createdAt)}
</span>
</div>
</button>
))}
</div>
)}
</div>
<div className="shrink-0 px-3 pb-3">
{noDroneMessage && (
<p className="text-xs text-yellow-400 mt-2 px-2">{noDroneMessage}</p>
)}
{selectedDrone && (
<p className="text-[10px] text-text-muted mt-2 px-2 italic">
Select a chat to resume with {selectedDrone.hostname}
</p>
)}
</div>
</div>
</aside>
);
@ -432,6 +445,27 @@ export default function Home({ user }: HomeProps) {
const [selectedDrone, setSelectedDrone] = useState<DroneRegistration | null>(
null,
);
const [searchQuery, setSearchQuery] = useState("");
const [showSearchResults, setShowSearchResults] = useState(false);
const handleSearch = (query: string) => {
setSearchQuery(query);
setShowSearchResults(true);
};
const handleSearchSelect = (result: ISearchResult) => {
setShowSearchResults(false);
setSearchQuery("");
// Navigate to the chat session — the turn will be scrolled to via id attr
navigate(`/projects/${result.projectId}/chat-session/${result.sessionId}`, {
state: { scrollToTurnId: result.turnId },
});
};
const handleSearchClear = () => {
setSearchQuery("");
setShowSearchResults(false);
};
if (!user) {
return (
@ -447,7 +481,7 @@ export default function Home({ user }: HomeProps) {
}
const mainContent = selectedDrone ? (
<div className="relative z-10 flex-1 flex">
<div className="relative z-10 flex-1 flex min-h-0">
<DroneInspector
drone={selectedDrone}
onClose={() => setSelectedDrone(null)}
@ -463,10 +497,17 @@ export default function Home({ user }: HomeProps) {
/>
</div>
) : (
<div className="relative z-10 flex-1 flex flex-col">
<div className="flex-1 flex">
<div className="relative z-10 flex-1 flex flex-col min-h-0">
<div className="flex-1 flex min-h-0">
<div className="flex-1 flex items-center justify-center p-8">
<div className="text-center">
<div className="text-center w-full max-w-lg">
<div className="mb-6">
<SearchInput
placeholder="Search your work"
onSearch={handleSearch}
onClear={handleSearchClear}
/>
</div>
<h1 className="text-2xl font-semibold mb-4">
Welcome, {user.displayName}!
</h1>
@ -505,6 +546,13 @@ export default function Home({ user }: HomeProps) {
<GadgetGrid />
</div>
{mainContent}
{showSearchResults && searchQuery && (
<ChatSearchResults
query={searchQuery}
onSelect={handleSearchSelect}
onClose={handleSearchClear}
/>
)}
</div>
);
}

View File

@ -1,7 +1,7 @@
import { useState, useEffect } from "react";
import { useNavigate, useParams } from "react-router-dom";
import slug from "slug";
import type { User, Project, ProjectSkill, ProjectTask } from "../lib/api";
import type { User, Project, ProjectSkill, ProjectTask, ISearchResult } from "../lib/api";
import {
projectApi,
droneApi,
@ -11,6 +11,8 @@ import {
type AiProvider,
providerApi,
} from "../lib/api";
import SearchInput from "../components/SearchInput";
import ChatSearchResults from "../components/ChatSearchResults";
import gabAiImage from "./assets/gab-ai.jpeg";
interface ProjectManagerProps {
@ -1501,6 +1503,26 @@ export default function ProjectManager({ user }: ProjectManagerProps) {
const [loading, setLoading] = useState(true);
const [showNewForm, setShowNewForm] = useState(false);
const [showPullForm, setShowPullForm] = useState(false);
const [searchQuery, setSearchQuery] = useState("");
const [showSearchResults, setShowSearchResults] = useState(false);
const handleSearch = (query: string) => {
setSearchQuery(query);
setShowSearchResults(true);
};
const handleSearchSelect = (result: ISearchResult) => {
setShowSearchResults(false);
setSearchQuery("");
navigate(`/projects/${result.projectId}/chat-session/${result.sessionId}`, {
state: { scrollToTurnId: result.turnId },
});
};
const handleSearchClear = () => {
setSearchQuery("");
setShowSearchResults(false);
};
useEffect(() => {
loadProjects();
@ -1688,12 +1710,21 @@ export default function ProjectManager({ user }: ProjectManagerProps) {
) : selectedProject ? (
<>
{/* Center - Project Inspector */}
<div className="flex-1 flex flex-col overflow-hidden">
<div className="px-6 pt-4 pb-2 border-b border-border-subtle">
<SearchInput
placeholder="Search in project"
onSearch={handleSearch}
onClear={handleSearchClear}
/>
</div>
<ProjectInspector
project={selectedProject}
providers={providers}
onDelete={handleProjectDeleted}
onUpdate={handleProjectUpdated}
/>
</div>
{/* Right Sidebar - Drones & Chat Sessions */}
<RightSidebar
@ -1712,6 +1743,14 @@ export default function ProjectManager({ user }: ProjectManagerProps) {
</div>
)}
</div>
{showSearchResults && searchQuery && (
<ChatSearchResults
query={searchQuery}
filters={selectedProject ? { projectId: selectedProject._id } : undefined}
onSelect={handleSearchSelect}
onClose={handleSearchClear}
/>
)}
</div>
);
}

View File

@ -25,6 +25,8 @@
"@gadget/ai": "workspace:*",
"@gadget/api": "workspace:*",
"@gadget/config": "workspace:*",
"@langchain/textsplitters": "^1.0.1",
"@qdrant/js-client-rest": "^1.12.0",
"@react-three/drei": "^10.7.7",
"@react-three/fiber": "^9.6.1",
"ansicolor": "^2.0.3",

View File

@ -105,6 +105,15 @@ export default {
host: yamlConfig.mongodb?.host || "localhost",
database: yamlConfig.mongodb?.database || "",
},
qdrant: {
host: yamlConfig.qdrant?.host || "localhost",
port: yamlConfig.qdrant?.port || 6333,
apiKey: yamlConfig.qdrant?.apiKey,
collection: yamlConfig.qdrant?.collection || "gadget-chat-embeddings",
providerId: yamlConfig.qdrant?.providerId || "",
embeddingModel: yamlConfig.qdrant?.embeddingModel || "nomic-embed-text",
vectorSize: yamlConfig.qdrant?.vectorSize || 768,
},
redis: {
host: yamlConfig.redis?.host || "localhost",
port: yamlConfig.redis?.port || 6379,

View File

@ -35,6 +35,7 @@ export class ApiControllerV1 extends DtpController {
await this.loadChild(path.join(basePath, "drone.js"));
await this.loadChild(path.join(basePath, "project.js"));
await this.loadChild(path.join(basePath, "provider.js"));
await this.loadChild(path.join(basePath, "search.js"));
await this.loadChild(path.join(basePath, "user.js"));
}
}

View File

@ -0,0 +1,170 @@
// src/controllers/api/v1/search.ts
// Copyright (C) 2026 Robert Colbert <rob.colbert@openplatform.us>
// All Rights Reserved
import { Request, Response } from "express";
import { DtpController } from "../../../lib/controller.js";
import VectorStoreService, { ISearchFilters, ISearchResult } from "../../../services/vector-store.js";
import User from "../../../models/user.js";
import Project from "../../../models/project.js";
import ChatSession from "../../../models/chat-session.js";
import ChatTurn from "../../../models/chat-turn.js";
interface IHydratedSearchResult extends ISearchResult {
user?: { _id: string; displayName: string; username: string };
project?: { _id: string; name: string; slug: string };
session?: { _id: string; name: string };
turn?: { _id: string; status: string; mode: string; llm: string };
}
interface ISearchRequestBody {
query: string;
userId?: string;
projectId?: string;
sessionId?: string;
turnId?: string;
topK?: number;
}
class SearchController extends DtpController {
get name(): string {
return "SearchController";
}
get slug(): string {
return "ctrl:api:v1:search";
}
get route(): string {
return "/search";
}
async start(): Promise<void> {
this.router.post("/", this.requireUser(), this.search.bind(this));
}
/**
* POST /api/v1/search
* Semantic search across chat history.
*
* Enforces userId filter: users can only search their own corpus.
*/
private async search(req: Request, res: Response): Promise<void> {
try {
const body = req.body as ISearchRequestBody;
if (!body.query?.trim()) {
res.status(400).json({
success: false,
message: "query is required",
});
return;
}
// Enforce that the user can only search their own corpus
const userId = (req.user as any)?._id;
if (!userId) {
res.status(403).json({
success: false,
message: "Authentication required",
});
return;
}
const filters: ISearchFilters = {
userId,
projectId: body.projectId,
sessionId: body.sessionId,
turnId: body.turnId,
};
const topK = body.topK && body.topK > 0 && body.topK <= 50
? body.topK
: 10;
// Check if vector store is initialized
if (!VectorStoreService.isReady) {
res.status(503).json({
success: false,
message: "Vector store is not configured. Set qdrant.providerId in gadget-code.yaml to enable semantic search.",
});
return;
}
const results = await VectorStoreService.search(
body.query.trim(),
filters,
topK,
);
// Hydrate results with MongoDB documents
const hydratedResults = await this.hydrateResults(results);
res.json({
success: true,
data: hydratedResults,
});
} catch (error) {
this.log.error("search failed", { error });
res.status(500).json({
success: false,
message: (error as Error).message || "Search failed",
});
}
}
/**
* Hydrate search results with user, project, session, and turn data from MongoDB.
* Collects unique IDs to batch-load documents efficiently.
*/
private async hydrateResults(
results: ISearchResult[],
): Promise<IHydratedSearchResult[]> {
if (results.length === 0) return [];
// Collect unique IDs for batch loading
const userIds = [...new Set(results.map((r) => r.userId).filter(Boolean))];
const projectIds = [...new Set(results.map((r) => r.projectId).filter(Boolean))];
const sessionIds = [...new Set(results.map((r) => r.sessionId).filter(Boolean))];
const turnIds = [...new Set(results.map((r) => r.turnId).filter(Boolean))];
// Batch load from MongoDB
const [users, projects, sessions, turns] = await Promise.all([
userIds.length > 0
? User.find({ _id: { $in: userIds } })
.select("_id displayName username")
.lean()
: [],
projectIds.length > 0
? Project.find({ _id: { $in: projectIds } })
.select("_id name slug")
.lean()
: [],
sessionIds.length > 0
? ChatSession.find({ _id: { $in: sessionIds } })
.select("_id name")
.lean()
: [],
turnIds.length > 0
? ChatTurn.find({ _id: { $in: turnIds } })
.select("_id status mode llm")
.lean()
: [],
]);
// Build lookup maps
const userMap = new Map(users.map((u) => [u._id, u]));
const projectMap = new Map(projects.map((p) => [p._id, p]));
const sessionMap = new Map(sessions.map((s) => [s._id, s]));
const turnMap = new Map(turns.map((t) => [t._id, t]));
return results.map((result) => ({
...result,
user: userMap.get(result.userId) as any || undefined,
project: projectMap.get(result.projectId) as any || undefined,
session: sessionMap.get(result.sessionId) as any || undefined,
turn: turnMap.get(result.turnId) as any || undefined,
}));
}
}
export default SearchController;

View File

@ -23,6 +23,7 @@ import {
SubmitPromptCallback,
AbortWorkOrderCallback,
SubProcessStat,
IWorkOrderCompleteStats,
} from "@gadget/api";
import ChatSession from "../models/chat-session.ts";
@ -669,7 +670,18 @@ export class CodeSession extends SocketSession {
turnId: string,
success: boolean,
message?: string,
stats?: IWorkOrderCompleteStats,
): void {
this.socket.emit("workOrderComplete", turnId, success, message);
this.socket.emit("workOrderComplete", turnId, success, message, stats);
// Update in-memory session stats for UI reactivity
// Uses aggregate totals (master + subagent) when available, falls back to master-only
if (stats && this.chatSession) {
this.chatSession.stats.inputTokens += stats.totalInputTokens ?? stats.masterInputTokens;
this.chatSession.stats.outputTokens += stats.totalOutputTokens ?? stats.masterOutputTokens;
this.chatSession.stats.toolCallCount += stats.totalToolCallCount ?? stats.toolCallCount;
// contextWindowUsage is intentionally master-only (fuel gauge tracks master context)
this.chatSession.contextWindowUsage = (this.chatSession.contextWindowUsage ?? 0) + stats.masterInputTokens;
}
}
}

View File

@ -12,6 +12,7 @@ import {
WorkspaceMode,
IChatToolCall,
IChatTurnBlock,
IWorkOrderCompleteStats,
} from "@gadget/api";
import {
GadgetSocket,
@ -19,7 +20,9 @@ import {
SocketSessionType,
} from "./socket-session.js";
import { SocketService } from "../services/index.js";
import { VectorStoreService } from "../services/index.js";
import { ChatTurn } from "../models/chat-turn.js";
import ChatSession from "../models/chat-session.js";
import MessageQueue, { type QueuedMessage } from "./message-queue.js";
interface IStreamingBuffer {
@ -292,6 +295,7 @@ export class DroneSession extends SocketSession {
turnId: string,
success: boolean,
message?: string,
stats?: IWorkOrderCompleteStats,
): Promise<void> {
if (!this.chatSessionId) {
this.log.warn("workOrderComplete event received but no chat session is active");
@ -303,6 +307,8 @@ export class DroneSession extends SocketSession {
await this.flushBuffer(turnId);
this.streamingBuffers.delete(turnId);
let enrichedStats: IWorkOrderCompleteStats | undefined;
const turn = await ChatTurn.findById(turnId);
if (turn) {
if (success && message === "aborted") {
@ -314,20 +320,73 @@ export class DroneSession extends SocketSession {
turn.errorMessage = message;
}
}
// UPDATE TURN STATS — master agent tokens only
if (stats) {
turn.stats.inputTokens = stats.masterInputTokens;
turn.stats.thinkingTokenCount = stats.masterThinkingTokens;
turn.stats.responseTokens = stats.masterOutputTokens;
turn.stats.toolCallCount = stats.toolCallCount;
turn.stats.durationMs = stats.durationMs;
turn.stats.durationLabel = this.formatDurationLabel(stats.durationMs);
}
await turn.save();
// Ingest to vector store (fire-and-forget for finished turns)
if (turn.status === ChatTurnStatus.Finished) {
VectorStoreService.ingestTurn(turnId).catch((err) => {
this.log.error("Failed to ingest turn to vector store", {
turnId,
error: err.message,
});
});
}
// COMPUTE AGGREGATE: walk the turn's subagent records for aggregate totals
let totalInputTokens = stats?.masterInputTokens ?? 0;
let totalOutputTokens = stats?.masterOutputTokens ?? 0;
let totalToolCallCount = stats?.toolCallCount ?? 0;
if (stats) {
for (const tc of turn.toolCalls) {
if ((tc as any).subagent?.stats) {
const subStats = (tc as any).subagent.stats;
totalInputTokens += subStats.inputTokens;
totalOutputTokens += subStats.responseTokens + subStats.thinkingTokenCount;
totalToolCallCount += subStats.toolCallCount;
}
}
}
// Build enriched stats with aggregate values for in-memory consumers
enrichedStats = stats
? { ...stats, totalInputTokens, totalOutputTokens, totalToolCallCount }
: undefined;
// INCREMENT SESSION STATS — aggregate totals (master + subagents)
await ChatSession.updateOne(
{ _id: this.chatSessionId },
{ $inc: {
"stats.inputTokens": totalInputTokens,
"stats.outputTokens": totalOutputTokens,
"stats.toolCallCount": totalToolCallCount,
// Master-only field for the fuel gauge
"contextWindowUsage": stats?.masterInputTokens ?? 0,
}},
);
}
const codeSession = SocketService.getCodeSessionByChatSessionId(
this.chatSessionId,
);
codeSession.onWorkOrderComplete(turnId, success, message);
codeSession.onWorkOrderComplete(turnId, success, message, enrichedStats);
this.currentTurnId = undefined;
} catch (error) {
// Routing failed - queue to Redis
await MessageQueue.enqueue(this.chatSessionId, {
type: 'workOrderComplete',
args: [turnId, success, message],
args: [turnId, success, message, stats],
timestamp: Date.now(),
});
this.log.debug("queued workOrderComplete message", { chatSessionId: this.chatSessionId });
@ -343,6 +402,17 @@ export class DroneSession extends SocketSession {
this.streamingBuffers.clear();
}
private formatDurationLabel(durationMs: number): string {
if (durationMs < 1000) return `${durationMs}ms`;
const seconds = Math.floor(durationMs / 1000);
const minutes = Math.floor(seconds / 60);
const secs = seconds % 60;
if (minutes > 0) {
return `${minutes}m ${secs}s`;
}
return `${seconds}.${Math.floor((durationMs % 1000) / 100)}s`;
}
/**
* Gets or creates a streaming buffer for a turn.
*/
@ -669,7 +739,8 @@ export class DroneSession extends SocketSession {
codeSession.onWorkOrderComplete(
msg.args[0] as string,
msg.args[1] as boolean,
msg.args[2] as string,
msg.args[2] as string | undefined,
msg.args[3] as IWorkOrderCompleteStats | undefined,
);
break;
case 'log':

View File

@ -35,6 +35,3 @@ export const ApiClientLog = model<IApiClientLog>(
);
export default ApiClientLog;
(async () => {
await ApiClientLog.syncIndexes();
})();

View File

@ -55,6 +55,3 @@ const ApiClientSchema = new Schema<IApiClient>({
export const ApiClient = model<IApiClient>("ApiClient", ApiClientSchema);
export default ApiClient;
(async () => {
await ApiClient.syncIndexes();
})();

View File

@ -31,6 +31,7 @@ export const ChatSessionSchema = new Schema<IChatSession>({
default: "off",
},
numCtx: { type: Number },
contextWindowUsage: { type: Number, default: 0, required: true },
stats: {
turnCount: { type: Number, default: 0, required: true },
toolCallCount: { type: Number, default: 0, required: true },

View File

@ -41,6 +41,3 @@ const CsrfTokenSchema = new Schema<ICsrfToken>({
export const CsrfToken = model<ICsrfToken>("CsrfToken", CsrfTokenSchema);
export default CsrfToken;
(async () => {
await CsrfToken.syncIndexes();
})();

View File

@ -41,6 +41,3 @@ export const DroneMonitor = model<IDroneMonitor>(
);
export default DroneMonitor;
(async () => {
await DroneMonitor.syncIndexes();
})();

View File

@ -34,6 +34,3 @@ export const DroneRegistration = model<IDroneRegistration>(
);
export default DroneRegistration;
(async () => {
await DroneRegistration.syncIndexes();
})();

View File

@ -29,6 +29,3 @@ export const EmailLogSchema = new Schema<IEmailLog>({
export const EmailLog = model<IEmailLog>("EmailLog", EmailLogSchema);
export default EmailLog;
(async () => {
await EmailLog.syncIndexes();
})();

View File

@ -38,6 +38,3 @@ export const EmailVerification = model<IEmailVerification>(
);
export default EmailVerification;
(async () => {
await EmailVerification.syncIndexes();
})();

View File

@ -21,6 +21,3 @@ IdeSessionSchema.index({
export const IdeSession = model<IIdeSession>("IdeSession", IdeSessionSchema);
export default IdeSession;
(async () => {
await IdeSession.syncIndexes();
})();

View File

@ -31,6 +31,3 @@ export const UserSchema = new Schema<IUser>({
export const User = model<IUser>("User", UserSchema);
export default User;
(async () => {
await User.syncIndexes();
})();

View File

@ -26,6 +26,3 @@ export const WebTokenSchema = new Schema<IWebToken>({
export const WebToken = model<IWebToken>("WebToken", WebTokenSchema);
export default WebToken;
(async () => {
await WebToken.syncIndexes();
})();

View File

@ -43,6 +43,3 @@ export const WebVisitSchema = new Schema<IWebVisit>({
});
export const WebVisit = model<IWebVisit>("WebVisit", WebVisitSchema);
(async () => {
await WebVisit.syncIndexes();
})();

View File

@ -131,6 +131,7 @@ class ChatSessionService extends DtpService {
inputTokens: 0,
outputTokens: 0,
},
contextWindowUsage: 0,
pins: [],
});

View File

@ -13,6 +13,7 @@ import SessionService from "./session.js";
import SocketService from "./socket.js";
import StorageService from "./storage.js";
import UserService from "./user.js";
import VectorStoreService from "./vector-store.js";
export async function startServices(): Promise<void> {
await ApiClientService.start();
@ -26,6 +27,7 @@ export async function startServices(): Promise<void> {
await SocketService.start();
await StorageService.start();
await UserService.start();
await VectorStoreService.start();
}
export async function stopServices(): Promise<void> {
@ -40,6 +42,7 @@ export async function stopServices(): Promise<void> {
await SocketService.stop();
await StorageService.stop();
await UserService.stop();
await VectorStoreService.stop();
}
export {
@ -54,4 +57,5 @@ export {
SocketService,
StorageService,
UserService,
VectorStoreService,
};

View File

@ -0,0 +1,495 @@
// src/services/vector-store.ts
// Copyright (C) 2026 Robert Colbert <rob.colbert@openplatform.us>
// All Rights Reserved
import { QdrantClient } from "@qdrant/js-client-rest";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
import env from "../config/env.js";
import { DtpService } from "../lib/service.js";
import { ChatTurnStatus } from "@gadget/api";
import {
createAiApi,
IAiEnvironment,
IAiProvider as IAiApiProvider,
AiApi,
} from "@gadget/ai";
import ChatTurn from "../models/chat-turn.js";
import AiProvider from "../models/ai-provider.js";
import { ChatSessionService } from "./index.js";
export interface ISearchFilters {
userId?: string;
projectId?: string;
sessionId?: string;
turnId?: string;
}
export interface ISearchResult {
id: string;
content: string;
score: number;
userId: string;
projectId: string;
sessionId: string;
turnId: string;
role: string;
createdAt: string;
}
const aiEnv: IAiEnvironment = {
NODE_ENV: env.NODE_ENV || "develop",
services: {
google: {
cse: {
apiKey: env.google.cse.apiKey,
engineId: env.google.cse.engineId,
},
},
},
};
class VectorStoreService extends DtpService {
private client!: QdrantClient;
private aiApi!: AiApi;
private splitter!: RecursiveCharacterTextSplitter;
private _initialized = false;
private _dimensionMismatch = false;
get name(): string {
return "VectorStoreService";
}
get slug(): string {
return "svc:vector-store";
}
async start(): Promise<void> {
if (!env.qdrant.providerId) {
this.log.warn(
"qdrant.providerId is not configured — vector store service will not start. " +
"Set providerId in gadget-code.yaml under qdrant to enable semantic search.",
);
return;
}
this.log.info("initializing Qdrant client", {
host: env.qdrant.host,
port: env.qdrant.port,
collection: env.qdrant.collection,
providerId: env.qdrant.providerId,
embeddingModel: env.qdrant.embeddingModel,
vectorSize: env.qdrant.vectorSize,
});
// Initialize Qdrant client
this.client = new QdrantClient({
url: `http://${env.qdrant.host}:${env.qdrant.port}`,
apiKey: env.qdrant.apiKey || undefined,
});
// Load the configured AI provider for embeddings
const providerDoc = await AiProvider.findById(env.qdrant.providerId);
if (!providerDoc) {
this.log.error(
`Qdrant providerId "${env.qdrant.providerId}" not found in database — ` +
"vector store service will not start.",
);
return;
}
const aiProvider: IAiApiProvider = {
_id: providerDoc._id,
name: providerDoc.name,
sdk: providerDoc.apiType, // map apiType → sdk
baseUrl: providerDoc.baseUrl,
apiKey: providerDoc.apiKey,
};
this.aiApi = createAiApi(aiEnv, aiProvider, this.log);
// Initialize text splitter
this.splitter = new RecursiveCharacterTextSplitter({
chunkSize: 1000,
chunkOverlap: 200,
});
// Ensure the Qdrant collection exists and validate dimensions
await this.ensureCollection();
// Validate that the embedding model produces vectors of the configured size
await this.validateEmbeddingDimensions();
if (this._dimensionMismatch) {
this.log.error(
"VectorStoreService started with DIMENSION MISMATCH — searches and ingestion will fail. " +
"See previous error logs for fix instructions.",
);
}
this._initialized = true;
this.log.info("started", {
host: env.qdrant.host,
port: env.qdrant.port,
collection: env.qdrant.collection,
providerId: env.qdrant.providerId,
embeddingModel: env.qdrant.embeddingModel,
vectorSize: env.qdrant.vectorSize,
dimensionMismatch: this._dimensionMismatch,
});
}
async stop(): Promise<void> {
this._initialized = false;
this.log.info("stopped");
}
/**
* Check if the service is initialized and ready.
*/
get isReady(): boolean {
return this._initialized;
}
/**
* Create the Qdrant collection if it doesn't already exist.
* Validates existing collection dimensions against the configured vectorSize.
*/
private async ensureCollection(): Promise<void> {
const collections = await this.client.getCollections();
const exists = collections.collections.some(
(c) => c.name === env.qdrant.collection,
);
if (!exists) {
await this.client.createCollection(env.qdrant.collection, {
vectors: {
size: env.qdrant.vectorSize,
distance: "Cosine" as const,
},
});
this.log.info(`created Qdrant collection "${env.qdrant.collection}"`, {
vectorSize: env.qdrant.vectorSize,
distance: "Cosine",
});
} else {
// Validate existing collection dimensions against config
try {
const collectionInfo = await this.client.getCollection(
env.qdrant.collection,
);
const vectorsConfig = collectionInfo.config?.params?.vectors;
// Handle both named and unnamed vector configurations
// Unnamed: { size: N, distance: "Cosine" } — Named: { "default": { size: N, distance: "Cosine" } }
const actualSize =
(vectorsConfig as Record<string, any>)?.size ??
(
Object.values(
(vectorsConfig as Record<string, any>) || {},
)[0] as Record<string, any>
)?.size;
if (actualSize && actualSize !== env.qdrant.vectorSize) {
this._dimensionMismatch = true;
this.log.error(
"QDRANT COLLECTION DIMENSION MISMATCH — searches and ingestion will fail!",
{
collectionName: env.qdrant.collection,
configuredVectorSize: env.qdrant.vectorSize,
actualCollectionVectorSize: actualSize,
fix: `Either (1) delete the collection and restart to recreate it, or (2) update qdrant.vectorSize in your config to ${actualSize}`,
},
);
} else {
this.log.info(
`Qdrant collection "${env.qdrant.collection}" already exists`,
{
vectorSize: actualSize || env.qdrant.vectorSize,
},
);
}
} catch (err) {
this.log.warn("Could not validate Qdrant collection dimensions", {
error: (err as Error).message,
});
}
}
}
/**
* Validate that the embedding model produces vectors matching the configured vectorSize.
* Sends a test embedding and compares its length against env.qdrant.vectorSize.
*/
private async validateEmbeddingDimensions(): Promise<void> {
try {
const testEmbedding = await this.getEmbedding("test");
if (testEmbedding.length !== env.qdrant.vectorSize) {
this._dimensionMismatch = true;
this.log.error(
"EMBEDDING MODEL DIMENSION MISMATCH — the configured vectorSize does not match the model output!",
{
configuredVectorSize: env.qdrant.vectorSize,
actualModelDimensions: testEmbedding.length,
embeddingModel: env.qdrant.embeddingModel,
fix: `Update qdrant.vectorSize in your config to ${testEmbedding.length}`,
},
);
} else {
this.log.info("embedding dimension validation passed", {
vectorSize: testEmbedding.length,
model: env.qdrant.embeddingModel,
});
}
} catch (err) {
this.log.warn("Could not validate embedding dimensions at startup", {
error: (err as Error).message,
});
}
}
/**
* Generate an embedding vector for the given text.
*/
private async getEmbedding(text: string): Promise<number[]> {
const response = await this.aiApi.embeddings(
env.qdrant.embeddingModel,
text,
env.qdrant.vectorSize,
);
return response.embedding;
}
/**
* Ingest a chat turn into the vector store.
* Fetches the turn by ID, extracts content, chunks, embeds, and upserts to Qdrant.
* Fire-and-forget safe logs errors but does not throw.
*/
async ingestTurn(turnId: string): Promise<void> {
if (!this._initialized) {
this.log.warn(
"ingestTurn called but service is not initialized — skipping",
{ turnId },
);
return;
}
if (this._dimensionMismatch) {
this.log.error(
"ingestTurn skipped — vector dimension mismatch detected at startup. Fix config and restart.",
{
turnId,
configuredVectorSize: env.qdrant.vectorSize,
},
);
return;
}
try {
// Fetch and populate the turn
let turn = await ChatTurn.findById(turnId);
if (!turn) {
this.log.warn("ingestTurn: turn not found", { turnId });
return;
}
// Only ingest finished turns
if (turn.status !== ChatTurnStatus.Finished) {
this.log.debug("ingestTurn: skipping non-finished turn", {
turnId,
status: turn.status,
});
return;
}
// Populate references for metadata
turn = await ChatTurn.populate(turn, ChatSessionService.populateChatTurn);
// Extract content
const userPrompt = turn.prompts.user || "";
// Extract agent response: concatenate all 'responding' mode blocks
let agentResponse = "";
for (const block of turn.blocks) {
if (block.mode === "responding" && typeof block.content === "string") {
agentResponse += block.content + "\n";
}
}
// Combine user + agent text for chunking
const combinedText = [
userPrompt ? `User: ${userPrompt}` : "",
agentResponse ? `Agent: ${agentResponse.trim()}` : "",
]
.filter(Boolean)
.join("\n\n");
if (!combinedText.trim()) {
this.log.debug("ingestTurn: no content to ingest", { turnId });
return;
}
// Chunk the text
const chunks = await this.splitter.splitText(combinedText);
// Determine role for each chunk based on content
const points = [];
for (let i = 0; i < chunks.length; i++) {
const chunk = chunks[i]!;
let role: string;
if (chunk.startsWith("User:") && !chunk.includes("Agent:")) {
role = "user";
} else if (chunk.startsWith("Agent:") && !chunk.includes("User:")) {
role = "agent";
} else {
role = "both";
}
const embedding = await this.getEmbedding(chunk);
// Validate embedding dimensions before upsert
if (embedding.length !== env.qdrant.vectorSize) {
this.log.error(
"embedding dimension mismatch during ingest — skipping chunk",
{
expected: env.qdrant.vectorSize,
actual: embedding.length,
turnId,
chunkIndex: i,
fix: `Update qdrant.vectorSize in your config to ${embedding.length}`,
},
);
continue;
}
points.push({
id: `${turnId}:${i}`,
vector: embedding,
payload: {
content: chunk,
userId: String(turn.user),
projectId: turn.project ? String(turn.project) : "",
sessionId: String(turn.session),
turnId: turn._id,
role,
createdAt: turn.createdAt.toISOString(),
},
});
}
// Upsert all points
await this.client.upsert(env.qdrant.collection, {
wait: false,
points,
});
this.log.info("ingested turn to vector store", {
turnId,
chunkCount: points.length,
});
} catch (error) {
this.log.error("ingestTurn failed", {
turnId,
error,
});
// Do not rethrow — this is designed to be fire-and-forget
}
}
/**
* Search the vector store for relevant chunks.
*/
async search(
query: string,
filters?: ISearchFilters,
topK: number = 10,
): Promise<ISearchResult[]> {
if (!this._initialized) {
throw new Error("VectorStoreService is not initialized");
}
if (this._dimensionMismatch) {
throw new Error(
`Vector dimension mismatch: the Qdrant collection dimensions do not match the ` +
`configured vectorSize (${env.qdrant.vectorSize}). Either delete the collection ` +
`and restart, or update qdrant.vectorSize in your config.`,
);
}
const queryVector = await this.getEmbedding(query);
// Validate embedding dimensions before searching
if (queryVector.length !== env.qdrant.vectorSize) {
throw new Error(
`Embedding dimension mismatch: model produced ${queryVector.length} dimensions, ` +
`but collection expects ${env.qdrant.vectorSize}. ` +
`Update qdrant.vectorSize in your config to ${queryVector.length}.`,
);
}
// Build Qdrant filter from provided filters
const must: Array<{
key: string;
match: { value: string };
}> = [];
if (filters?.userId) {
must.push({ key: "userId", match: { value: filters.userId } });
}
if (filters?.projectId) {
must.push({ key: "projectId", match: { value: filters.projectId } });
}
if (filters?.sessionId) {
must.push({ key: "sessionId", match: { value: filters.sessionId } });
}
if (filters?.turnId) {
must.push({ key: "turnId", match: { value: filters.turnId } });
}
const searchResults = await this.client.search(env.qdrant.collection, {
vector: queryVector,
limit: topK,
filter: must.length > 0 ? { must } : undefined,
with_payload: true,
});
return searchResults.map((point) => ({
id: point.id as string,
content: (point.payload as any)?.content || "",
score: point.score ?? 0,
userId: (point.payload as any)?.userId || "",
projectId: (point.payload as any)?.projectId || "",
sessionId: (point.payload as any)?.sessionId || "",
turnId: (point.payload as any)?.turnId || "",
role: (point.payload as any)?.role || "",
createdAt: (point.payload as any)?.createdAt || "",
}));
}
/**
* Remove all vector points for a given turn.
* Used for cleanup when a turn is deleted.
*/
async removeTurnPoints(turnId: string): Promise<void> {
if (!this._initialized) {
this.log.warn(
"removeTurnPoints called but service is not initialized — skipping",
{ turnId },
);
return;
}
try {
await this.client.delete(env.qdrant.collection, {
filter: {
must: [{ key: "turnId", match: { value: turnId } }],
},
});
this.log.info("removed vector points for turn", { turnId });
} catch (error) {
this.log.error("removeTurnPoints failed", {
turnId,
error,
});
}
}
}
export default new VectorStoreService();

View File

@ -1,5 +1,4 @@
import { describe, it, expect, beforeEach, vi } from "vitest";
import { Types } from "mongoose";
import { DroneSession } from "../src/lib/drone-session";
import {
IDroneRegistration,
@ -14,6 +13,27 @@ import { nanoid } from "nanoid";
// Mock dependencies
vi.mock("../src/services/socket");
vi.mock("../src/models/chat-turn");
vi.mock("../src/models/chat-session", () => ({
default: {
updateOne: vi.fn().mockResolvedValue({ acknowledged: true, modifiedCount: 1 }),
},
}));
vi.mock("../src/services/index.js", async (importOriginal) => {
const actual = await importOriginal() as any;
return {
...actual,
SocketService: actual.SocketService,
VectorStoreService: {
ingestTurn: vi.fn().mockResolvedValue(undefined),
},
};
});
vi.mock("../src/lib/message-queue.js", () => ({
default: {
enqueue: vi.fn().mockResolvedValue(undefined),
drain: vi.fn().mockResolvedValue([]),
},
}));
describe("DroneSession", () => {
let mockSocket: any;
@ -220,6 +240,7 @@ describe("DroneSession", () => {
};
const mockTurn = {
status: ChatTurnStatus.Processing,
toolCalls: [],
save: vi.fn().mockResolvedValue(undefined),
};
vi.mocked(SocketService.getCodeSessionByChatSessionId).mockReturnValue(
@ -238,6 +259,7 @@ describe("DroneSession", () => {
mockTurnId,
true,
undefined,
undefined,
);
expect(droneSession.currentTurnId).toBeUndefined();
});
@ -250,6 +272,7 @@ describe("DroneSession", () => {
const mockTurn = {
status: ChatTurnStatus.Processing,
errorMessage: "",
toolCalls: [],
save: vi.fn().mockResolvedValue(undefined),
};
vi.mocked(SocketService.getCodeSessionByChatSessionId).mockReturnValue(

View File

@ -0,0 +1,320 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
const mockEmbeddings = vi.hoisted(() => vi.fn());
const mockFindById = vi.hoisted(() =>
vi.fn().mockResolvedValue({
_id: "test-provider-id",
name: "Test Provider",
apiType: "ollama",
baseUrl: "http://test:11434",
apiKey: "",
}),
);
// Create mock methods shared between QdrantClient mock and test code
const mockQdrantMethods = vi.hoisted(() => ({
getCollections: vi.fn(),
createCollection: vi.fn(),
getCollection: vi.fn(),
search: vi.fn(),
upsert: vi.fn(),
delete: vi.fn(),
}));
vi.mock("../src/config/env.js", () => ({
default: {
NODE_ENV: "test",
INSTALL_DIR: "/tmp",
timezone: "UTC",
pkg: { name: "test", version: "0.0.0" },
site: {},
ai: { ollama: { apiUrl: "http://test:11434", apiKey: "" } },
auth: { jwtSecret: "test-secret" },
session: {
secret: "test-secret",
trustProxy: false,
cookie: { secure: false, sameSite: false },
},
google: { cse: { apiKey: "", engineId: "" } },
mongodb: { host: "localhost:27017", database: "test" },
qdrant: {
host: "localhost",
port: 6333,
collection: "test-collection",
providerId: "test-provider-id",
embeddingModel: "test-model",
vectorSize: 1024,
},
redis: {
host: "localhost",
port: 6379,
password: "",
keyPrefix: "test:",
lazyConnect: true,
},
minio: {
endpoint: "localhost",
port: 9000,
useSsl: false,
accessKey: "",
secretKey: "",
buckets: { uploads: "", images: "", videos: "", audios: "" },
},
user: { passwordSalt: "test-salt" },
https: { enabled: false, address: "localhost", port: 3443 },
socket: { maxHttpBufferSize: 1048576 },
frontend: { port: 5173 },
email: { enabled: false, smtp: {}, contact: {} },
log: {
https: { enabled: false },
console: { enabled: false },
file: { enabled: false },
},
},
}));
vi.mock("../src/models/ai-provider.js", () => ({
default: { findById: mockFindById },
}));
vi.mock("@gadget/ai", () => ({
createAiApi: vi.fn().mockReturnValue({
embeddings: mockEmbeddings,
}),
}));
vi.mock("@langchain/textsplitters", () => ({
RecursiveCharacterTextSplitter: vi.fn(function () {
return {
splitText: vi.fn().mockResolvedValue(["chunk1"]),
};
}),
}));
// QdrantClient mock constructor that returns the shared mock methods
vi.mock("@qdrant/js-client-rest", () => ({
QdrantClient: vi.fn(function () {
return mockQdrantMethods;
}),
}));
import VectorStoreService from "../src/services/vector-store.js";
const svc = VectorStoreService as unknown as {
_initialized: boolean;
_dimensionMismatch: boolean;
client: typeof mockQdrantMethods;
aiApi: {
embeddings: ReturnType<typeof vi.fn>;
};
};
function setMockDefaults() {
mockQdrantMethods.getCollections.mockResolvedValue({ collections: [] });
mockQdrantMethods.createCollection.mockResolvedValue(undefined);
mockQdrantMethods.getCollection.mockResolvedValue({});
mockQdrantMethods.search.mockResolvedValue([]);
mockQdrantMethods.upsert.mockResolvedValue(undefined);
mockQdrantMethods.delete.mockResolvedValue(undefined);
}
function ensureInstanceSetup() {
// For tests that don't call start(), manually assign client/aiApi
// since start() normally does this via new QdrantClient() + createAiApi()
if (!svc.client) {
(svc as Record<string, unknown>).client = mockQdrantMethods;
}
if (!svc.aiApi) {
(svc as Record<string, unknown>).aiApi = { embeddings: mockEmbeddings };
}
}
describe("VectorStoreService", () => {
beforeEach(() => {
vi.clearAllMocks();
setMockDefaults();
// Reset internal state on the instance
svc._initialized = false;
svc._dimensionMismatch = false;
});
describe("search", () => {
beforeEach(() => {
ensureInstanceSetup();
});
it("throws when service is not initialized", async () => {
svc._initialized = false;
svc._dimensionMismatch = false;
await expect(VectorStoreService.search("test query")).rejects.toThrow(
"VectorStoreService is not initialized",
);
});
it("throws when dimension mismatch flag is set", async () => {
svc._initialized = true;
svc._dimensionMismatch = true;
await expect(VectorStoreService.search("test query")).rejects.toThrow(
"Vector dimension mismatch: the Qdrant collection dimensions do not match the configured vectorSize (1024)",
);
});
it("throws when query embedding dimensions mismatch config", async () => {
svc._initialized = true;
svc._dimensionMismatch = false;
mockEmbeddings.mockResolvedValue({
embedding: new Array(512).fill(0.1),
model: "test-model",
});
await expect(VectorStoreService.search("test query")).rejects.toThrow(
"Embedding dimension mismatch: model produced 512 dimensions, but collection expects 1024",
);
});
it("passes vectorSize dimensions to the AI API when embedding", async () => {
svc._initialized = true;
svc._dimensionMismatch = false;
mockQdrantMethods.search.mockResolvedValue([]);
mockEmbeddings.mockResolvedValue({
embedding: new Array(1024).fill(0.1),
model: "test-model",
});
await VectorStoreService.search("test query");
expect(mockEmbeddings).toHaveBeenCalledWith(
"test-model",
"test query",
1024,
);
});
it("returns hydrated search results", async () => {
svc._initialized = true;
svc._dimensionMismatch = false;
mockEmbeddings.mockResolvedValue({
embedding: new Array(1024).fill(0.1),
model: "test-model",
});
mockQdrantMethods.search.mockResolvedValue([
{
id: "point-1",
score: 0.95,
payload: {
content: "some content",
userId: "user-1",
projectId: "proj-1",
sessionId: "session-1",
turnId: "turn-1",
role: "user",
createdAt: "2026-01-01T00:00:00.000Z",
},
},
]);
const results = await VectorStoreService.search("test query", undefined, 5);
expect(results).toHaveLength(1);
expect(results[0]).toMatchObject({
id: "point-1",
content: "some content",
score: 0.95,
userId: "user-1",
});
});
});
describe("start / ensureCollection", () => {
it("creates collection when it does not exist", async () => {
mockQdrantMethods.getCollections.mockResolvedValue({
collections: [],
});
mockQdrantMethods.createCollection.mockResolvedValue(undefined);
mockEmbeddings.mockResolvedValue({
embedding: new Array(1024).fill(0.1),
model: "test-model",
});
await VectorStoreService.start();
expect(mockQdrantMethods.createCollection).toHaveBeenCalledWith(
"test-collection",
{ vectors: { size: 1024, distance: "Cosine" } },
);
expect(svc._initialized).toBe(true);
expect(svc._dimensionMismatch).toBe(false);
});
it("detects dimension mismatch when existing collection has wrong size", async () => {
mockQdrantMethods.getCollections.mockResolvedValue({
collections: [{ name: "test-collection" }],
});
mockQdrantMethods.getCollection.mockResolvedValue({
config: {
params: {
vectors: { size: 768, distance: "Cosine" },
},
},
});
mockEmbeddings.mockResolvedValue({
embedding: new Array(1024).fill(0.1),
model: "test-model",
});
await VectorStoreService.start();
expect(svc._initialized).toBe(true);
expect(svc._dimensionMismatch).toBe(true);
});
it("sets dimension mismatch when embedding model produces wrong dimensions", async () => {
mockQdrantMethods.getCollections.mockResolvedValue({
collections: [],
});
mockQdrantMethods.createCollection.mockResolvedValue(undefined);
mockEmbeddings.mockResolvedValue({
embedding: new Array(512).fill(0.1),
model: "test-model",
});
await VectorStoreService.start();
expect(svc._initialized).toBe(true);
expect(svc._dimensionMismatch).toBe(true);
});
it("starts cleanly when everything matches", async () => {
mockQdrantMethods.getCollections.mockResolvedValue({
collections: [],
});
mockQdrantMethods.createCollection.mockResolvedValue(undefined);
mockEmbeddings.mockResolvedValue({
embedding: new Array(1024).fill(0.1),
model: "test-model",
});
await VectorStoreService.start();
expect(svc._dimensionMismatch).toBe(false);
expect(svc._initialized).toBe(true);
});
it("skips start when no providerId is configured", async () => {
const env = await import("../src/config/env.js");
const originalProviderId = env.default.qdrant.providerId;
env.default.qdrant.providerId = "";
await VectorStoreService.start();
expect(svc._initialized).toBe(false);
env.default.qdrant.providerId = originalProviderId;
});
});
});

View File

@ -39,6 +39,7 @@ describe("AgentService", () => {
inputTokens: 0,
outputTokens: 0,
},
contextWindowUsage: 0,
pins: [],
};
const workOrder: IAgentWorkOrder = {
@ -150,6 +151,7 @@ describe("AgentService", () => {
inputTokens: 0,
outputTokens: 0,
},
contextWindowUsage: 0,
pins: [],
};
@ -314,6 +316,7 @@ describe("AgentService", () => {
inputTokens: 0,
outputTokens: 0,
},
contextWindowUsage: 0,
pins: [],
};
const rawResponse = "Subagent completed the task successfully.";

View File

@ -28,7 +28,9 @@ import {
type ServerToClientEvents,
type ClientToServerEvents,
ChatSessionMode,
ChatTurnStatus,
type IProject,
type IWorkOrderCompleteStats,
} from "@gadget/api";
import AiService from "./ai.ts";
@ -37,6 +39,7 @@ import WorkspaceService from "./workspace.ts";
import { GadgetService } from "../lib/service.ts";
import {
AiToolbox,
ExportTool,
FileEditTool,
FileReadTool,
FileWriteTool,
@ -57,6 +60,8 @@ import {
type DroneToolboxEnvironment,
} from "../tools/index.ts";
import type { IChatExportData } from "@gadget/ai-toolbox";
export interface IAgentWorkOrder {
createdAt: Date;
turn: IChatTurn;
@ -133,6 +138,11 @@ class AgentService extends GadgetService {
subagentTool.setSpawner((agentType: string, prompt: string) => this.spawnSubagent(agentType, prompt));
this.toolbox.register(subagentTool, readOnlyModes);
// Chat tools — export: available in all modes
const exportTool = new ExportTool(this.toolbox);
exportTool.setSessionDataProvider(() => this.getExportData());
this.toolbox.register(exportTool, readOnlyModes);
// Project tools — skill discovery: available in all modes
this.toolbox.register(new ListSkillsTool(this.toolbox), readOnlyModes);
this.toolbox.register(new ReadSkillTool(this.toolbox), readOnlyModes);
@ -156,8 +166,10 @@ class AgentService extends GadgetService {
const { turn } = workOrder;
let toolCallCount = 0;
let inputTokens = 0;
let outputTokens = 0;
let masterInputTokens = 0;
let masterOutputTokens = 0;
let masterThinkingTokens = 0;
const startTime = Date.now();
// Build the full message array that grows with each iteration
const messages: IContextChatMessage[] = [];
@ -250,6 +262,11 @@ class AgentService extends GadgetService {
throw new Error("Model failed to respond (still loading or error)");
}
// Accumulate real token counts from the AI API response
masterInputTokens += response.stats?.tokenCounts?.input ?? 0;
masterOutputTokens += response.stats?.tokenCounts?.response ?? 0;
masterThinkingTokens += response.stats?.tokenCounts?.thinking ?? 0;
// Process tool calls if present
if (response.toolCalls && response.toolCalls.length > 0) {
continueLoop = true;
@ -296,6 +313,7 @@ class AgentService extends GadgetService {
agentId: toolCall.callId,
response: responseText,
subagent: subagentData,
stats: subagentData?.stats as Record<string, unknown> | undefined,
});
} else {
socket.emit("toolCall", toolCall.callId, toolCall.function.name, toolCall.function.arguments, result);
@ -330,17 +348,30 @@ class AgentService extends GadgetService {
toolName: toolCall.function.name,
});
inputTokens += Math.ceil(toolCall.function.arguments.length / 4);
outputTokens += Math.ceil(result.length / 4);
}
}
}
socket.emit("workOrderComplete", turn._id, true);
const durationMs = Date.now() - startTime;
const stats: IWorkOrderCompleteStats = {
masterInputTokens,
masterOutputTokens,
masterThinkingTokens,
toolCallCount,
durationMs,
};
socket.emit("workOrderComplete", turn._id, true, undefined, stats);
} catch (cause) {
if (cause instanceof Error && cause.name === "AbortError") {
this.log.info("work order aborted by user", { turnId: turn._id });
socket.emit("workOrderComplete", turn._id, true, "aborted");
const abortStats: IWorkOrderCompleteStats = {
masterInputTokens,
masterOutputTokens,
masterThinkingTokens,
toolCallCount,
durationMs: Date.now() - startTime,
};
socket.emit("workOrderComplete", turn._id, true, "aborted", abortStats);
return;
}
const msg = cause instanceof Error ? cause.message : String(cause);
@ -348,7 +379,14 @@ class AgentService extends GadgetService {
turnId: turn._id,
error: msg,
});
socket.emit("workOrderComplete", turn._id, false, msg);
const errorStats: IWorkOrderCompleteStats = {
masterInputTokens,
masterOutputTokens,
masterThinkingTokens,
toolCallCount,
durationMs: Date.now() - startTime,
};
socket.emit("workOrderComplete", turn._id, false, msg, errorStats);
throw cause;
} finally {
this.abortController = null;
@ -540,6 +578,64 @@ class AgentService extends GadgetService {
return modelRecord?.settings;
}
/**
* Assembles all data needed for a chat session export, applying security
* scrubbing to remove PII, credentials, and infrastructure details.
* Only includes turns with finished/aborted/error status (excludes processing).
*/
private async getExportData(): Promise<IChatExportData> {
const workOrder = this.currentWorkOrder;
if (!workOrder) {
throw new Error("No active work order — cannot export");
}
const session = workOrder.turn.session as IChatSession;
if (!session) {
throw new Error("Chat session is not available on the current turn");
}
const user = session.user as IUser;
const provider = session.provider as IAiProvider;
const project = workOrder.turn.project as IProject;
// Combine context turns (already completed) with current turn if finished
const allTurns = [...workOrder.context];
if (workOrder.turn.status !== ChatTurnStatus.Processing) {
allTurns.push(workOrder.turn);
}
// Filter to only finished/aborted/error turns
const exportableTurns = allTurns.filter(
(t) => t.status !== ChatTurnStatus.Processing,
);
// Build the model config (excluding the provider reference — we send it separately scrubbed)
const reasoning = workOrder.turn.reasoningEffort === "off"
? false
: workOrder.turn.reasoningEffort || false;
const modelConfig = this.buildDroneModelConfig(
provider,
session.selectedModel,
reasoning as boolean | "low" | "medium" | "high",
session.numCtx,
);
return {
session,
turns: exportableTurns,
user,
provider,
project: {
name: project.name,
slug: project.slug,
description: project.description,
},
modelConfig,
gadgetCodeVersion: env.pkg.version,
exportedAt: new Date(),
};
}
private async executeTool(name: string, argsJson: string): Promise<string> {
const tool = this.toolbox.getTool(name);
if (!tool) {
@ -674,11 +770,11 @@ class AgentService extends GadgetService {
inputTokens += response.stats?.tokenCounts?.input ?? 0;
if (response.thinking) {
thinkingTokenCount += Math.ceil(response.thinking.length / 4);
thinkingTokenCount += response.stats?.tokenCounts?.thinking ?? 0;
}
if (response.response) {
fullResponse += response.response;
outputTokens += Math.ceil(response.response.length / 4);
outputTokens += response.stats?.tokenCounts?.response ?? 0;
}
if (response.thinking) {
fullThinking = (fullThinking + response.thinking).trim();
@ -734,8 +830,6 @@ class AgentService extends GadgetService {
toolName: toolCall.function.name,
});
inputTokens += Math.ceil(toolArgsRaw.length / 4);
outputTokens += Math.ceil(result.length / 4);
}
}
}

View File

@ -18,6 +18,7 @@ export {
PlanFileWriteTool,
PlanFileEditTool,
PlanListTool,
ExportTool,
SubagentTool,
ListSkillsTool,
ReadSkillTool,

View File

@ -1,11 +1,12 @@
{
"name": "gadget-workspace",
"version": "1.0.1",
"version": "1.0.2",
"private": true,
"description": "Gadget Code monorepo workspace root",
"packageManager": "pnpm@10.33.2",
"license": "Apache-2.0",
"scripts": {
"clean": "rm -rf packages/ai/dist packages/api/dist packages/config/dist gadget-code/dist gadget-drone/dist",
"link:global": "pnpm --filter gadget-drone exec -- pnpm link --global && pnpm --filter gadget-tasks exec -- pnpm link --global"
}
}

View File

@ -0,0 +1,728 @@
// src/chat/export.ts
// Copyright (C) 2026 Rob Colbert <rob.colbert@openplatform.us>
// Licensed under the Apache License, Version 2.0
import fs from "node:fs/promises";
import path from "node:path";
import type { IAiLogger, IToolArguments, IToolDefinition } from "@gadget/ai";
import { formatError } from "@gadget/ai";
import type {
IChatSession,
IChatTurn,
IChatTurnBlock,
IChatTurnBlockThinking,
IChatTurnBlockResponding,
IChatTurnBlockTool,
IChatToolCall,
IChatSubagentProcess,
IChatTurnStats,
IAiProvider,
IDroneModelConfig,
IUser,
} from "@gadget/api";
import { ChatTurnStatus } from "@gadget/api";
import { GadgetTool } from "../tool.ts";
const VALID_FORMATS = ["markdown", "json"] as const;
type ExportFormat = (typeof VALID_FORMATS)[number];
/**
* Safe user representation PII stripped.
* Only _id and displayName are included. The persona is included because
* it is user-authored and non-sensitive.
*/
export interface ISafeUser {
_id: string;
displayName: string;
persona?: string;
}
/**
* Safe AI provider representation secrets and infrastructure stripped.
* baseUrl is excluded (internal network info). apiKey is excluded (secret).
*/
export interface ISafeAiProvider {
_id: string;
name: string;
apiType: string;
enabled: boolean;
}
/**
* Safe project representation only non-sensitive metadata.
* gitUrl is excluded (could contain auth tokens in practice).
* system is excluded (may contain infrastructure details).
*/
export interface ISafeProject {
_id: string;
name: string;
slug: string;
description?: string;
}
/**
* A single turn scrubbed for export. `prompts.system` is always excluded.
*/
export interface IExportTurn {
_id: string;
createdAt: string;
mode: string;
status: string;
prompts: {
user: string;
};
blocks: IChatTurnBlock[];
toolCalls: IChatToolCall[];
subagents: IChatSubagentProcess[];
stats: IChatTurnStats;
errorMessage?: string;
}
/**
* The full export payload delivered by the AgentService data provider.
*/
export interface IChatExportData {
session: IChatSession;
turns: IChatTurn[];
user: IUser;
provider: IAiProvider;
project: { name: string; slug: string; description?: string };
modelConfig: Omit<IDroneModelConfig, "provider">;
gadgetCodeVersion: string;
exportedAt: Date;
}
// ── Utility helpers ────────────────────────────────────────────────────
function sanitizeSessionName(name: string): string {
return name
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, 50) || "session";
}
function formatTimestamp(date: Date): string {
const d = date instanceof Date ? date : new Date(date);
const pad = (n: number) => String(n).padStart(2, "0");
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}`;
}
function toISO(date: Date | string): string {
return new Date(date).toISOString();
}
/** Strip PII, credentials, infrastructure info from a User record. */
function safeUser(user: IUser): ISafeUser {
return {
_id: String(user._id),
displayName: user.displayName,
persona: user.persona,
};
}
/** Strip secrets and infrastructure info from an AiProvider record. */
function safeProvider(provider: IAiProvider): ISafeAiProvider {
return {
_id: String(provider._id),
name: provider.name,
apiType: provider.apiType,
enabled: provider.enabled,
};
}
/** Strip sensitive info from a project record. */
function safeProject(project: { name: string; slug: string; description?: string; gitUrl?: string; system?: string }): ISafeProject {
return {
_id: String((project as any)._id),
name: project.name,
slug: project.slug,
description: project.description,
};
}
/** Strip `prompts.system` from a turn and normalize dates. */
function scrubTurn(turn: IChatTurn): IExportTurn {
return {
_id: String(turn._id),
createdAt: toISO(turn.createdAt),
mode: turn.mode,
status: turn.status,
prompts: {
user: turn.prompts.user,
},
blocks: turn.blocks || [],
toolCalls: (turn.toolCalls || []).map(scrubToolCall),
subagents: (turn.subagents || []).map(scrubSubagent),
stats: turn.stats,
errorMessage: turn.errorMessage,
};
}
/** Scrub a tool call for export no changes needed to the core structure,
* but we strip any subagent-embedded provider/user references. */
function scrubToolCall(tc: IChatToolCall): IChatToolCall {
const result: IChatToolCall = {
callId: tc.callId,
name: tc.name,
parameters: tc.parameters,
response: tc.response,
};
if (tc.subagent) {
result.subagent = scrubSubagent(tc.subagent);
}
return result;
}
/** Scrub a subagent process for export. */
function scrubSubagent(sa: IChatSubagentProcess): IChatSubagentProcess {
return {
prompt: sa.prompt,
thinking: sa.thinking,
response: sa.response,
toolCalls: (sa.toolCalls || []).map(scrubToolCall),
stats: sa.stats,
};
}
// ── Markdown formatter ─────────────────────────────────────────────────
function formatAsMarkdown(data: IChatExportData): string {
const { session, turns, user, provider, project, modelConfig, gadgetCodeVersion } = data;
const safeUser_ = safeUser(user);
const safeProvider_ = safeProvider(provider);
const finishedTurns = turns.filter(
(t) => t.status !== ChatTurnStatus.Processing,
);
const totalInputTokens = finishedTurns.reduce((sum, t) => sum + (t.stats?.inputTokens || 0), 0);
const totalOutputTokens = finishedTurns.reduce((sum, t) => sum + (t.stats?.responseTokens || 0), 0);
const totalThinkingTokens = finishedTurns.reduce((sum, t) => sum + (t.stats?.thinkingTokenCount || 0), 0);
const lines: string[] = [];
// ── Header ──
lines.push(`# ${session.name}`);
lines.push("");
lines.push(`**Session ID:** ${session._id}`);
lines.push(`**Created:** ${toISO(session.createdAt)}`);
if (session.lastMessageAt) {
lines.push(`**Last Message:** ${toISO(session.lastMessageAt)}`);
}
lines.push(`**Mode:** ${session.mode}`);
lines.push(`**Project:** ${project.name} (${project.slug})`);
lines.push(`**AI Provider:** ${safeProvider_.name} (${safeProvider_.apiType})`);
lines.push(`**Model:** ${session.selectedModel}`);
lines.push(`**Reasoning Effort:** ${session.reasoningEffort || "off"}`);
lines.push(`**Context Window:** ${session.numCtx || modelConfig.params.numCtx}`);
lines.push(`**Exported by:** Gadget Code v${gadgetCodeVersion}`);
lines.push(`**Exported at:** ${data.exportedAt.toISOString()}`);
lines.push("");
lines.push(`**Turns:** ${finishedTurns.length}`);
lines.push(`**Tool Calls:** ${session.stats?.toolCallCount || 0}`);
lines.push(`**Total Input Tokens:** ${totalInputTokens.toLocaleString()}`);
lines.push(`**Total Output Tokens:** ${totalOutputTokens.toLocaleString()}`);
if (totalThinkingTokens > 0) {
lines.push(`**Total Thinking Tokens:** ${totalThinkingTokens.toLocaleString()}`);
}
lines.push("");
// ── Pins ──
if (session.pins && session.pins.length > 0) {
lines.push("## Pinned Context");
lines.push("");
for (const pin of session.pins) {
lines.push(`- ${pin.content}`);
}
lines.push("");
}
lines.push("---");
lines.push("");
// ── Turns ──
for (const [i, turn] of finishedTurns.entries()) {
const scrubbed = scrubTurn(turn);
lines.push(`## Turn ${i + 1}`);
lines.push("");
lines.push(`**Timestamp:** ${scrubbed.createdAt}`);
lines.push(`**Mode:** ${scrubbed.mode}`);
lines.push(`**Status:** ${scrubbed.status}`);
if (scrubbed.stats?.durationLabel) {
lines.push(`**Duration:** ${scrubbed.stats.durationLabel}`);
}
lines.push("");
// User prompt
lines.push("### User");
lines.push("");
lines.push(scrubbed.prompts.user);
lines.push("");
// Thinking blocks
const thinkingBlocks = scrubbed.blocks.filter(
(b) => b.mode === "thinking",
) as IChatTurnBlockThinking[];
if (thinkingBlocks.length > 0) {
lines.push("### Thinking");
lines.push("");
for (const block of thinkingBlocks) {
lines.push(block.content);
lines.push("");
}
}
// Response blocks
const responseBlocks = scrubbed.blocks.filter(
(b) => b.mode === "responding",
) as IChatTurnBlockResponding[];
if (responseBlocks.length > 0) {
lines.push("### Assistant");
lines.push("");
for (const block of responseBlocks) {
lines.push(block.content);
lines.push("");
}
}
// Tool call blocks
const toolBlocks = scrubbed.blocks.filter(
(b) => b.mode === "tool",
) as IChatTurnBlockTool[];
if (toolBlocks.length > 0 || scrubbed.toolCalls.length > 0) {
lines.push("### Tool Calls");
lines.push("");
// From blocks
for (const block of toolBlocks) {
const tc = block.content;
lines.push(`#### ${tc.name}`);
lines.push("");
lines.push("**Call ID:** " + tc.callId);
lines.push("");
lines.push("**Parameters:**");
lines.push("```json");
try {
lines.push(JSON.stringify(JSON.parse(tc.parameters), null, 2));
} catch {
lines.push(tc.parameters);
}
lines.push("```");
lines.push("");
if (tc.response) {
lines.push("**Response:**");
lines.push("```");
// Truncate very long responses for readability
const resp = tc.response.length > 5000
? tc.response.slice(0, 5000) + "\n... (truncated for readability)"
: tc.response;
lines.push(resp);
lines.push("```");
lines.push("");
}
if (tc.subagent) {
lines.push(formatSubagentMarkdown(tc.subagent));
}
}
// From toolCalls array (if not already covered by blocks)
const blockCallIds = new Set(toolBlocks.map((b) => b.content.callId));
for (const tc of scrubbed.toolCalls) {
if (blockCallIds.has(tc.callId)) continue;
lines.push(`#### ${tc.name}`);
lines.push("");
lines.push("**Call ID:** " + tc.callId);
lines.push("");
lines.push("**Parameters:**");
lines.push("```json");
try {
lines.push(JSON.stringify(JSON.parse(tc.parameters), null, 2));
} catch {
lines.push(tc.parameters);
}
lines.push("```");
lines.push("");
if (tc.response) {
lines.push("**Response:**");
lines.push("```");
const resp = tc.response.length > 5000
? tc.response.slice(0, 5000) + "\n... (truncated for readability)"
: tc.response;
lines.push(resp);
lines.push("```");
lines.push("");
}
if (tc.subagent) {
lines.push(formatSubagentMarkdown(tc.subagent));
}
}
}
// Subagents
if (scrubbed.subagents.length > 0) {
lines.push("### Subagents");
lines.push("");
for (const sa of scrubbed.subagents) {
lines.push(formatSubagentMarkdown(sa));
}
}
// Error
if (scrubbed.errorMessage) {
lines.push("### Error");
lines.push("");
lines.push(`> ${scrubbed.errorMessage}`);
lines.push("");
}
// Turn stats
lines.push(`*Token usage: ${scrubbed.stats?.inputTokens || 0} input / ${scrubbed.stats?.responseTokens || 0} output${scrubbed.stats?.thinkingTokenCount ? ` / ${scrubbed.stats.thinkingTokenCount} thinking` : ""}*`);
lines.push("");
lines.push("---");
lines.push("");
}
return lines.join("\n");
}
function formatSubagentMarkdown(sa: IChatSubagentProcess): string {
const lines: string[] = [];
lines.push(`##### Subagent`);
lines.push("");
lines.push("**Prompt:** " + sa.prompt.slice(0, 200) + (sa.prompt.length > 200 ? "..." : ""));
lines.push("");
if (sa.thinking) {
lines.push("**Thinking:**");
lines.push("> " + sa.thinking.slice(0, 1000).replace(/\n/g, "\n> "));
lines.push("");
}
if (sa.response) {
lines.push("**Response:**");
lines.push(sa.response.slice(0, 2000) + (sa.response.length > 2000 ? "\n... (truncated)" : ""));
lines.push("");
}
if (sa.stats) {
lines.push(`*Subagent tokens: ${sa.stats.inputTokens || 0} input / ${sa.stats.responseTokens || 0} output / ${sa.stats.thinkingTokenCount || 0} thinking — ${sa.stats.durationLabel || "unknown"}*`);
lines.push("");
}
return lines.join("\n");
}
// ── JSON formatter ─────────────────────────────────────────────────────
function formatAsJson(data: IChatExportData): string {
const { session, turns, user, provider, project, modelConfig, gadgetCodeVersion } = data;
const finishedTurns = turns.filter(
(t) => t.status !== ChatTurnStatus.Processing,
);
const totalInputTokens = finishedTurns.reduce((sum, t) => sum + (t.stats?.inputTokens || 0), 0);
const totalOutputTokens = finishedTurns.reduce((sum, t) => sum + (t.stats?.responseTokens || 0), 0);
const exportObj = {
exportMetadata: {
format: "gadget-code-chat-export-v1",
version: "1.0.0",
exportedAt: data.exportedAt.toISOString(),
gadgetCodeVersion,
gadgetCodeUrl: "https://g4dge7.com/",
},
session: {
_id: String(session._id),
name: session.name,
mode: session.mode,
createdAt: toISO(session.createdAt),
lastMessageAt: session.lastMessageAt ? toISO(session.lastMessageAt) : undefined,
selectedModel: session.selectedModel,
reasoningEffort: session.reasoningEffort || "off",
numCtx: session.numCtx || modelConfig.params.numCtx,
stats: session.stats,
pins: session.pins || [],
},
user: safeUser(user),
provider: safeProvider(provider),
project: safeProject(project),
modelConfig: {
modelId: modelConfig.modelId,
params: modelConfig.params,
},
turns: finishedTurns.map(scrubTurn),
aggregateStats: {
turnCount: finishedTurns.length,
totalInputTokens,
totalOutputTokens,
},
};
return JSON.stringify(exportObj, null, 2);
}
// ── Companion Markdown (for JSON exports) ───────────────────────────────
function formatCompanionMarkdown(data: IChatExportData): string {
const { session, user, provider, modelConfig, gadgetCodeVersion } = data;
const safeUser_ = safeUser(user);
const safeProvider_ = safeProvider(provider);
const finishedTurns = (data.turns || []).filter(
(t) => t.status !== ChatTurnStatus.Processing,
);
const totalInputTokens = finishedTurns.reduce((sum, t) => sum + (t.stats?.inputTokens || 0), 0);
const totalOutputTokens = finishedTurns.reduce((sum, t) => sum + (t.stats?.responseTokens || 0), 0);
const lines: string[] = [];
lines.push("# Chat Export — Companion Document");
lines.push("");
lines.push("## Export Details");
lines.push("");
lines.push(`- **Format:** Gadget Code Chat Export v1 (JSON)`);
lines.push(`- **Exported:** ${data.exportedAt.toISOString()}`);
lines.push(`- **Gadget Code Version:** ${gadgetCodeVersion}`);
lines.push(`- **Get Gadget Code:** https://g4dge7.com/`);
lines.push("");
lines.push("## Authors");
lines.push("");
lines.push("### Human Author");
lines.push(`- **Name:** ${safeUser_.displayName}`);
lines.push("");
lines.push("### AI Agent");
lines.push(`- **Agent:** Gadget Code version ${gadgetCodeVersion}`);
lines.push(`- **Provider:** ${safeProvider_.name} (${safeProvider_.apiType})`);
lines.push(`- **Model:** ${modelConfig.modelId}`);
lines.push(`- **Parameters:**`);
lines.push(` - Temperature: ${modelConfig.params.temperature}`);
lines.push(` - Top P: ${modelConfig.params.topP}`);
lines.push(` - Top K: ${modelConfig.params.topK}`);
lines.push(` - Context Window: ${modelConfig.params.numCtx}`);
lines.push(` - Max Completion Tokens: ${modelConfig.params.maxCompletionTokens}`);
lines.push(` - Num Predict: ${modelConfig.params.numPredict}`);
lines.push(` - Reasoning: ${modelConfig.params.reasoning}`);
lines.push("");
lines.push("## Session Summary");
lines.push("");
lines.push(`- **Session:** ${session.name}`);
lines.push(`- **Session ID:** ${session._id}`);
lines.push(`- **Mode:** ${session.mode}`);
lines.push(`- **Turns:** ${finishedTurns.length}`);
lines.push(`- **Tool Calls:** ${session.stats?.toolCallCount || 0}`);
lines.push(`- **Total Input Tokens:** ${totalInputTokens.toLocaleString()}`);
lines.push(`- **Total Output Tokens:** ${totalOutputTokens.toLocaleString()}`);
lines.push("");
lines.push("## About This Export");
lines.push("");
lines.push("This Markdown document accompanies a JSON export of a Gadget Code chat session.");
lines.push("The JSON file contains the complete machine-readable conversation data, suitable");
lines.push("for import into other systems, analysis, or archival purposes.");
lines.push("");
lines.push("The JSON export format is designed to be self-contained and interoperable. It");
lines.push("includes all conversation turns, tool calls, subagent interactions, and session");
lines.push("metadata needed to reconstruct the full conversation in another system.");
lines.push("");
lines.push("For more information about Gadget Code, visit https://g4dge7.com/");
return lines.join("\n");
}
// ── Tool implementation ───────────────────────────────────────────────
export class ExportTool extends GadgetTool {
private _getSessionData: (() => Promise<IChatExportData>) | null = null;
setSessionDataProvider(fn: () => Promise<IChatExportData>): void {
this._getSessionData = fn;
}
get name(): string {
return "chat_export";
}
get category(): string {
return "chat";
}
get definition(): IToolDefinition {
return {
type: "function",
function: {
name: this.name,
description:
"Export the current chat session and all its conversation turns to disk as a document. " +
"The default format is Markdown, which produces a well-formatted conversation log including " +
"tool calls, responses, thinking blocks, and related metadata. " +
"The JSON format produces a complete machine-readable export suitable for import into " +
"other systems, analysis, or archival — it is always accompanied by a companion Markdown " +
"document that describes and credits the export. " +
"All exports exclude sensitive information (emails, API keys, hostnames, system prompts).",
parameters: {
type: "object",
properties: {
format: {
type: "string",
enum: [...VALID_FORMATS],
description:
"Export format: 'markdown' for a human-readable conversation log, " +
"'json' for a complete machine-readable export with a companion Markdown document. " +
"Defaults to 'markdown'.",
},
},
required: ["format"],
},
},
};
}
async execute(args: IToolArguments, logger: IAiLogger): Promise<string> {
const { format } = args;
// ── Validate format parameter ──
if (!format) {
return formatError({
code: "MISSING_PARAMETER",
message: "The 'format' parameter is required.",
parameter: "format",
expected: "Either 'markdown' or 'json'",
recoveryHint: "Specify 'markdown' for a readable log or 'json' for a machine-readable export.",
});
}
if (!VALID_FORMATS.includes(format as ExportFormat)) {
return formatError({
code: "INVALID_PARAMETER",
message: `Invalid format: '${format}'. Must be one of: ${VALID_FORMATS.join(", ")}`,
parameter: "format",
expected: `One of: ${VALID_FORMATS.join(", ")}`,
recoveryHint: "Use 'markdown' for a human-readable conversation log or 'json' for a complete data export.",
});
}
// ── Check session data provider ──
if (!this._getSessionData) {
return formatError({
code: "OPERATION_FAILED",
message: "ExportTool has not been initialized with a session data provider.",
recoveryHint: "This is an internal initialization error. The agent service must wire the data provider during startup.",
});
}
// ── Get export data ──
let data: IChatExportData;
try {
data = await this._getSessionData();
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
logger.error("failed to get export data", { error: message });
return formatError({
code: "OPERATION_FAILED",
message: `Failed to retrieve session data for export: ${message}`,
recoveryHint: "Ensure the chat session is active and try again.",
});
}
// ── Determine file paths ──
const projectDir = this.toolbox.env.workspace?.projectDir;
if (!projectDir) {
return formatError({
code: "OPERATION_FAILED",
message: "Workspace project directory is not available. The workspace must be initialized before exporting.",
recoveryHint: "Ensure the agent is running within an active workspace.",
});
}
const exportDir = path.join(projectDir, ".gadget", "exports");
const baseName = sanitizeSessionName(data.session.name);
const timestamp = formatTimestamp(data.exportedAt);
const exportFormat = format as ExportFormat;
logger.debug("exporting chat session", {
sessionId: String(data.session._id),
format: exportFormat,
turnCount: data.turns.length,
});
try {
// Ensure export directory exists
await fs.mkdir(exportDir, { recursive: true });
const results: string[] = [];
if (exportFormat === "markdown") {
// ── Markdown export ──
const content = formatAsMarkdown(data);
const fileName = `${baseName}-${timestamp}.md`;
const filePath = path.join(exportDir, fileName);
await fs.writeFile(filePath, content, "utf-8");
const byteCount = Buffer.byteLength(content, "utf-8");
results.push(`.gadget/exports/${fileName}`);
// Build success response
const finishedTurns = data.turns.filter(
(t) => t.status !== ChatTurnStatus.Processing,
);
const totalInput = finishedTurns.reduce((s, t) => s + (t.stats?.inputTokens || 0), 0);
const totalOutput = finishedTurns.reduce((s, t) => s + (t.stats?.responseTokens || 0), 0);
return [
"CHAT EXPORT SUCCESS",
`Format: markdown`,
`Session: ${data.session.name}`,
`Turns: ${finishedTurns.length}`,
`Total Input Tokens: ${totalInput.toLocaleString()}`,
`Total Output Tokens: ${totalOutput.toLocaleString()}`,
`Bytes: ${byteCount.toLocaleString()}`,
`Files:`,
...results.map((r) => ` - ${r}`),
].join("\n");
}
// ── JSON export (always with companion markdown) ──
const jsonContent = formatAsJson(data);
const jsonFileName = `${baseName}-${timestamp}.json`;
const jsonFilePath = path.join(exportDir, jsonFileName);
await fs.writeFile(jsonFilePath, jsonContent, "utf-8");
const jsonBytes = Buffer.byteLength(jsonContent, "utf-8");
results.push(`.gadget/exports/${jsonFileName}`);
const companionContent = formatCompanionMarkdown(data);
const companionFileName = `${baseName}-${timestamp}-readme.md`;
const companionFilePath = path.join(exportDir, companionFileName);
await fs.writeFile(companionFilePath, companionContent, "utf-8");
const companionBytes = Buffer.byteLength(companionContent, "utf-8");
results.push(`.gadget/exports/${companionFileName}`);
// Build success response
const finishedTurns = data.turns.filter(
(t) => t.status !== ChatTurnStatus.Processing,
);
const totalInput = finishedTurns.reduce((s, t) => s + (t.stats?.inputTokens || 0), 0);
const totalOutput = finishedTurns.reduce((s, t) => s + (t.stats?.responseTokens || 0), 0);
return [
"CHAT EXPORT SUCCESS",
`Format: json + companion markdown`,
`Session: ${data.session.name}`,
`Turns: ${finishedTurns.length}`,
`Total Input Tokens: ${totalInput.toLocaleString()}`,
`Total Output Tokens: ${totalOutput.toLocaleString()}`,
`JSON Bytes: ${jsonBytes.toLocaleString()}`,
`Readme Bytes: ${companionBytes.toLocaleString()}`,
`Files:`,
...results.map((r) => ` - ${r}`),
].join("\n");
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
logger.error("failed to write export file", { error: message });
return formatError({
code: "OPERATION_FAILED",
message: `Failed to write export file: ${message}`,
recoveryHint: "Check filesystem permissions and available disk space, then try again.",
});
}
}
}

View File

@ -3,3 +3,4 @@
// Licensed under the Apache License, Version 2.0
export { SubagentTool } from "./subagent.ts";
export { ExportTool, type IChatExportData, type ISafeUser, type ISafeAiProvider, type ISafeProject, type IExportTurn } from "./export.ts";

View File

@ -170,6 +170,11 @@ export interface IAiModelProbeResult {
};
}
export interface IAiEmbeddingResponse {
embedding: number[];
model: string;
}
export abstract class AiApi {
protected env: IAiEnvironment;
protected provider: IAiProvider;
@ -196,6 +201,11 @@ export abstract class AiApi {
streamCallback?: IAiResponseStreamFn,
): Promise<IAiChatResponse>;
/**
* Generate an embedding vector for the given text.
*/
abstract embeddings(modelId: string, text: string, dimensions?: number): Promise<IAiEmbeddingResponse>;
/**
* Forcefully abort any in-progress API request.
* Provider-specific implementations (e.g., Ollama) should terminate

View File

@ -21,6 +21,7 @@ export {
AiApi,
type IAiModelListResult,
type IAiModelProbeResult,
type IAiEmbeddingResponse,
} from "./api.js";
export type {

View File

@ -19,6 +19,7 @@ import {
IAiModelProbeResult,
IAiProvider,
IAiResponseStreamFn,
IAiEmbeddingResponse,
} from "./api.js";
import { IAiEnvironment } from "./config/env.ts";
import type { Message as OllamaMessage } from "ollama";
@ -367,4 +368,9 @@ export class OllamaAiApi extends AiApi {
this.assertNonEmptyChatResponse(chatResponse);
return chatResponse;
}
async embeddings(modelId: string, text: string, dimensions?: number): Promise<IAiEmbeddingResponse> {
const response = await this.client.embed({ model: modelId, input: text, dimensions });
return { embedding: response.embeddings[0]!, model: modelId };
}
}

View File

@ -17,6 +17,7 @@ import {
IAiModelProbeResult,
IAiProvider,
IAiResponseStreamFn,
IAiEmbeddingResponse,
} from "./api.js";
import {
ChatCompletionFunctionTool,
@ -64,6 +65,12 @@ interface StreamingToolCallAccumulator {
};
}
interface OpenAiUsage {
promptTokens: number;
completionTokens: number;
reasoningTokens: number;
}
interface OpenAiChatIterationResult {
response: string;
thinking?: string;
@ -80,6 +87,7 @@ interface OpenAiChatIterationResult {
chunkCount: number;
contentDeltaCount: number;
toolDeltaCount: number;
usage?: OpenAiUsage;
}
export class OpenAiApi extends AiApi {
@ -187,17 +195,13 @@ export class OpenAiApi extends AiApi {
options: IAiGenerateOptions,
streamCallback?: IAiResponseStreamFn,
): Promise<IAiGenerateResponse> {
await this.log.debug("OpenAiApi.generate called", {
provider: model.provider.name,
modelId: model.modelId,
});
if (options.signal?.aborted) {
throw new DOMException("The operation was aborted", "AbortError");
}
const startTime = Date.now();
const response = await this.client.chat.completions.create({
const response = await this.client.chat.completions.create(
{
model: model.modelId,
messages: [
...(options.systemPrompt
@ -206,6 +210,7 @@ export class OpenAiApi extends AiApi {
{ role: "user" as const, content: options.prompt },
],
stream: true,
stream_options: { include_usage: true },
...(model.params.maxCompletionTokens
? { max_completion_tokens: model.params.maxCompletionTokens }
: {}),
@ -219,16 +224,24 @@ export class OpenAiApi extends AiApi {
| "high",
}
: {}),
}, options.signal ? { signal: options.signal } : undefined);
},
options.signal ? { signal: options.signal } : undefined,
);
let accumulatedResponse = "";
let accumulatedThinking = "";
let usage: OpenAiUsage | undefined;
for await (const chunk of response) {
if (options.signal?.aborted) {
throw new DOMException("The operation was aborted", "AbortError");
}
// Capture usage from the final streaming chunk
if (chunk.usage) {
usage = this.extractUsage(chunk.usage);
}
const delta = chunk.choices[0]?.delta;
if (delta) {
if (delta.content) {
@ -265,9 +278,9 @@ export class OpenAiApi extends AiApi {
text: numeral(durationMs / 1000).format("hh:mm:ss"),
},
tokenCounts: {
input: 0,
response: 0,
thinking: 0,
input: usage?.promptTokens ?? 0,
response: usage?.completionTokens ?? 0,
thinking: usage?.reasoningTokens ?? 0,
},
},
};
@ -278,11 +291,6 @@ export class OpenAiApi extends AiApi {
options: IAiChatOptions,
streamCallback?: IAiResponseStreamFn,
): Promise<IAiChatResponse> {
await this.log.debug("OpenAiApi.chat called", {
provider: model.provider.name,
modelId: model.modelId,
});
const startTime = Date.now();
const messages = this.buildMessages(options);
const tools = this.buildTools(options);
@ -295,27 +303,16 @@ export class OpenAiApi extends AiApi {
options.signal,
);
await this.log.debug("OpenAI chat stream iteration finished", {
chunkCount: iteration.chunkCount,
contentDeltaCount: iteration.contentDeltaCount,
toolDeltaCount: iteration.toolDeltaCount,
responseLength: iteration.response.length,
thinkingLength: iteration.thinking?.length || 0,
toolCallCount: iteration.toolCalls.length,
finishReason: iteration.finishReason,
});
if (this.isEmptyIteration(iteration)) {
iteration = await this.readNonStreamingChatCompletion(model, messages, tools, options.signal);
iteration = await this.readNonStreamingChatCompletion(
model,
messages,
tools,
options.signal,
);
if (streamCallback && iteration.response) {
await streamCallback({ type: "response", data: iteration.response });
}
await this.log.warn("OpenAI stream was empty; used non-streaming fallback", {
responseLength: iteration.response.length,
thinkingLength: iteration.thinking?.length || 0,
toolCallCount: iteration.toolCalls.length,
finishReason: iteration.finishReason,
});
}
if (this.isEmptyIteration(iteration)) {
@ -325,7 +322,7 @@ export class OpenAiApi extends AiApi {
thinking: undefined,
toolCalls: undefined,
toolCallResults: undefined,
stats: this.buildStats(startTime),
stats: this.buildStats(startTime, iteration.usage),
});
}
@ -333,20 +330,32 @@ export class OpenAiApi extends AiApi {
done: true,
response: iteration.response,
thinking: iteration.thinking,
toolCalls: iteration.toolCalls.length > 0 ? iteration.toolCalls : undefined,
stats: this.buildStats(startTime),
toolCalls:
iteration.toolCalls.length > 0 ? iteration.toolCalls : undefined,
stats: this.buildStats(startTime, iteration.usage),
};
this.assertNonEmptyChatResponse(finalResponse);
return finalResponse;
}
private extractUsage(usage: {
prompt_tokens: number;
completion_tokens: number;
completion_tokens_details?: { reasoning_tokens?: number };
}): OpenAiUsage {
return {
promptTokens: usage.prompt_tokens,
completionTokens: usage.completion_tokens,
reasoningTokens: usage.completion_tokens_details?.reasoning_tokens ?? 0,
};
}
private buildMessages(options: IAiChatOptions): ChatCompletionMessageParam[] {
const messages: ChatCompletionMessageParam[] = [];
if (options.systemPrompt) {
messages.push({ role: "system", content: options.systemPrompt });
}
for (const msg of options.context || []) {
if (!msg.content?.trim()) continue;
if (msg.role === "tool") {
messages.push({
role: "tool",
@ -355,7 +364,11 @@ export class OpenAiApi extends AiApi {
});
continue;
}
if (msg.role === "assistant" && msg.toolCalls && msg.toolCalls.length > 0) {
if (
msg.role === "assistant" &&
msg.toolCalls &&
msg.toolCalls.length > 0
) {
messages.push({
role: "assistant",
content: msg.content,
@ -375,14 +388,16 @@ export class OpenAiApi extends AiApi {
}
private buildTools(options: IAiChatOptions): ChatCompletionTool[] {
return (options.tools || []).map((tool): ChatCompletionFunctionTool => ({
return (options.tools || []).map(
(tool): ChatCompletionFunctionTool => ({
type: tool.definition.type,
function: {
name: tool.definition.function.name,
description: tool.definition.function.description,
parameters: tool.definition.function.parameters,
},
}));
}),
);
}
private async readStreamingChatCompletion(
@ -396,11 +411,13 @@ export class OpenAiApi extends AiApi {
throw new DOMException("The operation was aborted", "AbortError");
}
const response = await this.client.chat.completions.create({
const response = await this.client.chat.completions.create(
{
model: model.modelId,
messages,
tools,
stream: true,
stream_options: { include_usage: true },
...(model.params.maxCompletionTokens
? { max_completion_tokens: model.params.maxCompletionTokens }
: {}),
@ -414,7 +431,9 @@ export class OpenAiApi extends AiApi {
| "high",
}
: {}),
}, signal ? { signal } : undefined);
},
signal ? { signal } : undefined,
);
let content = "";
let thinking = "";
@ -423,6 +442,7 @@ export class OpenAiApi extends AiApi {
let toolDeltaCount = 0;
let finishReason: string | null | undefined;
const toolCallMap = new Map<number, StreamingToolCallAccumulator>();
let usage: OpenAiUsage | undefined;
for await (const chunk of response) {
if (signal?.aborted) {
@ -431,6 +451,12 @@ export class OpenAiApi extends AiApi {
chunkCount++;
finishReason = chunk.choices[0]?.finish_reason ?? finishReason;
// Capture usage from the final streaming chunk
if (chunk.usage) {
usage = this.extractUsage(chunk.usage);
}
const delta = chunk.choices[0]?.delta;
if (!delta) continue;
@ -444,7 +470,10 @@ export class OpenAiApi extends AiApi {
if ("reasoning" in delta && delta.reasoning) {
thinking += delta.reasoning as string;
if (streamCallback) {
await streamCallback({ type: "thinking", data: delta.reasoning as string });
await streamCallback({
type: "thinking",
data: delta.reasoning as string,
});
}
}
if (delta.tool_calls) {
@ -463,6 +492,7 @@ export class OpenAiApi extends AiApi {
chunkCount,
contentDeltaCount,
toolDeltaCount,
usage,
};
}
@ -476,7 +506,8 @@ export class OpenAiApi extends AiApi {
throw new DOMException("The operation was aborted", "AbortError");
}
const response = await this.client.chat.completions.create({
const response = await this.client.chat.completions.create(
{
model: model.modelId,
messages,
tools,
@ -494,10 +525,16 @@ export class OpenAiApi extends AiApi {
| "high",
}
: {}),
}, signal ? { signal } : undefined);
},
signal ? { signal } : undefined,
);
const choice = response.choices[0];
const message = choice?.message;
const content = typeof message?.content === "string" ? message.content : "";
const usage = response.usage
? this.extractUsage(response.usage)
: undefined;
const assistantToolCalls = (message?.tool_calls || [])
.filter((tc) => tc.type === "function")
.map((tc) => ({
@ -524,6 +561,7 @@ export class OpenAiApi extends AiApi {
chunkCount: 0,
contentDeltaCount: content ? 1 : 0,
toolDeltaCount: assistantToolCalls.length,
usage,
};
}
@ -592,7 +630,10 @@ export class OpenAiApi extends AiApi {
);
}
private buildStats(startTime: number): IAiChatResponse["stats"] {
private buildStats(
startTime: number,
usage?: OpenAiUsage,
): IAiChatResponse["stats"] {
const seconds = (Date.now() - startTime) / 1000;
return {
duration: {
@ -600,10 +641,19 @@ export class OpenAiApi extends AiApi {
text: numeral(seconds).format("hh:mm:ss"),
},
tokenCounts: {
input: 0,
response: 0,
thinking: 0,
input: usage?.promptTokens ?? 0,
response: usage?.completionTokens ?? 0,
thinking: usage?.reasoningTokens ?? 0,
},
};
}
async embeddings(modelId: string, text: string, dimensions?: number): Promise<IAiEmbeddingResponse> {
const response = await this.client.embeddings.create({
model: modelId,
input: text,
dimensions,
});
return { embedding: response.data[0]!.embedding, model: modelId };
}
}

View File

@ -36,6 +36,7 @@ export interface IChatSession {
selectedModel: string;
reasoningEffort?: ReasoningEffort;
numCtx?: number;
contextWindowUsage: number;
stats: {
turnCount: number;
toolCallCount: number;

View File

@ -41,10 +41,23 @@ export type ToolCallMessage = (
response: string,
) => void;
export interface IWorkOrderCompleteStats {
masterInputTokens: number;
masterOutputTokens: number;
masterThinkingTokens: number;
toolCallCount: number;
durationMs: number;
/** Aggregate (master + subagent) totals for session-level accounting */
totalInputTokens?: number;
totalOutputTokens?: number;
totalToolCallCount?: number;
}
export type WorkOrderCompleteMessage = (
workOrderId: string,
success: boolean,
message?: string,
stats?: IWorkOrderCompleteStats,
) => void;
export type RequestCrashRecoveryMessage = (data: {

View File

@ -40,6 +40,15 @@ export interface GadgetCodeConfig {
host: string;
database: string;
};
qdrant: {
host: string;
port?: number; // default: 6333
apiKey?: string; // optional API key for secured Qdrant
collection: string;
providerId: string; // AiProvider._id (GadgetId type not available here)
embeddingModel: string;
vectorSize: number;
};
redis: {
host?: string;
port?: number;

View File

@ -25,6 +25,12 @@ importers:
'@gadget/config':
specifier: workspace:*
version: link:../packages/config
'@langchain/textsplitters':
specifier: ^1.0.1
version: 1.0.1(@langchain/core@1.1.47(openai@6.34.0(ws@8.18.3)(zod@4.4.3))(ws@8.18.3))
'@qdrant/js-client-rest':
specifier: ^1.12.0
version: 1.18.0(typescript@5.9.3)
'@react-three/drei':
specifier: ^10.7.7
version: 10.7.7(@react-three/fiber@9.6.1(@types/react@19.2.14)(react-dom@19.2.5(react@19.2.5))(react@19.2.5)(three@0.184.0))(@types/react@19.2.14)(@types/three@0.184.0)(react-dom@19.2.5(react@19.2.5))(react@19.2.5)(three@0.184.0)
@ -407,7 +413,7 @@ importers:
version: 0.6.3
openai:
specifier: ^6.34.0
version: 6.34.0(ws@8.18.3)
version: 6.34.0(ws@8.18.3)(zod@4.4.3)
playwright:
specifier: 1.59.1
version: 1.59.1
@ -496,7 +502,7 @@ importers:
version: 0.6.3
openai:
specifier: ^6.34.0
version: 6.34.0(ws@8.18.3)
version: 6.34.0(ws@8.18.3)(zod@4.4.3)
devDependencies:
'@types/node':
specifier: ^25.6.0
@ -642,6 +648,9 @@ packages:
resolution: {integrity: sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw==}
hasBin: true
'@cfworker/json-schema@4.1.1':
resolution: {integrity: sha512-gAmrUZSGtKc3AiBL71iNWxDsyUC5uMaKKGdvzYsBoTW/xi42JQHl7eKV2OYzCUqvc+D2RCcf7EXY2iCyFIk6og==}
'@codemirror/autocomplete@6.20.2':
resolution: {integrity: sha512-G5FPkgIiLjOgZMjqVjvuKQ1rGPtHogLldJr33eFJdVLtmwY+giGrlv/ewljLz6b9BSQLkjxuwBc6g6omDM+YxQ==}
@ -1246,6 +1255,16 @@ packages:
'@kwsites/promise-deferred@1.1.1':
resolution: {integrity: sha512-GaHYm+c0O9MjZRu0ongGBRbinu8gVAMd2UZjji6jVmqKtZluZnptXGWhz1E8j8D2HJ3f/yMxKAUC0b+57wncIw==}
'@langchain/core@1.1.47':
resolution: {integrity: sha512-+fiPu6ZFnJMrZyKeM77OIVPoMPAY6OKWacnPlojHtXTbMMzb2cEOKAJV0U07cDl86NHSCIYYa0i4CyKZzXbHQQ==}
engines: {node: '>=20'}
'@langchain/textsplitters@1.0.1':
resolution: {integrity: sha512-rheJlB01iVtrOUzttscutRgLybPH9qR79EyzBEbf1u97ljWyuxQfCwIWK+SjoQTM9O8M7GGLLRBSYE26Jmcoww==}
engines: {node: '>=20'}
peerDependencies:
'@langchain/core': ^1.0.0
'@lezer/common@1.5.2':
resolution: {integrity: sha512-sxQE460fPZyU3sdc8lafxiPwJHBzZRy/udNFynGQky1SePYBdhkBl1kOagA9uT3pxR8K09bOrmTUqA9wb/PjSQ==}
@ -1374,6 +1393,16 @@ packages:
engines: {node: '>=18'}
hasBin: true
'@qdrant/js-client-rest@1.18.0':
resolution: {integrity: sha512-/0dqX5uV9chC1DnYSnU4gNMrDqse/pt6hHg3Rqqpl5isH7xl1xSNvffjzBoxycDD79luWn7Ho6Rh/61sOs5DNw==}
engines: {node: '>=18.17.0', pnpm: '>=8'}
peerDependencies:
typescript: '>=4.7'
'@qdrant/openapi-typescript-fetch@1.2.6':
resolution: {integrity: sha512-oQG/FejNpItrxRHoyctYvT3rwGZOnK4jr3JdppO/c78ktDvkWiPXPHNsrDf33K9sZdRb6PR7gi4noIapu5q4HA==}
engines: {node: '>=18.0.0', pnpm: '>=8'}
'@react-three/drei@10.7.7':
resolution: {integrity: sha512-ff+J5iloR0k4tC++QtD/j9u3w5fzfgFAWDtAGQah9pF2B1YgOq/5JxqY0/aVoQG5r3xSZz0cv5tk2YuBob4xEQ==}
peerDependencies:
@ -2859,6 +2888,9 @@ packages:
js-stringify@1.0.2:
resolution: {integrity: sha512-rtS5ATOo2Q5k1G+DADISilDA6lv79zIiwFd6CcjuIxGKLFm5C+RLImRscVap9k55i+MOZwgliw+NejvkLuGD5g==}
js-tiktoken@1.0.21:
resolution: {integrity: sha512-biOj/6M5qdgx5TKjDnFT1ymSpM5tbd3ylwDtrQvFQSu0Z7bBYko2dF+W/aUkXUPuk6IVpRxk/3Q2sHOzGlS36g==}
js-tokens@4.0.0:
resolution: {integrity: sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==}
@ -2910,6 +2942,26 @@ packages:
resolution: {integrity: sha512-C3iHfuGUXK2u8/ipq9LfjFfXFxAZMQJJq7vLS45r3D9Y2xQ/m4S8zaR4zMLFWh9AsNPXmcFfUDhTEO8UIC/V6Q==}
engines: {node: '>=12.0.0'}
langsmith@0.7.1:
resolution: {integrity: sha512-Wjk90UjNoY5cBHMlNAC/eZx5clI8jnjBOBW8uJu8+MWBtx0QesNjsUiLtjI+I3UnrpxFFpDqGXcnhBjH654Mqg==}
peerDependencies:
'@opentelemetry/api': '*'
'@opentelemetry/exporter-trace-otlp-proto': '*'
'@opentelemetry/sdk-trace-base': '*'
openai: '*'
ws: '>=7'
peerDependenciesMeta:
'@opentelemetry/api':
optional: true
'@opentelemetry/exporter-trace-otlp-proto':
optional: true
'@opentelemetry/sdk-trace-base':
optional: true
openai:
optional: true
ws:
optional: true
lazy@1.0.11:
resolution: {integrity: sha512-Y+CjUfLmIpoUCCRl0ub4smrYtGGr5AOa2AKOaWelGHOGz33X/Y/KizefGqbkwfz44+cnq/+9habclf8vOmu2LA==}
engines: {node: '>=0.2.0'}
@ -3215,6 +3267,10 @@ packages:
resolution: {integrity: sha512-mo+QTzKlx8R7E5ylSXxWzGoXoZbOsRMpyitcht8By2KHvMbf3tjwosZ/Mu/XYU6UuJ3VZnODIrak5ZrPiPyB6A==}
engines: {node: '>= 10.16.0'}
mustache@4.2.0:
resolution: {integrity: sha512-71ippSywq5Yb7/tVYyGbkBggbU8H3u5Rz56fH60jGFgr8uHwxs+aSKeqmluIVzM0m0kB7xQjKS6qPfd0b2ZoqQ==}
hasBin: true
mute-stream@3.0.0:
resolution: {integrity: sha512-dkEJPVvun4FryqBmZ5KhDo0K9iDXAwn08tMLDinNdRBNPcYEDiWYysLcc6k3mjTMlbP9KyylvRpd4wFtwrT9rw==}
engines: {node: ^20.17.0 || >=22.9.0}
@ -3334,6 +3390,18 @@ packages:
resolution: {integrity: sha512-PRT7ZORmwu2MEFt4/fv3Q+mEfN4zetKxufQrkShY2oGvUms9r8otu5HfdyIFHkYXjO7laNsoVGmM2MANfuTA8g==}
engines: {node: '>=0.10.0'}
p-finally@1.0.0:
resolution: {integrity: sha512-LICb2p9CB7FS+0eR1oqWnHhp0FljGLZCWBE9aix0Uye9W8LTQPwMTYVGWQWIw9RdQiDg4+epXQODwIYJtSJaow==}
engines: {node: '>=4'}
p-queue@6.6.2:
resolution: {integrity: sha512-RwFpb72c/BhQLEXIZ5K2e+AhgNVmIejGlTgiB9MzZ0e93GRvqZ7uSi0dvRF7/XIXDeNkra2fNHBxTyPDGySpjQ==}
engines: {node: '>=8'}
p-timeout@3.2.0:
resolution: {integrity: sha512-rhIwUycgwwKcP9yTOOFK/AKsAopjjCakVqLHePO3CC6Mir1Z99xT+R63jZxAT5lFZLa2inS5h+ZS2GvR99/FBg==}
engines: {node: '>=8'}
parse-node-version@1.0.1:
resolution: {integrity: sha512-3YHlOa/JgH6Mnpr05jP9eDG254US9ek25LyIxZlDItp2iJtwyaXQb57lBYLdT3MowkUFYEV2XXNAYIPlESvJlA==}
engines: {node: '>= 0.10'}
@ -4018,6 +4086,10 @@ packages:
undici-types@7.19.2:
resolution: {integrity: sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg==}
undici@6.25.0:
resolution: {integrity: sha512-ZgpWDC5gmNiuY9CnLVXEH8rl50xhRCuLNA97fAUnKi8RRuV4E6KG31pDTsLVUKnohJE0I3XDrTeEydAXRw47xg==}
engines: {node: '>=18.17'}
undici@7.25.0:
resolution: {integrity: sha512-xXnp4kTyor2Zq+J1FfPI6Eq3ew5h6Vl0F/8d9XU5zZQf1tX9s2Su1/3PiMmUANFULpmksxkClamIZcaUqryHsQ==}
engines: {node: '>=20.18.1'}
@ -4281,6 +4353,9 @@ packages:
yauzl@2.10.0:
resolution: {integrity: sha512-p4a9I6X6nu6IhoGmBqAcbJy1mlC4j27vEPZX9F4L4/vZT3Lyq1VkFHw/V/PUcB9Buo+DG3iHkT0x3Qya58zc3g==}
zod@4.4.3:
resolution: {integrity: sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==}
zustand@4.5.7:
resolution: {integrity: sha512-CHOUy7mu3lbD6o6LJLfllpjkzhHXSBlX8B9+qPddUsIfeF5S/UZ5q0kmCsnRqT1UHFQZchNFDDzMbQsuesHWlw==}
engines: {node: '>=12.7.0'}
@ -4365,6 +4440,8 @@ snapshots:
dependencies:
css-tree: 3.2.1
'@cfworker/json-schema@4.1.1': {}
'@codemirror/autocomplete@6.20.2':
dependencies:
'@codemirror/language': 6.12.3
@ -4890,6 +4967,27 @@ snapshots:
'@kwsites/promise-deferred@1.1.1': {}
'@langchain/core@1.1.47(openai@6.34.0(ws@8.18.3)(zod@4.4.3))(ws@8.18.3)':
dependencies:
'@cfworker/json-schema': 4.1.1
'@standard-schema/spec': 1.1.0
js-tiktoken: 1.0.21
langsmith: 0.7.1(openai@6.34.0(ws@8.18.3)(zod@4.4.3))(ws@8.18.3)
mustache: 4.2.0
p-queue: 6.6.2
zod: 4.4.3
transitivePeerDependencies:
- '@opentelemetry/api'
- '@opentelemetry/exporter-trace-otlp-proto'
- '@opentelemetry/sdk-trace-base'
- openai
- ws
'@langchain/textsplitters@1.0.1(@langchain/core@1.1.47(openai@6.34.0(ws@8.18.3)(zod@4.4.3))(ws@8.18.3))':
dependencies:
'@langchain/core': 1.1.47(openai@6.34.0(ws@8.18.3)(zod@4.4.3))(ws@8.18.3)
js-tiktoken: 1.0.21
'@lezer/common@1.5.2': {}
'@lezer/cpp@1.1.5':
@ -5039,6 +5137,14 @@ snapshots:
dependencies:
playwright: 1.59.1
'@qdrant/js-client-rest@1.18.0(typescript@5.9.3)':
dependencies:
'@qdrant/openapi-typescript-fetch': 1.2.6
typescript: 5.9.3
undici: 6.25.0
'@qdrant/openapi-typescript-fetch@1.2.6': {}
'@react-three/drei@10.7.7(@react-three/fiber@9.6.1(@types/react@19.2.14)(react-dom@19.2.5(react@19.2.5))(react@19.2.5)(three@0.184.0))(@types/react@19.2.14)(@types/three@0.184.0)(react-dom@19.2.5(react@19.2.5))(react@19.2.5)(three@0.184.0)':
dependencies:
'@babel/runtime': 7.29.2
@ -6639,6 +6745,10 @@ snapshots:
js-stringify@1.0.2: {}
js-tiktoken@1.0.21:
dependencies:
base64-js: 1.5.1
js-tokens@4.0.0: {}
js-yaml@4.1.1:
@ -6738,6 +6848,13 @@ snapshots:
kareem@2.6.3: {}
langsmith@0.7.1(openai@6.34.0(ws@8.18.3)(zod@4.4.3))(ws@8.18.3):
dependencies:
p-queue: 6.6.2
optionalDependencies:
openai: 6.34.0(ws@8.18.3)(zod@4.4.3)
ws: 8.18.3
lazy@1.0.11: {}
lcid@1.0.0:
@ -7010,6 +7127,8 @@ snapshots:
concat-stream: 2.0.0
type-is: 1.6.18
mustache@4.2.0: {}
mute-stream@3.0.0: {}
mylas@2.1.14: {}
@ -7084,9 +7203,10 @@ snapshots:
dependencies:
wrappy: 1.0.2
openai@6.34.0(ws@8.18.3):
openai@6.34.0(ws@8.18.3)(zod@4.4.3):
optionalDependencies:
ws: 8.18.3
zod: 4.4.3
opn@5.3.0:
dependencies:
@ -7096,6 +7216,17 @@ snapshots:
dependencies:
lcid: 1.0.0
p-finally@1.0.0: {}
p-queue@6.6.2:
dependencies:
eventemitter3: 4.0.7
p-timeout: 3.2.0
p-timeout@3.2.0:
dependencies:
p-finally: 1.0.0
parse-node-version@1.0.1: {}
parse5@7.3.0:
@ -7845,6 +7976,8 @@ snapshots:
undici-types@7.19.2: {}
undici@6.25.0: {}
undici@7.25.0: {}
undici@8.1.0: {}
@ -8103,6 +8236,8 @@ snapshots:
buffer-crc32: 0.2.13
fd-slicer: 1.1.0
zod@4.4.3: {}
zustand@4.5.7(@types/react@19.2.14)(react@19.2.5):
dependencies:
use-sync-external-store: 1.6.0(react@19.2.5)

22
repairs.md Normal file
View File

@ -0,0 +1,22 @@
Summary of Changes (6 files, +36/-19 lines)
Root Cause
The token-economics commit (00969cf) correctly added subagent token aggregation to the DB write path (drone-session.ts), but the in-memory and frontend paths only used master-only values. This meant:
-
During a session: UI showed lower token counts than reality
-
After page refresh: numbers jumped up (DB values loaded)
-
Session toolCallCount missed subagent tool calls entirely
Fixes
1. packages/api/src/messages/drone.ts — Extended IWorkOrderCompleteStats with optional aggregate fields (totalInputTokens?, totalOutputTokens?, totalToolCallCount?)
2. gadget-code/src/lib/drone-session.ts — Now computes totalToolCallCount (master + subagent tool calls) alongside existing token aggregates, passes enrichedStats (with aggregate fields populated) to codeSession.onWorkOrderComplete()
3. gadget-code/src/lib/code-session.ts — In-memory session stats use stats.totalInputTokens ?? stats.masterInputTokens (aggregate when available, master fallback)
4. gadget-code/frontend/src/lib/api.ts — Frontend type matches the shared interface
5. gadget-code/frontend/src/pages/ChatSessionView.tsx — Frontend session stats update uses aggregate values; also now correctly increments toolCallCount (was previously missing)
6. gadget-drone/src/services/agent.ts — Three fixes:
-
Removed local IWorkOrderCompleteStats redefinition, now imports from @gadget/api
-
Master loop token accumulation uses optional chaining (response.stats?.tokenCounts?.input ?? 0) matching the subagent loop
-
agent:complete emit now includes stats field from subagent data