SubProcess prep work

gadget-code/data/prompts/agent/build/system.md

@@ -36,6 +36,8 @@ You must remain within the project directory, which is the current working direc
 
 **DO NOT** spawn a subagent as a workaround for lacking tool features, tool failures, and tool errors. Instead, please respond by writing out what you were trying to do, the parameters you used when calling the tool, and the full response you received. Don't work around tool errors. Report faulty tool performance and behavior, then stop. Let the User help determine what to do next.
 
+{{process_management_block}}
+
 ## INSTRUCTIONS
 
 You always provide regular updates to explain your thinking and reasoning while working and calling tools. You always end a turn by summarizing what you did to the User for their review and convenience. When the user sends you a prompt:

gadget-code/data/prompts/agent/dev/system.md

@@ -32,6 +32,8 @@ Use your tools proactively. When working on the Gadget Code codebase, immediatel
 
 NOTICE: IF YOU EXPERIENCE DIFFICULTY USING ANY TOOLS OR RECEIVE A RESPONSE THAT IS UNEXPECTED OR SEEMS ERRONEOUS (TOOL MALFUNCTION), PLEASE **IMMEDIATELY** DOCUMENT WHAT THE TOOL DID THAT YOU DIDN'T EXPECT - AND STOP. WHEN IN THE DEVELOP MODE, YOU ARE WORKING WITH A DEVELOPER (THE USER) DIRECTLY ON THIS AGENTIC HARNESS (GADGET CODE). WE MAY NEED TO DEBUG OR DIAGNOSE A PROBLEM WITH A TOOL AS WE WORK.
 
+{{process_management_block}}
+
 ## INSTRUCTIONS
 
 Work in a loop through the User's request for this turn, resolving the work items that need done until finished, explaining your thinking and reasoning while calling tools and doing your work.

gadget-code/data/prompts/agent/plan/system.md

@@ -40,6 +40,8 @@ Don't announce tool usage, just execute and use findings to inform your planning
 
 {{tool_block}}
 
+{{process_management_block}}
+
 ## INSTRUCTIONS
 
 When the user sends you a prompt:

gadget-code/data/prompts/agent/ship/system.md

@@ -22,6 +22,8 @@ Use your tools decisively. Run tests, check builds, verify deployments, and exec
 
 {{tool_block}}
 
+{{process_management_block}}
+
 ## INSTRUCTIONS
 
 When the user sends you a prompt:

gadget-code/data/prompts/agent/test/system.md

@@ -22,6 +22,8 @@ Use your tools aggressively for testing. Run tests frequently, read test files,
 
 {{tool_block}}
 
+{{process_management_block}}
+
 ## INSTRUCTIONS
 
 When the user sends you a prompt:

gadget-code/data/prompts/common/process-management-block.md

@@ -0,0 +1,7 @@
+## PROCESS MANAGEMENT
+
+The `subprocess` tool is for managing subprocesses related to the current project. Your project may produce servers and other background processes that should be started, stopped, and managed differently than what `shell_cmd` is designed for.
+
+You can create, list, and kill child processes using the `subprocess` tool. You can also read their `stdout` and `stderr` logs, which are being written to disk (and managed) as a convenience. When you stop a process, it's log file is closed and removed.
+
+If you want to stop a process _without_ removing the log, you use the `halt` subprocess command (cmd=halt, pid=###). You can then examine the logs. When you are done examining the logs, you use [cmd=kill, pid=###] to remove the process entry and logs.

gadget-code/src/services/chat-session.ts

@@ -317,18 +317,22 @@ class ChatSessionService extends DtpService {
     );
 
     const common = {
+      processManagementBlock: await fs.promises.readFile(
+        path.join(commonDir, "process-management-block.md"),
+        "utf-8",
+      ),
       scopeBlock: await fs.promises.readFile(
         path.join(commonDir, "scope-block.md"),
         "utf-8",
       ),
-      toolsBlock: await fs.promises.readFile(
-        path.join(commonDir, "tools-block.md"),
-        "utf-8",
-      ),
       subagentsBlock: await fs.promises.readFile(
         path.join(commonDir, "subagents.md"),
         "utf-8",
       ),
+      toolsBlock: await fs.promises.readFile(
+        path.join(commonDir, "tools-block.md"),
+        "utf-8",
+      ),
     };
 
     /*

gadget-drone/docs/subprocess.md

@@ -0,0 +1,50 @@
+# Gadget Drone Sub-Processes
+
+The Gadget Code agent wants to manage child processes, and give them a fancier name. Because we're fancy.
+
+In the Gadget ecosystem, we refer to a child process spawned by gadget-drone as a [SubProcess](../src/services/subprocess.ts). A `SubProcess` is just a child process, but it's also one that's being managed and controlled by gadget-drone based on commands received from the Agentic Workflow Loop using the `subprocess` tool (coming soon).
+
+## Intent
+
+This document is currently more of a specification to calibrate the model for working on Phase 1 of the SubProcess task.
+
+Phase 1: Basic Services, internal to drone.
+Phase 2: Expose SubProcess to gadget-code:frontend (IDE) via gadget-code:backend.
+
+We will work together through Phase 1 now in this session. At the end of this session, with the knowledge we gain by implementing the agent tool, we will refine this document. And that will be the end of Phase 1.
+
+We will then start a new session, and expose SubProcess to the IDE for User observability of the subprocesses.
+
+## SubProcessService
+
+[SubProcess](../src/services/subprocess.ts) is a service implemented by Gadget Drone for managing child processes. It offers methods for creating and managing child processes on behalf of Gadget Drone.
+
+This is a service, and not entirely implemented within an Agent tool in the toolbox, because the gadget-drone process will also implement TypeScript code to manage and monitor the processes.
+
+In the near future, `gadget-drone` will use this service to provide child process listing, status, and log services to `gadget-code:frontend` via `gadget-code:backend`. You can ignore this paragraph for this session because we will implement this in a different session. I mention it so you can include this detail in the refined version of this document at the end of this session.
+
+## SubProcess Tool
+
+A tool that is needed in the Agent's toolbox is the `subprocess` tool. This is what we are building in this session.
+
+The `subprocess` tool will offer the agent `create` (spawn), `list` (ps), `kill(pid:number)`, `killAll`, and `log(which: "stdout" | "stderr")`. We will define a standard Gadget tool that the agent can use to manage child processes.
+
+The tool makes use of methods on the `SubProcessService` service to execute tool function intent.
+
+## Process Management
+
+`gadget-drone` itself will need to close all running child processes when terminating. I have already added this logic to the gadget-drone main process during normal shutdown by registering the service, starting it, and stopping it using the standard Gadget service pattern.
+
+I have extended the system prompts for relevant modes with information about the `subprocess` tool using [process-management-block.md](../../gadget-code/data/prompts/common/process-management-block.md). That block is now being integrated into the system prompt for all modes (this work is already completed). Please ensure that the service and tool offer a clearly-defined and well-described path for the agent to take for managing processes related to the project.
+
+## Acceptance Criteria
+
+Let this section guide the creation of your TODO list.
+
+[ ] SubProcessService feature-complete
+[ ] Unit tests for SubProcessService
+[ ] `subprocess` agent tool feature-complete in agent toolbox
+[ ] Unit tests for `subprocess` tool
+[ ] gadget-drone documentation updated with the knowledge gained from this session
+
+When done, these become documentation describing what's here in the subprocess feature suite, and how to use it.

gadget-drone/src/gadget-drone.ts

@@ -13,6 +13,7 @@ import { input as inqInput, password as inqPassword } from "@inquirer/prompts";
 import AgentService, { IAgentWorkOrder } from "./services/agent.ts";
 import AiService from "./services/ai.ts";
 import PlatformService from "./services/platform.ts";
+import SubProcessService from "./services/subprocess.ts";
 import WorkspaceService from "./services/workspace.ts";
 import {
   DroneStatus,
@@ -177,6 +178,7 @@ class GadgetDrone extends GadgetProcess {
     await AgentService.start();
     await AiService.start();
     await PlatformService.start();
+    await SubProcessService.start();
 
     this.log.info("services started");
   }
@@ -187,6 +189,7 @@ class GadgetDrone extends GadgetProcess {
     await AgentService.stop();
     await AiService.stop();
     await PlatformService.stop();
+    await SubProcessService.stop();
 
     this.log.info("services stopped");
   }
@@ -252,26 +255,14 @@ class GadgetDrone extends GadgetProcess {
         this.onReleaseSessionLock.bind(this),
       );
       this.socket.on("sessionHeartbeat", this.onSessionHeartbeat.bind(this));
-      this.socket.on(
+      this.socket.on("abortWorkOrder", this.onAbortWorkOrder.bind(this));
-        "abortWorkOrder",
-        this.onAbortWorkOrder.bind(this),
-      );
       this.socket.on(
         "requestTermination",
         this.onRequestTermination.bind(this),
       );
-      this.socket.on(
+      this.socket.on("fileTreeRequest", this.onFileTreeRequest.bind(this));
-        "fileTreeRequest",
+      this.socket.on("fileReadRequest", this.onFileReadRequest.bind(this));
-        this.onFileTreeRequest.bind(this),
+      this.socket.on("fileWriteRequest", this.onFileWriteRequest.bind(this));
-      );
-      this.socket.on(
-        "fileReadRequest",
-        this.onFileReadRequest.bind(this),
-      );
-      this.socket.on(
-        "fileWriteRequest",
-        this.onFileWriteRequest.bind(this),
-      );
 
       /*
        * Handle socket disconnect: clear the heartbeat timer to prevent
@@ -375,9 +366,7 @@ class GadgetDrone extends GadgetProcess {
     /*
      * Check if this project is already deployed (by ID, not slug).
      */
-    const haveProjectInWorkspace = WorkspaceService.hasProjectById(
+    const haveProjectInWorkspace = WorkspaceService.hasProjectById(project._id);
-      project._id,
-    );
     if (!haveProjectInWorkspace) {
       this.socket.emit("status", `deploying project [slug=${project.slug}]`);
       await WorkspaceService.deployProject(project);
@@ -726,7 +715,10 @@ class GadgetDrone extends GadgetProcess {
 
   async onFileTreeRequest(
     args: FileTreeRequestArgs,
-    cb: (success: boolean, data: { entries?: FileTreeEntry[]; error?: string }) => void,
+    cb: (
+      success: boolean,
+      data: { entries?: FileTreeEntry[]; error?: string },
+    ) => void,
   ): Promise<void> {
     if (!this.sessionLock) {
       return cb(false, { error: "No session lock active" });
@@ -743,7 +735,9 @@ class GadgetDrone extends GadgetProcess {
     });
 
     if (!projectRoot) {
-      return cb(false, { error: `Project directory not found for slug: ${projectSlug}` });
+      return cb(false, {
+        error: `Project directory not found for slug: ${projectSlug}`,
+      });
     }
 
     // If no path specified, list from project root
@@ -755,7 +749,10 @@ class GadgetDrone extends GadgetProcess {
     // Security: Ensure resolved path is within project root
     const normalizedTarget = path.normalize(targetPath);
     const normalizedRoot = path.normalize(projectRoot);
-    if (!normalizedTarget.startsWith(normalizedRoot + path.sep) && normalizedTarget !== normalizedRoot) {
+    if (
+      !normalizedTarget.startsWith(normalizedRoot + path.sep) &&
+      normalizedTarget !== normalizedRoot
+    ) {
       this.log.warn("fileTreeRequest path traversal attempt", {
         targetPath: normalizedTarget,
         projectRoot: normalizedRoot,
@@ -785,12 +782,13 @@ class GadgetDrone extends GadgetProcess {
 
       this.log.debug("fileTreeRequest completed", {
         entryCount: entries.length,
-        entries: entries.slice(0, 10).map(e => e.name), // Log first 10 entries
+        entries: entries.slice(0, 10).map((e) => e.name), // Log first 10 entries
       });
 
       cb(true, { entries });
     } catch (error) {
-      const errorMessage = error instanceof Error ? error.message : String(error);
+      const errorMessage =
+        error instanceof Error ? error.message : String(error);
       this.log.error("failed to list directory for file tree", {
         path: args.path,
         targetPath,
@@ -819,7 +817,9 @@ class GadgetDrone extends GadgetProcess {
     });
 
     if (!projectRoot) {
-      return cb(false, { error: `Project directory not found for slug: ${projectSlug}` });
+      return cb(false, {
+        error: `Project directory not found for slug: ${projectSlug}`,
+      });
     }
 
     // Resolve path relative to project root
@@ -828,7 +828,10 @@ class GadgetDrone extends GadgetProcess {
     // Security: Ensure resolved path is within project root
     const normalizedTarget = path.normalize(targetPath);
     const normalizedRoot = path.normalize(projectRoot);
-    if (!normalizedTarget.startsWith(normalizedRoot + path.sep) && normalizedTarget !== normalizedRoot) {
+    if (
+      !normalizedTarget.startsWith(normalizedRoot + path.sep) &&
+      normalizedTarget !== normalizedRoot
+    ) {
       this.log.warn("fileReadRequest path traversal attempt", {
         targetPath: normalizedTarget,
         projectRoot: normalizedRoot,
@@ -850,14 +853,16 @@ class GadgetDrone extends GadgetProcess {
       // Check file size (limit to 1MB)
       const maxSize = 1 * 1024 * 1024; // 1MB
       if (stat.size > maxSize) {
-        return cb(false, { error: `File too large (${(stat.size / 1024 / 1024).toFixed(2)}MB). Maximum size is 1MB.` });
+        return cb(false, {
+          error: `File too large (${(stat.size / 1024 / 1024).toFixed(2)}MB). Maximum size is 1MB.`,
+        });
       }
 
       // Read file content
       const content = await fs.readFile(targetPath, "utf-8");
 
       // Check for binary file (null bytes)
-      if (content.includes('\0')) {
+      if (content.includes("\0")) {
         return cb(false, { error: "Cannot edit binary files" });
       }
 
@@ -868,7 +873,8 @@ class GadgetDrone extends GadgetProcess {
 
       cb(true, { content });
     } catch (error) {
-      const errorMessage = error instanceof Error ? error.message : String(error);
+      const errorMessage =
+        error instanceof Error ? error.message : String(error);
       this.log.error("failed to read file", {
         path: args.path,
         targetPath,
@@ -905,7 +911,9 @@ class GadgetDrone extends GadgetProcess {
     });
 
     if (!projectRoot) {
-      return cb(false, { error: `Project directory not found for slug: ${projectSlug}` });
+      return cb(false, {
+        error: `Project directory not found for slug: ${projectSlug}`,
+      });
     }
 
     // Resolve path relative to project root
@@ -914,7 +922,10 @@ class GadgetDrone extends GadgetProcess {
     // Security: Ensure resolved path is within project root
     const normalizedTarget = path.normalize(targetPath);
     const normalizedRoot = path.normalize(projectRoot);
-    if (!normalizedTarget.startsWith(normalizedRoot + path.sep) && normalizedTarget !== normalizedRoot) {
+    if (
+      !normalizedTarget.startsWith(normalizedRoot + path.sep) &&
+      normalizedTarget !== normalizedRoot
+    ) {
       this.log.warn("fileWriteRequest path traversal attempt", {
         targetPath: normalizedTarget,
         projectRoot: normalizedRoot,
@@ -942,7 +953,8 @@ class GadgetDrone extends GadgetProcess {
 
       cb(true, { success: true });
     } catch (error) {
-      const errorMessage = error instanceof Error ? error.message : String(error);
+      const errorMessage =
+        error instanceof Error ? error.message : String(error);
       this.log.error("failed to write file", {
         path: args.path,
         targetPath,
@@ -987,7 +999,11 @@ class GadgetDrone extends GadgetProcess {
       const fileTreeEntry: FileTreeEntry = {
         name: entry.name,
         path: relativePath,
-        type: entry.isSymbolicLink() ? "symlink" : entry.isDirectory() ? "directory" : "file",
+        type: entry.isSymbolicLink()
+          ? "symlink"
+          : entry.isDirectory()
+            ? "directory"
+            : "file",
         size: stat.size,
         modified: stat.mtime.toISOString(),
         isHidden: entry.name.startsWith("."),
@@ -1002,7 +1018,9 @@ class GadgetDrone extends GadgetProcess {
     return results;
   }
 
-  async onAbortWorkOrder(cb: (success: boolean, message?: string) => void): Promise<void> {
+  async onAbortWorkOrder(
+    cb: (success: boolean, message?: string) => void,
+  ): Promise<void> {
     this.log.info("abortWorkOrder received from platform", {
       registrationId: this.registration?._id,
       isProcessing: this.isProcessingWorkOrder,

gadget-drone/src/services/subprocess.ts

@@ -0,0 +1,165 @@
+// src/services/subprocess.ts
+// Copyright (C) 2026 Rob Colbert <rob.colbert@openplatform.us>
+// Licensed under the Apache License, Version 2.0
+
+import env from "../config/env.ts";
+import assert from "node:assert";
+
+import path from "node:path";
+import fs from "node:fs";
+
+import { ChildProcess, spawn } from "node:child_process";
+
+import { GadgetId, IProject } from "@gadget/api";
+
+import WorkspaceService from "./workspace.js";
+
+import { GadgetService } from "../lib/service.ts";
+
+export interface SubProcess {
+  project: IProject;
+  createdAt: Date;
+  updatedAt: Date;
+  process: ChildProcess;
+  pid: number;
+  stdoutFileName: string;
+  stdoutFile: fs.WriteStream;
+  stderrFileName: string;
+  stderrFile: fs.WriteStream;
+}
+
+type SubProcessMap = Map<number, SubProcess>;
+
+class SubProcessService extends GadgetService {
+  private processMap: SubProcessMap = new Map<number, SubProcess>();
+
+  get name(): string {
+    return "SubprocessService";
+  }
+  get slug(): string {
+    return "svc:subprocess";
+  }
+
+  async start(): Promise<void> {
+    this.log.info("started");
+  }
+
+  async stop(): Promise<void> {
+    await this.killAll();
+    this.log.info("stopped");
+  }
+
+  async create(
+    project: IProject,
+    cmd: string,
+    args?: string[],
+  ): Promise<SubProcess> {
+    const NOW = new Date();
+    assert(
+      WorkspaceService.gadgetDir,
+      "Gadget workspace directory not initialized",
+    );
+
+    const projectDir = WorkspaceService.getProjectDirectory(project.slug);
+    const child = spawn(cmd, args, { cwd: projectDir });
+    assert(child.pid, "subprocess did not define a process id");
+
+    const pid = child.pid;
+
+    const logFilePath = path.join(
+      WorkspaceService.gadgetDir,
+      "subprocess-logs",
+    );
+    const stdoutFileName = path.join(logFilePath, `subproc-${pid}.stdout.log`);
+    const stderrFileName = path.join(logFilePath, `subproc-${pid}.stderr.log`);
+
+    const sp: SubProcess = {
+      project,
+      createdAt: NOW,
+      updatedAt: NOW,
+      process: child,
+      pid,
+      stdoutFileName,
+      stdoutFile: fs.createWriteStream(stdoutFileName, "utf-8"),
+      stderrFileName,
+      stderrFile: fs.createWriteStream(stderrFileName, "utf-8"),
+    };
+    child.stdout.on("data", (data: any) => {
+      sp.updatedAt = new Date();
+      sp.stdoutFile.write(data);
+    });
+    child.stderr.on("data", (data: any) => {
+      sp.updatedAt = new Date();
+      sp.stderrFile.write(data);
+    });
+    this.processMap.set(pid, sp);
+
+    /*
+     * AGENT: This is how to create the tool function response that spawns a new
+     * child process. This lets the agent know the pid of the process so it can
+     * manage it. It also lets the agent know where stdout and stderr are being
+     * written.
+     */
+    const _response = [
+      "SUBPROCESS CREATED",
+      `pid: ${pid}`,
+      `stdout: ${stdoutFileName}`,
+      `stderr: ${stderrFileName}`,
+    ].join("\n");
+
+    return sp;
+  }
+
+  ps(): MapIterator<SubProcess> {
+    return this.processMap.values();
+  }
+
+  async killPid(pid: number): Promise<boolean> {
+    const sp = this.processMap.get(pid);
+    if (!sp) {
+      throw new Error("process not found");
+    }
+    return this.killSubProcess(sp);
+  }
+
+  async killProject(projectId: GadgetId): Promise<void> {
+    for (const sp of this.processMap.values()) {
+      if (sp.project._id !== projectId) {
+        continue;
+      }
+      this.processMap.delete(sp.pid);
+    }
+  }
+
+  async killSubProcess(sp: SubProcess): Promise<boolean> {
+    const pid = sp.pid;
+    this.log.info("killing subprocess", { pid });
+    if (!sp.process.kill("SIGINT")) {
+      return false;
+    }
+
+    sp.process.stdout?.removeAllListeners();
+    sp.process.stderr?.removeAllListeners();
+
+    this.log.info("closing subprocess logs", { pid });
+    sp.stdoutFile.close();
+    await fs.promises.rm(sp.stdoutFileName, { force: true });
+    sp.stderrFile.close();
+    await fs.promises.rm(sp.stderrFileName, { force: true });
+
+    this.processMap.delete(pid);
+    return true;
+  }
+
+  async killAll(): Promise<boolean> {
+    for (const [pid, sp] of this.processMap) {
+      if (!(await this.killSubProcess(sp))) {
+        return false;
+      }
+      this.processMap.delete(pid);
+    }
+    return true;
+  }
+}
+
+export default new SubProcessService();

gadget-drone/src/services/workspace.ts

@@ -68,7 +68,7 @@ export interface WorkOrderCache {
 }
 
 class WorkspaceService extends GadgetService {
-  private gadgetDir: string = "";
+  private _gadgetDir: string = "";
   private cacheDir: string = "";
 
   private workspaceFile: string = "";
@@ -84,6 +84,10 @@ class WorkspaceService extends GadgetService {
     return "svc:workspace";
   }
 
+  get gadgetDir(): string | undefined {
+    return this._gadgetDir || undefined;
+  }
+
   get workspaceData(): WorkspaceData | null {
     return this._workspaceData;
   }
@@ -114,12 +118,12 @@ class WorkspaceService extends GadgetService {
    * Validates or creates workspace.json with persistent identity.
    */
   async initialize(workspaceDir: string): Promise<void> {
-    this.gadgetDir = path.join(workspaceDir, ".gadget");
+    this._gadgetDir = path.join(workspaceDir, ".gadget");
-    this.cacheDir = path.join(this.gadgetDir, "cache");
+    this.cacheDir = path.join(this._gadgetDir, "cache");
-    this.workspaceFile = path.join(this.gadgetDir, "workspace.json");
+    this.workspaceFile = path.join(this._gadgetDir, "workspace.json");
 
     // Create directory structure
-    await fs.promises.mkdir(this.gadgetDir, { recursive: true });
+    await fs.promises.mkdir(this._gadgetDir, { recursive: true });
     await fs.promises.mkdir(this.cacheDir, { recursive: true });
 
     // Load or create workspace data
@@ -311,9 +315,7 @@ class WorkspaceService extends GadgetService {
    */
   markProjectDeployed(slug: string): void {
     if (!this._workspaceData) return;
-    const project = this._workspaceData.projects.find(
+    const project = this._workspaceData.projects.find((p) => p.slug === slug);
-      (p) => p.slug === slug,
-    );
     if (project) {
       const now = new Date().toISOString();
       project.clonedAt = now;