dtp/gadget

Rob Colbert 3e31d4d501 agent, tools, toolbox, tool loop, AI environment

2026-05-07 00:10:57 -04:00

23 KiB

Raw Blame History

Agent Toolbox Documentation

Overview

The Agent Toolbox is a foundational component of the Gadget platform that enables AI agents to perform real-world actions through a standardized, extensible tool system. Tools are functions that agents can invoke to interact with external services, manipulate files, search the web, and execute various operations.

Architecture

Design Philosophy

The toolbox system is built on these core principles:

Provider Agnostic: Tools work with any AI provider (Ollama, OpenAI, etc.)
Type Safety: Full TypeScript typing from definition to execution
Extensibility: Easy to add new tools without modifying core infrastructure
Security: Tools require explicit credentials configured in the environment
Error Handling: Comprehensive error reporting with recovery hints

Component Overview

┌─────────────────┐
│  AiToolbox      │ ← Manages tool registration and lookup
│  - env          │ ← Holds IAiEnvironment with credentials
│  - tools        │ ← Map of all registered tools
│  - modeSets     │ ← Tools organized by ChatSessionMode
└────────┬────────┘
         │
         │ register()
         ▼
┌─────────────────┐
│  AiTool         │ ← Abstract base class for all tools
│  - name         │ ← Unique tool identifier
│  - category     │ ← Tool category (search, file, etc.)
│  - definition   │ ← JSON Schema for AI provider
│  - execute()    │ ← Tool implementation
│  - toolbox      │ ← Access to environment & other tools
└────────┬────────┘
         │
         │ extends
         ▼
┌─────────────────┐
│ GoogleSearch    │ ← Example tool implementation
│ GrepTool        │ ← Future tools...
│ ...             │
└─────────────────┘

Core Interfaces

IAiEnvironment

The IAiEnvironment interface carries configuration and credentials from the application layer (which has access to YAML configs) down to the @gadget/ai package (which cannot read configs directly).

export interface IAiEnvironment {
  NODE_ENV: string;
  services?: {
    google?: {
      cse?: {
        apiKey?: string;
        engineId?: string;
      };
    };
    github?: {
      token?: string;
    };
    slack?: {
      token?: string;
      signingSecret?: string;
    };
    [key: string]: unknown;
  };
}

Key Design Decisions:

Services Object: Credentials are organized by service (google, github, slack, etc.)
Optional Everything: All fields are optional to support partial configurations
Extensible: The [key: string]: unknown index signature allows future services without breaking changes
Type Safety: Known services have typed interfaces

AiToolbox

The toolbox manages tool registration, organization, and retrieval:

export class AiToolbox {
  constructor(env: IAiEnvironment);

  // Register a tool for use by agents
  register(tool: AiTool, modes?: string[]): void;

  // Get a tool by name (for system tools)
  getTool(name: string): AiTool | undefined;

  // Get all tools for a specific mode
  getModeSet(mode: string): ToolSet | undefined;

  // Access environment credentials
  get env(): IAiEnvironment;
}

Registration Modes:

System Tools (no modes): Called by the platform itself (e.g., auto-naming chat sessions)
Agent Tools (with modes): Available to AI agents in specific ChatSessionModes (e.g., "code", "research", "debug")

AiTool

All tools extend this abstract base class:

export abstract class AiTool {
  protected _toolbox: AiToolbox;

  // Unique identifier for the tool
  abstract get name(): string;

  // Category for organization (search, file, code, etc.)
  abstract get category(): string;

  // JSON Schema definition sent to AI provider
  abstract get definition(): IToolDefinition;

  // Execute the tool with provided arguments
  abstract execute(args: IToolArguments, logger: IAiLogger): Promise<string>;
}

IToolDefinition

The tool definition follows the JSON Schema format expected by AI providers:

export interface IToolDefinition {
  type: "function";
  function: {
    name: string;
    description: string;
    parameters: IToolArguments;
  };
}

export interface IToolArguments {
  [key: string]: unknown;
}

Tool Execution Flow

The tool execution flow demonstrates the abstraction layer between the common tool interface and provider-specific implementations:

┌─────────────────────────────────────────────────────────────┐
│  Application Layer (gadget-drone / gadget-code)             │
│  - Reads YAML config                                        │
│  - Constructs IAiEnvironment                                │
│  - Creates AiToolbox                                        │
│  - Registers tools                                          │
└────────────────────┬────────────────────────────────────────┘
                     │
                     │ Pass tools to AI API
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  @gadget/ai - Provider Abstraction Layer                    │
│  - AiApi.chat() receives tools array                        │
│  - Transforms IToolDefinition → Provider format             │
│  - Sends to AI provider (Ollama/OpenAI)                     │
│  - Receives tool_calls from response                        │
│  - Executes tools via executeToolCalls()                    │
│  - Feeds results back to AI for next iteration              │
└────────────────────┬────────────────────────────────────────┘
                     │
                     │ Provider-specific SDK calls
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  AI Provider (Ollama / OpenAI)                              │
│  - Receives tool definitions in native format               │
│  - Decides when to call tools based on conversation         │
│  - Returns tool_calls in response                           │
└─────────────────────────────────────────────────────────────┘

Transformation Process

Tool Definition Transformation

Each provider has its own tool format. The @gadget/ai package transforms our common IToolDefinition to the provider's expected format:

OpenAI:

const openaiTool: ChatCompletionTool = {
  type: tool.definition.type,
  function: {
    name: tool.definition.function.name,
    description: tool.definition.function.description,
    parameters: tool.definition.function.parameters,
  },
};

Ollama:

const ollamaTool: Tool = {
  type: tool.definition.type,
  function: {
    name: tool.definition.function.name,
    description: tool.definition.function.description,
    parameters: tool.definition.function.parameters,
  },
};

Tool Call Execution

When the AI provider returns tool calls, they're executed by the base AiApi class:

protected async executeToolCalls(
  toolCalls: IToolCall[],
  tools: AiTool[],
): Promise<IToolCallResult[]> {
  const results: IToolCallResult[] = [];

  for (const toolCall of toolCalls) {
    // Find the tool by name
    const tool = tools.find(t => t.name === toolCall.function.name);

    // Parse arguments and execute
    const args = JSON.parse(toolCall.function.arguments);
    const result = await tool.execute(args, this.log);

    results.push({
      callId: toolCall.callId,
      functionName: toolCall.function.name,
      result,
    });
  }

  return results;
}

Result Transformation

Tool results are converted back to the provider's message format and appended to the conversation:

OpenAI:

messages.push({
  role: "tool",
  tool_call_id: result.callId,
  content: result.error || result.result,
});

Ollama:

messages.push({
  role: "tool",
  content: result.error || result.result,
  tool_name: result.functionName,
});

Iterative Execution

Both OpenAI and Ollama implementations support multiple rounds of tool calls:

AI receives tools and conversation
AI returns tool calls (or final response)
Tools are executed, results collected
Results appended to conversation as tool messages
Loop back to step 1 (up to maxToolIterations, default: 5)
Return final response or max iterations reached

This allows complex multi-step operations where the AI can:

Call multiple tools in parallel
Use results from one tool to inform the next
Refine its approach based on tool output

Configuration

Environment Setup

Tools requiring credentials need them configured in your YAML config files.

gadget-code.yaml

# Add to gadget-code.yaml
google:
  cse:
    apiKey: "${GOOGLE_CSE_API_KEY}"
    engineId: "${GOOGLE_CSE_ENGINE_ID}"

gadget-drone.yaml

# Add to gadget-drone.yaml
google:
  cse:
    apiKey: "${GOOGLE_CSE_API_KEY}"
    engineId: "${GOOGLE_CSE_ENGINE_ID}"

Environment Variables

Set the actual values in your shell or deployment environment:

export GOOGLE_CSE_API_KEY="your-api-key-here"
export GOOGLE_CSE_ENGINE_ID="your-engine-id-here"

TypeScript Usage

In consumer applications, construct the environment and pass to the AI API:

// gadget-drone/src/services/ai.ts
import { createAiApi, IAiEnvironment } from "@gadget/ai";
import env from "../config/env.js";

const aiEnv: IAiEnvironment = {
  NODE_ENV: env.NODE_ENV,
  services: {
    google: {
      cse: {
        apiKey: env.google.cse.apiKey,
        engineId: env.google.cse.engineId,
      },
    },
  },
};

const api = createAiApi(aiEnv, providerConfig, logger);

Example Tool: Google Search

The GoogleSearchTool demonstrates a complete tool implementation.

Overview

Purpose: Perform Google Custom Search Engine queries to find relevant web content.

Category: search

Credentials Required:

Google CSE API Key
Google CSE Engine ID

Tool Definition

{
  type: "function",
  function: {
    name: "search_google",
    description: "Perform a Google search for relevant information on the web.",
    parameters: {
      type: "object",
      properties: {
        query: {
          type: "string",
          description: "The search query string.",
        },
        num_results: {
          type: "number",
          description: "Number of search results to return (default: 10, max: 10).",
        },
        siteSearch: {
          type: "string",
          description: "Optional site to restrict the search to (e.g. github.com).",
        },
        dateRestrict: {
          type: "string",
          description: "Restrict results by date range (d1, d7, d30, d365).",
        },
        fileType: {
          type: "string",
          description: "Restrict results to file type (pdf, doc, xls, ppt).",
        },
        sort: {
          type: "string",
          enum: ["relevance", "date"],
          description: "Sort order for results.",
        },
        start: {
          type: "number",
          description: "Starting index for pagination (default: 1).",
        },
      },
      required: ["query"],
    },
  }
}

Usage Examples

Basic Search:

AI Agent: "Let me search for the latest TypeScript 5.0 features."
Tool Call: search_google({ query: "TypeScript 5.0 new features" })

Site-Restricted Search:

AI Agent: "I'll search the React documentation for useEffect best practices."
Tool Call: search_google({
  query: "useEffect best practices",
  siteSearch: "react.dev"
})

Date-Restricted Search:

AI Agent: "Let me find recent news about AI regulation from the past week."
Tool Call: search_google({
  query: "AI regulation news",
  dateRestrict: "d7",
  sort: "date"
})

File Type Search:

AI Agent: "I'll search for Python style guide PDFs."
Tool Call: search_google({
  query: "Python style guide",
  fileType: "pdf"
})

Implementation Details

File: packages/ai/src/tools/search/google.ts

Key Methods:

execute(args, logger): Validates input, calls search, formats results
search(query, options): Makes the Google CSE API call
parseCseError(error): Converts API errors to formatted tool errors

Error Handling:

The tool provides detailed error messages for common issues:

401 Unauthorized: Invalid API key
403 Forbidden: Engine ID or permission issues
429 Rate Limited: Quota exceeded with retry-after hint
Network Errors: Connection failures
Missing Parameters: Clear guidance on required fields

Result Formatting:

Search results are formatted as human-readable text:

Here are some relevant search results I found:

Title: TypeScript 5.0 Release Notes
Link: https://devblogs.microsoft.com/typescript/announcing-typescript-5-0/
Source: devblogs.microsoft.com
Snippet: TypeScript 5.0 introduces decorators, const type parameters, and more...

Title: What's New in TypeScript 5.0
Link: https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-0.html
Snippet: A comprehensive guide to all the new features in TypeScript 5.0...

Error Handling

IToolError Interface

All tool errors follow a standardized format:

export interface IToolError {
  code: ToolErrorCode;
  message: string;
  parameter?: string;
  expected?: string;
  example?: string;
  recoveryHint?: string;
}

Error Codes

Available error codes in ToolErrorCode:

MISSING_PARAMETER - Required parameter not provided
INVALID_PARAMETER - Parameter value is invalid
NOT_FOUND - Resource not found
PERMISSION_DENIED - Insufficient permissions
OPERATION_FAILED - General operation failure
OPERATION_NOT_ALLOWED - Operation not permitted
VALIDATION_ERROR - Input validation failed
RATE_LIMITED - Rate limit exceeded
LIMIT_EXCEEDED - Quota or limit exceeded
INVALID_CRON_SPEC - Invalid cron expression
INVALID_OPERATION - Operation not supported
TIMEOUT - Operation timed out
INVALID_TOOL_ARGUMENTS - Tool arguments invalid
SUBAGENT_FAILED - Subagent execution failed
TOOL_EXECUTION_FAILED - General tool execution error
UNAUTHORIZED - Authentication required
FORBIDDEN - Access denied
RATE_LIMIT_EXCEEDED - Rate limit exceeded
NETWORK_ERROR - Network connectivity issue
SECURITY_VIOLATION - Security policy violation

formatError Function

The formatError function creates consistent error messages:

const error: IToolError = {
  code: "MISSING_PARAMETER",
  message: "The 'query' parameter is required.",
  parameter: "query",
  expected: "A non-empty string containing the search query.",
  example: 'search_google(query: "latest AI news")',
  recoveryHint:
    "Provide a 'query' parameter with your search terms and try again.",
};

return formatError(error);

Output:

TOOL ERROR: MISSING_PARAMETER
The 'query' parameter is required.
PARAMETER: query
EXPECTED: A non-empty string containing the search query.
EXAMPLE: search_google(query: "latest AI news")
RECOVERY HINT: Provide a 'query' parameter with your search terms and try again.

Creating New Tools

Step 1: Create Tool Class

Extend AiTool and implement the required methods:

import { AiTool, IToolArguments, IToolDefinition } from "../tool.js";
import { IAiLogger } from "../../api.js";
import { formatError } from "../tool-error.js";

export class MyNewTool extends AiTool {
  get name(): string {
    return "my_tool_name";
  }

  get category(): string {
    return "category_name"; // e.g., "search", "file", "code"
  }

  get definition(): IToolDefinition {
    return {
      type: "function",
      function: {
        name: this.name,
        description: "Clear description of what the tool does.",
        parameters: {
          type: "object",
          properties: {
            param1: {
              type: "string",
              description: "Description of param1",
            },
            param2: {
              type: "number",
              description: "Description of param2",
            },
          },
          required: ["param1"],
        },
      },
    };
  }

  async execute(args: IToolArguments, logger: IAiLogger): Promise<string> {
    // Validate parameters
    if (!args.param1) {
      return formatError({
        code: "MISSING_PARAMETER",
        message: "param1 is required",
        parameter: "param1",
      });
    }

    // Access credentials if needed
    const apiKey = this.toolbox.env.services?.myService?.apiKey;
    if (!apiKey) {
      throw new Error("API key not configured in environment");
    }

    // Perform the operation
    logger.debug("executing my tool", { args });

    try {
      const result = await this.doSomething(args.param1);
      return `Operation successful: ${result}`;
    } catch (error) {
      return formatError({
        code: "OPERATION_FAILED",
        message: (error as Error).message,
      });
    }
  }

  private async doSomething(param1: string): Promise<string> {
    // Implementation here
    return "result";
  }
}

Step 2: Add Credentials (if needed)

Update IAiEnvironment in packages/ai/src/config/env.ts:

export interface IAiEnvironment {
  NODE_ENV: string;
  services?: {
    google?: {
      cse?: {
        apiKey?: string;
        engineId?: string;
      };
    };
    myService?: {
      apiKey?: string;
      endpoint?: string;
    };
    [key: string]: unknown;
  };
}

Update config types in packages/config/src/types.ts:

export interface GadgetDroneConfig {
  // ... existing fields
  myService?: {
    apiKey: string;
    endpoint: string;
  };
}

Update consumer config readers to populate the environment.

Step 3: Register the Tool

In gadget-drone startup code (to be implemented):

import { AiToolbox } from "@gadget/ai";
import { MyNewTool } from "@gadget/ai/tools/my-tool.js";

const toolbox = new AiToolbox(aiEnv);

// Register as system tool (no modes)
toolbox.register(new MyNewTool());

// Or register for specific modes
toolbox.register(new MyNewTool(), ["code", "debug"]);

Testing Tools

Unit Tests

Test tool validation and execution:

import { describe, it, expect } from "vitest";
import { GoogleSearchTool } from "../src/tools/search/google.js";
import { AiToolbox } from "../src/toolbox.js";
import { createEmptyEnvironment } from "../src/config/env.js";

describe("GoogleSearchTool", () => {
  const env = createEmptyEnvironment();
  const toolbox = new AiToolbox(env);
  const tool = new GoogleSearchTool(toolbox);

  it("should have correct name", () => {
    expect(tool.name).toBe("search_google");
  });

  it("should have correct category", () => {
    expect(tool.category).toBe("search");
  });

  it("should have query in required parameters", () => {
    const params = tool.definition.function.parameters;
    expect(params.required).toContain("query");
  });

  it("should return error for missing query", async () => {
    const result = await tool.execute(
      {},
      { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} },
    );
    expect(result).toContain("TOOL ERROR: MISSING_PARAMETER");
    expect(result).toContain("query");
  });
});

Integration Tests

Test with actual credentials (use mock API):

import { describe, it, expect, vi } from "vitest";
import { GoogleSearchTool } from "../src/tools/search/google.js";
import { AiToolbox } from "../src/toolbox.js";

describe("GoogleSearchTool integration", () => {
  it("should call Google CSE API with correct parameters", async () => {
    const env = {
      NODE_ENV: "test",
      services: {
        google: {
          cse: {
            apiKey: "test-key",
            engineId: "test-engine",
          },
        },
      },
    };

    const toolbox = new AiToolbox(env);
    const tool = new GoogleSearchTool(toolbox);

    // Mock the googleapis client
    // ... mock setup ...

    const result = await tool.execute(
      { query: "test query", num_results: 5 },
      logger,
    );

    expect(result).toContain("search results");
    // Verify API was called with correct params
  });
});

Best Practices

1. Validate Early, Validate Often

Check all required parameters at the start of execute():

async execute(args: IToolArguments, logger: IAiLogger): Promise<string> {
  if (!args.query || typeof args.query !== "string") {
    return formatError({
      code: "INVALID_PARAMETER",
      message: "query must be a string",
      parameter: "query",
      expected: "A string containing the search query",
    });
  }
  // ... rest of implementation
}

2. Use Structured Errors

Always use formatError() for consistent error reporting:

// ✅ Good
return formatError({
  code: "NOT_FOUND",
  message: "File not found",
  parameter: "path",
  recoveryHint: "Check that the file path exists and is accessible",
});

// ❌ Bad
throw new Error("File not found");

3. Log Appropriately

Use the logger for debugging and auditing:

logger.debug("starting tool execution", { args });
logger.info("tool completed successfully", { resultLength: result.length });
logger.warn("rate limit approaching", { remaining: 10 });
logger.error("tool failed", { error: error.message });

4. Handle Credentials Safely

Never log credentials or include them in error messages:

// ✅ Good
if (!apiKey) {
  throw new Error("API key not configured");
}

// ❌ Bad - exposes credential
logger.debug("using API key", { apiKey });

5. Document Thoroughly

Include in your tool:

Clear description in the definition
Parameter descriptions with examples
Error scenarios and recovery hints
Usage examples in comments

Future Enhancements

Planned improvements to the toolbox system:

Parallel Execution: Execute independent tool calls in parallel
Caching: Cache tool results for repeated queries
Rate Limiting: Built-in rate limiting per tool
Tool Metadata: Version, author, usage statistics
Result Streaming: Stream large results back to AI

Configuration Guide - Setting up tool credentials
Architecture Overview - System architecture
AI API Reference - @gadget/ai package documentation

23 KiB Raw Blame History

Agent Toolbox Documentation

Overview

Architecture

Design Philosophy

Component Overview

Core Interfaces

IAiEnvironment

AiToolbox

AiTool

IToolDefinition

Tool Execution Flow

Transformation Process

Iterative Execution

Configuration

Environment Setup

gadget-code.yaml

gadget-drone.yaml

Environment Variables

TypeScript Usage

Example Tool: Google Search

Overview

Tool Definition

Usage Examples

Implementation Details

Error Handling

IToolError Interface

Error Codes

formatError Function

Creating New Tools

Step 1: Create Tool Class

Step 2: Add Credentials (if needed)

Step 3: Register the Tool

Testing Tools

Unit Tests

Integration Tests

Best Practices

1. Validate Early, Validate Often

2. Use Structured Errors

3. Log Appropriately

4. Handle Credentials Safely

5. Document Thoroughly

Future Enhancements

Related Documentation

23 KiB

Raw Blame History