gadget/docs/agent-toolbox.md

Agent Toolbox Documentation

Overview

The Agent Toolbox is a foundational component of the Gadget platform that enables AI agents to perform real-world actions through a standardized, extensible tool system. Tools are functions that agents can invoke to interact with external services, manipulate files, search the web, and execute various operations.

Architecture

Design Philosophy

The toolbox system is built on these core principles:

1. Provider Agnostic: Tools work with any AI provider (Ollama, OpenAI, etc.)
2. Type Safety: Full TypeScript typing from definition to execution
3. Extensibility: Easy to add new tools without modifying core infrastructure
4. Security: Tools require explicit credentials configured in the environment
5. Error Handling: Comprehensive error reporting with recovery hints

Component Overview

┌─────────────────┐
│  AiToolbox      │ ← Manages tool registration and lookup
│  - env          │ ← Holds IAiEnvironment with credentials
│  - tools        │ ← Map of all registered tools
│  - modeSets     │ ← Tools organized by ChatSessionMode
└────────┬────────┘
         │
         │ register()
         ▼
┌─────────────────┐
│  AiTool         │ ← Abstract base class for all tools
│  - name         │ ← Unique tool identifier
│  - category     │ ← Tool category (search, file, etc.)
│  - definition   │ ← JSON Schema for AI provider
│  - execute()    │ ← Tool implementation
│  - toolbox      │ ← Access to environment & other tools
└────────┬────────┘
         │
         │ extends
         ▼
┌─────────────────┐
│ GoogleSearch    │ ← Example tool implementation
│ GrepTool        │ ← Future tools...
│ ...             │
└─────────────────┘

Core Interfaces

IAiEnvironment

The IAiEnvironment interface carries configuration and credentials from the application layer (which has access to YAML configs) down to the @gadget/ai package (which cannot read configs directly).

export interface IAiEnvironment {
  NODE_ENV: string;
  services?: {
    google?: {
      cse?: {
        apiKey?: string;
        engineId?: string;
      };
    };
    github?: {
      token?: string;
    };
    slack?: {
      token?: string;
      signingSecret?: string;
    };
    [key: string]: unknown;
  };
}

Key Design Decisions:

• Services Object: Credentials are organized by service (google, github, slack, etc.)
• Optional Everything: All fields are optional to support partial configurations
• Extensible: The [key: string]: unknown index signature allows future services without breaking changes
• Type Safety: Known services have typed interfaces

AiToolbox

The toolbox manages tool registration, organization, and retrieval:

export class AiToolbox {
  constructor(env: IAiEnvironment);

  // Register a tool for use by agents
  register(tool: AiTool, modes?: string[]): void;

  // Get a tool by name (for system tools)
  getTool(name: string): AiTool | undefined;

  // Get all tools for a specific mode
  getModeSet(mode: string): ToolSet | undefined;

  // Access environment credentials
  get env(): IAiEnvironment;
}

Registration Modes:

• System Tools (no modes): Called by the platform itself (e.g., auto-naming chat sessions)
• Agent Tools (with modes): Available to AI agents in specific ChatSessionModes (e.g., "code", "research", "debug")

AiTool

All tools extend this abstract base class:

export abstract class AiTool {
  protected _toolbox: AiToolbox;

  // Unique identifier for the tool
  abstract get name(): string;

  // Category for organization (search, file, code, etc.)
  abstract get category(): string;

  // JSON Schema definition sent to AI provider
  abstract get definition(): IToolDefinition;

  // Execute the tool with provided arguments
  abstract execute(args: IToolArguments, logger: IAiLogger): Promise<string>;
}

IToolDefinition

The tool definition follows the JSON Schema format expected by AI providers:

export interface IToolDefinition {
  type: "function";
  function: {
    name: string;
    description: string;
    parameters: IToolArguments;
  };
}

export interface IToolArguments {
  [key: string]: unknown;
}

Tool Execution Flow

The tool execution flow demonstrates the abstraction layer between the common tool interface and provider-specific implementations:

┌─────────────────────────────────────────────────────────────┐
│  Application Layer (gadget-drone / gadget-code)             │
│  - Reads YAML config                                        │
│  - Constructs IAiEnvironment                                │
│  - Creates AiToolbox                                        │
│  - Registers tools                                          │
└────────────────────┬────────────────────────────────────────┘
                     │
                     │ Pass tools to AI API
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  @gadget/ai - Provider Abstraction Layer                    │
│  - AiApi.chat() receives tools array                        │
│  - Transforms IToolDefinition → Provider format             │
│  - Sends to AI provider (Ollama/OpenAI)                     │
│  - Receives tool_calls from response                        │
│  - Executes tools via executeToolCalls()                    │
│  - Feeds results back to AI for next iteration              │
└────────────────────┬────────────────────────────────────────┘
                     │
                     │ Provider-specific SDK calls
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  AI Provider (Ollama / OpenAI)                              │
│  - Receives tool definitions in native format               │
│  - Decides when to call tools based on conversation         │
│  - Returns tool_calls in response                           │
└─────────────────────────────────────────────────────────────┘

Transformation Process

1. Tool Definition Transformation

   Each provider has its own tool format. The @gadget/ai package transforms our common IToolDefinition to the provider's expected format:

   OpenAI:

   const openaiTool: ChatCompletionTool = {
     type: tool.definition.type,
     function: {
       name: tool.definition.function.name,
       description: tool.definition.function.description,
       parameters: tool.definition.function.parameters,
     },
   };
   

   Ollama:

   const ollamaTool: Tool = {
     type: tool.definition.type,
     function: {
       name: tool.definition.function.name,
       description: tool.definition.function.description,
       parameters: tool.definition.function.parameters,
     },
   };
   

2. Tool Call Execution

   When the AI provider returns tool calls, they're executed by the base AiApi class:

   protected async executeToolCalls(
     toolCalls: IToolCall[],
     tools: AiTool[],
   ): Promise<IToolCallResult[]> {
     const results: IToolCallResult[] = [];
   
     for (const toolCall of toolCalls) {
       // Find the tool by name
       const tool = tools.find(t => t.name === toolCall.function.name);
   
       // Parse arguments and execute
       const args = JSON.parse(toolCall.function.arguments);
       const result = await tool.execute(args, this.log);
   
       results.push({
         callId: toolCall.callId,
         functionName: toolCall.function.name,
         result,
       });
     }
   
     return results;
   }
   

3. Result Transformation

   Tool results are converted back to the provider's message format and appended to the conversation:

   OpenAI:

   messages.push({
     role: "tool",
     tool_call_id: result.callId,
     content: result.error || result.result,
   });
   

   Ollama:

   messages.push({
     role: "tool",
     content: result.error || result.result,
     tool_name: result.functionName,
   });
   

Iterative Execution

Both OpenAI and Ollama implementations support multiple rounds of tool calls:

1. AI receives tools and conversation
2. AI returns tool calls (or final response)
3. Tools are executed, results collected
4. Results appended to conversation as tool messages
5. Loop back to step 1
6. Return final response

This allows complex multi-step operations where the AI can:

• Call multiple tools in parallel
• Use results from one tool to inform the next
• Refine its approach based on tool output

Configuration

Environment Setup

Tools requiring credentials need them configured in your YAML config files.

gadget-code.yaml

# Add to gadget-code.yaml
google:
  cse:
    apiKey: "${GOOGLE_CSE_API_KEY}"
    engineId: "${GOOGLE_CSE_ENGINE_ID}"

gadget-drone.yaml

# Add to gadget-drone.yaml
google:
  cse:
    apiKey: "${GOOGLE_CSE_API_KEY}"
    engineId: "${GOOGLE_CSE_ENGINE_ID}"

Environment Variables

Set the actual values in your shell or deployment environment:

export GOOGLE_CSE_API_KEY="your-api-key-here"
export GOOGLE_CSE_ENGINE_ID="your-engine-id-here"

TypeScript Usage

In consumer applications, construct the environment and pass to the AI API:

// gadget-drone/src/services/ai.ts
import { createAiApi, IAiEnvironment } from "@gadget/ai";
import env from "../config/env.js";

const aiEnv: IAiEnvironment = {
  NODE_ENV: env.NODE_ENV,
  services: {
    google: {
      cse: {
        apiKey: env.google.cse.apiKey,
        engineId: env.google.cse.engineId,
      },
    },
  },
};

const api = createAiApi(aiEnv, providerConfig, logger);

Example Tool: Google Search

The GoogleSearchTool demonstrates a complete tool implementation.

Overview

Purpose: Perform Google Custom Search Engine queries to find relevant web content.

Category: search

Credentials Required:

• Google CSE API Key
• Google CSE Engine ID

Tool Definition

{
  type: "function",
  function: {
    name: "search_google",
    description: "Perform a Google search for relevant information on the web.",
    parameters: {
      type: "object",
      properties: {
        query: {
          type: "string",
          description: "The search query string.",
        },
        num_results: {
          type: "number",
          description: "Number of search results to return (default: 10, max: 10).",
        },
        siteSearch: {
          type: "string",
          description: "Optional site to restrict the search to (e.g. github.com).",
        },
        dateRestrict: {
          type: "string",
          description: "Restrict results by date range (d1, d7, d30, d365).",
        },
        fileType: {
          type: "string",
          description: "Restrict results to file type (pdf, doc, xls, ppt).",
        },
        sort: {
          type: "string",
          enum: ["relevance", "date"],
          description: "Sort order for results.",
        },
        start: {
          type: "number",
          description: "Starting index for pagination (default: 1).",
        },
      },
      required: ["query"],
    },
  }
}

Usage Examples

Basic Search:

AI Agent: "Let me search for the latest TypeScript 5.0 features."
Tool Call: search_google({ query: "TypeScript 5.0 new features" })

Site-Restricted Search:

AI Agent: "I'll search the React documentation for useEffect best practices."
Tool Call: search_google({
  query: "useEffect best practices",
  siteSearch: "react.dev"
})

Date-Restricted Search:

AI Agent: "Let me find recent news about AI regulation from the past week."
Tool Call: search_google({
  query: "AI regulation news",
  dateRestrict: "d7",
  sort: "date"
})

File Type Search:

AI Agent: "I'll search for Python style guide PDFs."
Tool Call: search_google({
  query: "Python style guide",
  fileType: "pdf"
})

Implementation Details

File: packages/ai/src/tools/search/google.ts

Key Methods:

1. execute(args, logger): Validates input, calls search, formats results
2. search(query, options): Makes the Google CSE API call
3. parseCseError(error): Converts API errors to formatted tool errors

Error Handling:

The tool provides detailed error messages for common issues:

• 401 Unauthorized: Invalid API key
• 403 Forbidden: Engine ID or permission issues
• 429 Rate Limited: Quota exceeded with retry-after hint
• Network Errors: Connection failures
• Missing Parameters: Clear guidance on required fields

Result Formatting:

Search results are formatted as human-readable text:

Here are some relevant search results I found:

Title: TypeScript 5.0 Release Notes
Link: https://devblogs.microsoft.com/typescript/announcing-typescript-5-0/
Source: devblogs.microsoft.com
Snippet: TypeScript 5.0 introduces decorators, const type parameters, and more...

Title: What's New in TypeScript 5.0
Link: https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-0.html
Snippet: A comprehensive guide to all the new features in TypeScript 5.0...

Error Handling

IToolError Interface

All tool errors follow a standardized format:

export interface IToolError {
  code: ToolErrorCode;
  message: string;
  parameter?: string;
  expected?: string;
  example?: string;
  recoveryHint?: string;
}

Error Codes

Available error codes in ToolErrorCode:

• MISSING_PARAMETER - Required parameter not provided
• INVALID_PARAMETER - Parameter value is invalid
• NOT_FOUND - Resource not found
• PERMISSION_DENIED - Insufficient permissions
• OPERATION_FAILED - General operation failure
• OPERATION_NOT_ALLOWED - Operation not permitted
• VALIDATION_ERROR - Input validation failed
• RATE_LIMITED - Rate limit exceeded
• LIMIT_EXCEEDED - Quota or limit exceeded
• INVALID_CRON_SPEC - Invalid cron expression
• INVALID_OPERATION - Operation not supported
• TIMEOUT - Operation timed out
• INVALID_TOOL_ARGUMENTS - Tool arguments invalid
• SUBAGENT_FAILED - Subagent execution failed
• TOOL_EXECUTION_FAILED - General tool execution error
• UNAUTHORIZED - Authentication required
• FORBIDDEN - Access denied
• RATE_LIMIT_EXCEEDED - Rate limit exceeded
• NETWORK_ERROR - Network connectivity issue
• SECURITY_VIOLATION - Security policy violation

formatError Function

The formatError function creates consistent error messages:

const error: IToolError = {
  code: "MISSING_PARAMETER",
  message: "The 'query' parameter is required.",
  parameter: "query",
  expected: "A non-empty string containing the search query.",
  example: 'search_google(query: "latest AI news")',
  recoveryHint:
    "Provide a 'query' parameter with your search terms and try again.",
};

return formatError(error);

Output:

TOOL ERROR: MISSING_PARAMETER
The 'query' parameter is required.
PARAMETER: query
EXPECTED: A non-empty string containing the search query.
EXAMPLE: search_google(query: "latest AI news")
RECOVERY HINT: Provide a 'query' parameter with your search terms and try again.

Creating New Tools

Step 1: Create Tool Class

Extend AiTool and implement the required methods:

import { AiTool, IToolArguments, IToolDefinition } from "../tool.js";
import { IAiLogger } from "../../api.js";
import { formatError } from "../tool-error.js";

export class MyNewTool extends AiTool {
  get name(): string {
    return "my_tool_name";
  }

  get category(): string {
    return "category_name"; // e.g., "search", "file", "code"
  }

  get definition(): IToolDefinition {
    return {
      type: "function",
      function: {
        name: this.name,
        description: "Clear description of what the tool does.",
        parameters: {
          type: "object",
          properties: {
            param1: {
              type: "string",
              description: "Description of param1",
            },
            param2: {
              type: "number",
              description: "Description of param2",
            },
          },
          required: ["param1"],
        },
      },
    };
  }

  async execute(args: IToolArguments, logger: IAiLogger): Promise<string> {
    // Validate parameters
    if (!args.param1) {
      return formatError({
        code: "MISSING_PARAMETER",
        message: "param1 is required",
        parameter: "param1",
      });
    }

    // Access credentials if needed
    const apiKey = this.toolbox.env.services?.myService?.apiKey;
    if (!apiKey) {
      throw new Error("API key not configured in environment");
    }

    // Perform the operation
    logger.debug("executing my tool", { args });

    try {
      const result = await this.doSomething(args.param1);
      return `Operation successful: ${result}`;
    } catch (error) {
      return formatError({
        code: "OPERATION_FAILED",
        message: (error as Error).message,
      });
    }
  }

  private async doSomething(param1: string): Promise<string> {
    // Implementation here
    return "result";
  }
}

Step 2: Add Credentials (if needed)

Update IAiEnvironment in packages/ai/src/config/env.ts:

export interface IAiEnvironment {
  NODE_ENV: string;
  services?: {
    google?: {
      cse?: {
        apiKey?: string;
        engineId?: string;
      };
    };
    myService?: {
      apiKey?: string;
      endpoint?: string;
    };
    [key: string]: unknown;
  };
}

Update config types in packages/config/src/types.ts:

export interface GadgetDroneConfig {
  // ... existing fields
  myService?: {
    apiKey: string;
    endpoint: string;
  };
}

Update consumer config readers to populate the environment.

Step 3: Register the Tool

In gadget-drone startup code (to be implemented):

import { AiToolbox } from "@gadget/ai";
import { MyNewTool } from "@gadget/ai/tools/my-tool.js";

const toolbox = new AiToolbox(aiEnv);

// Register as system tool (no modes)
toolbox.register(new MyNewTool());

// Or register for specific modes
toolbox.register(new MyNewTool(), ["code", "debug"]);

Testing Tools

Unit Tests

Test tool validation and execution:

import { describe, it, expect } from "vitest";
import { GoogleSearchTool } from "../src/tools/search/google.js";
import { AiToolbox } from "../src/toolbox.js";
import { createEmptyEnvironment } from "../src/config/env.js";

describe("GoogleSearchTool", () => {
  const env = createEmptyEnvironment();
  const toolbox = new AiToolbox(env);
  const tool = new GoogleSearchTool(toolbox);

  it("should have correct name", () => {
    expect(tool.name).toBe("search_google");
  });

  it("should have correct category", () => {
    expect(tool.category).toBe("search");
  });

  it("should have query in required parameters", () => {
    const params = tool.definition.function.parameters;
    expect(params.required).toContain("query");
  });

  it("should return error for missing query", async () => {
    const result = await tool.execute(
      {},
      { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} },
    );
    expect(result).toContain("TOOL ERROR: MISSING_PARAMETER");
    expect(result).toContain("query");
  });
});

Integration Tests

Test with actual credentials (use mock API):

import { describe, it, expect, vi } from "vitest";
import { GoogleSearchTool } from "../src/tools/search/google.js";
import { AiToolbox } from "../src/toolbox.js";

describe("GoogleSearchTool integration", () => {
  it("should call Google CSE API with correct parameters", async () => {
    const env = {
      NODE_ENV: "test",
      services: {
        google: {
          cse: {
            apiKey: "test-key",
            engineId: "test-engine",
          },
        },
      },
    };

    const toolbox = new AiToolbox(env);
    const tool = new GoogleSearchTool(toolbox);

    // Mock the googleapis client
    // ... mock setup ...

    const result = await tool.execute(
      { query: "test query", num_results: 5 },
      logger,
    );

    expect(result).toContain("search results");
    // Verify API was called with correct params
  });
});

Best Practices

1. Validate Early, Validate Often

Check all required parameters at the start of execute():

async execute(args: IToolArguments, logger: IAiLogger): Promise<string> {
  if (!args.query || typeof args.query !== "string") {
    return formatError({
      code: "INVALID_PARAMETER",
      message: "query must be a string",
      parameter: "query",
      expected: "A string containing the search query",
    });
  }
  // ... rest of implementation
}

2. Use Structured Errors

Always use formatError() for consistent error reporting:

// ✅ Good
return formatError({
  code: "NOT_FOUND",
  message: "File not found",
  parameter: "path",
  recoveryHint: "Check that the file path exists and is accessible",
});

// ❌ Bad
throw new Error("File not found");

3. Log Appropriately

Use the logger for debugging and auditing:

logger.debug("starting tool execution", { args });
logger.info("tool completed successfully", { resultLength: result.length });
logger.warn("rate limit approaching", { remaining: 10 });
logger.error("tool failed", { error: error.message });

4. Handle Credentials Safely

Never log credentials or include them in error messages:

// ✅ Good
if (!apiKey) {
  throw new Error("API key not configured");
}

// ❌ Bad - exposes credential
logger.debug("using API key", { apiKey });

5. Document Thoroughly

Include in your tool:

• Clear description in the definition
• Parameter descriptions with examples
• Error scenarios and recovery hints
• Usage examples in comments

Future Enhancements

Planned improvements to the toolbox system:

1. Parallel Execution: Execute independent tool calls in parallel
2. Caching: Cache tool results for repeated queries
3. Rate Limiting: Built-in rate limiting per tool
4. Tool Metadata: Version, author, usage statistics
5. Result Streaming: Stream large results back to AI

Related Documentation

• Configuration Guide - Setting up tool credentials
• Architecture Overview - System architecture
• AI API Reference - @gadget/ai package documentation