LLM Tool Calling Guide: Design Schemas, Handle Errors, and Build Reliable Agent Workflows

Overview

LLM tool calling, also called function calling, lets a model request an operation defined by your application. The model proposes a tool name and arguments; your server decides whether to execute that request, runs the underlying function, and sends a structured result back to the model. The model may then produce a final answer or request another tool.

The important boundary is that the model does not directly receive database credentials, execute arbitrary code, or bypass your application logic. It generates a candidate action. Your application remains responsible for authentication, authorization, validation, rate limits, side effects, and error handling.

A dependable implementation usually follows this sequence:

1. Define a small set of tools with precise names, descriptions, and input schemas.
2. Send the available tools with the user request and relevant system instructions.
3. Inspect the model response for tool requests.
4. Validate the tool name and arguments against application-owned rules.
5. Check permissions and risk before performing the operation.
6. Execute the tool with timeouts, logging, and bounded retries.
7. Return a predictable result or a structured error to the model.
8. Stop after a defined number of steps and return a user-facing response.

This pattern works across hosted model APIs, OpenAI-compatible endpoints, and agent frameworks, although exact request fields, schema support, streaming behavior, and parallel-call controls vary. Keep provider-specific code behind a small adapter so the rest of your application operates on one internal tool-call format. The OpenAI-compatible API tutorial provides related guidance for separating application logic from provider integration.

Checklist by scenario

1. Designing a new tool

• Give it one responsibility. Prefer find_customer and create_support_ticket over a general-purpose manage_customer_data tool.
• Use an action-oriented name. The name should make the intended operation clear to both the model and a reviewer.
• Describe limits and side effects. State whether the tool reads data, changes data, sends a message, charges an account, or starts an external job.
• Make arguments explicit. Use enums for fixed choices, bounded numbers where appropriate, and required fields for values the operation cannot safely infer.
• Keep schemas narrow. Do not expose internal fields, unrestricted query fragments, arbitrary URLs, or raw code unless the application has a separate safety design for them.
• Separate preview from commit. For consequential actions, provide a read-only preview or confirmation step before the final mutation.

A provider-neutral function definition can look like this:

{
  "name": "search_orders",
  "description": "Find orders for the authenticated customer using safe, bounded filters. Read-only.",
  "parameters": {
    "type": "object",
    "properties": {
      "status": {
        "type": "string",
        "enum": ["open", "shipped", "delivered", "cancelled"]
      },
      "limit": {
        "type": "integer",
        "minimum": 1,
        "maximum": 20
      }
    },
    "required": ["limit"],
    "additionalProperties": false
  }
}

Exact schema keywords accepted by a model API may differ. Treat the provider's supported subset as the minimum contract, then apply a second validation pass in your own code. A schema helps the model form valid arguments; it is not a security boundary.

2. Handling a single tool call

For a basic request, implement an explicit dispatcher rather than dynamically importing a function named by the model. A simplified flow is:

tool = registry.get(call.name)
if tool is None:
    return tool_error("unknown_tool", "This tool is not available." )

args = parse_json(call.arguments)
validate_against_schema(tool.input_schema, args)
authorize(user, tool, args)
result = execute_with_timeout(tool.handler, args)
return normalize_result(result)

Use a registry with allowlisted names and handlers. Parse arguments as JSON, reject duplicate or unexpected fields, normalize types where your policy permits it, and record a correlation ID for the model turn, tool call, user, and execution attempt. Never use the model's text as a substitute for structured arguments.

3. Handling malformed arguments and failures

Return errors in a form that helps the model recover without revealing internal details:

{
  "ok": false,
  "error": {
    "code": "invalid_argument",
    "message": "limit must be between 1 and 20",
    "retryable": false
  }
}

Useful error categories include unknown_tool, invalid_json, invalid_argument, unauthorized, not_found, rate_limited, dependency_timeout, and temporary_failure. Keep the public message concise. Log the detailed exception, request metadata, and upstream response securely on the server.

Retry only failures that are demonstrably temporary. Use an overall deadline, a small attempt limit, and backoff. Do not automatically retry non-idempotent operations such as sending a message or creating a payment unless the underlying API supports an idempotency key. A model retry is also a new decision, so cap the number of tool-call rounds.

4. Coordinating parallel calls

Parallel tool calls can reduce latency when operations are independent, such as retrieving account details and a service-status record. Before enabling them, confirm that calls do not compete for mutable state, exceed a downstream rate limit, or create inconsistent snapshots.

• Assign a unique ID to every call.
• Validate and authorize each call independently.
• Run only an allowlisted set concurrently.
• Apply per-call and batch-level timeouts.
• Return results in a stable, identifiable structure.
• Define whether one failure cancels the batch or produces a partial result.

5. Building a multi-step agent workflow

Use a state machine or bounded loop instead of an unrestricted agent. Store the conversation state, tool results, step count, elapsed time, and approval status. A practical workflow might be classify request → retrieve records → draft action → request confirmation → commit action. Make transitions explicit, especially before side effects.

For retrieval-heavy applications, tool results should contain only the fields needed for the next decision. This reduces context growth and limits accidental exposure of sensitive data. If retrieved content can contain instructions, treat it as untrusted data and apply prompt-injection defenses; see the prompt injection prevention guide.

6. Provider comparison checklist

Do not assume that a feature available in one API has the same semantics in another. Check the selected model and SDK documentation during integration tests, then pin or record the supported configuration in your application.

What to double-check

• Authorization: derive identity and tenant scope from the authenticated request, not from model-supplied arguments.
• Argument validation: enforce business rules after schema validation. For example, a syntactically valid order ID may still belong to another account.
• Permission boundaries: separate read tools from write tools, and require an application-controlled confirmation for high-impact actions.
• Result formatting: return compact, typed results with explicit empty states. Avoid dumping raw database rows or upstream HTML into the context.
• Timeouts: set limits at the HTTP client, individual tool, whole turn, and workflow levels.
• Observability: capture tool name, validation outcome, latency, retry count, result size, and final workflow status while excluding secrets and unnecessary personal data.
• Evaluation: test correct tool selection, missing information, ambiguous requests, invalid arguments, permission failures, prompt injection, timeout recovery, and repeated calls. The LLM evaluation guide covers a broader regression workflow.
• Versioning: treat tool names, argument meanings, and result formats as API contracts. Version or deprecate them deliberately rather than silently changing behavior. See prompt versioning and change management.

Common mistakes

1. Trusting the schema alone. A model can produce valid JSON that is still unauthorized, unsafe, or semantically wrong. Validate against user identity and domain rules.
2. Exposing one giant tool. Broad tools make selection ambiguous and increase the impact of malformed arguments. Split operations by responsibility.
3. Letting the model decide permissions. Instructions such as “only refund eligible orders” support behavior but cannot replace server-side authorization.
4. Retrying every error. Repeating a rejected argument or non-idempotent side effect creates cost and operational risk. Classify errors first.
5. Allowing unbounded loops. Set maximum steps, elapsed time, tool calls, and context growth. Return a useful fallback when the workflow cannot complete.
6. Returning raw failures. Stack traces, SQL errors, credentials, and internal URLs do not belong in model context or user responses.
7. Skipping confirmation for side effects. Reading an account and deleting an account should not have the same execution path.
8. Testing only the happy path. Reliability is mostly revealed by ambiguity, partial outages, malformed input, and changes to tool definitions.

When to revisit

Review this checklist whenever you add, remove, or rename a tool; change an argument or result format; switch model providers; enable parallel calls; introduce streaming; or move an agent workflow into a new cloud environment. Provider SDK releases and model changes are also reasons to rerun integration and regression tests, particularly around schema support, tool-choice controls, and response parsing.

Before a seasonal planning cycle or another predictable traffic increase, load-test downstream dependencies, review rate limits, check timeout budgets, and confirm that retries cannot duplicate side effects. Revisit permissions when your product adds a tenant, role, region, or data classification.

Use this practical update sequence:

1. Export representative tool-call traces, including failures and human escalations.
2. Run them against the current model, SDK, schemas, and provider adapter.
3. Compare tool-selection accuracy, validation failures, latency, cost signals, and side-effect safety.
4. Review any changed schema or prompt with application, security, and operations owners.
5. Deploy behind a feature flag or limited rollout, then monitor before widening access.
6. Record the new contract and retire obsolete tool definitions only after dependent workflows are migrated.

Reliable tool calling is less about finding a perfect prompt than about designing a controlled interface around model decisions. Keep tools narrow, validate every boundary, make failures recoverable, and measure the complete workflow. For related patterns across summarization, extraction, classification, and routing, review these AI workflow templates for developers.