Building AI Agents
What makes something an "agent"
An LLM by itself takes input and produces output. Once. It has no memory beyond its context window, no ability to take actions in the world, no way to check its own work.
An AI agent is an LLM augmented with:
- Tools: functions the model can call (search the web, run code, query a database)
- Memory: persistence beyond the context window
- Planning: the ability to break a goal into steps and execute them in sequence
- Feedback loops: the ability to observe the result of actions and adjust
The simplest agent: an LLM that can call a web search tool. When it doesn't know something, it searches. When it gets results, it uses them to answer.
Tool calling (function calling)
Modern LLMs can be given a list of available tools (defined as JSON schemas). When the model determines it needs a tool, it outputs a structured tool call instead of plain text. Your code executes the tool, returns the result, and the model continues.
Example tool schema:`json
{
"name": "search_web",
"description": "Search the web for current information",
"parameters": {
"query": {"type": "string", "description": "The search query"}
}
}
The model decides when to call tools. It might call search_web, then call calculate, then call send_email — all in sequence, as needed.
Key insight: the LLM isn't executing code. It's outputting structured text that says "call this function with these parameters." Your orchestration layer does the actual execution.
Memory architecture
Agents need multiple types of memory:
In-context memory: everything in the current context window. Fast but limited. Disappears after the session.
External memory (RAG): long-term storage retrieved at query time. Unlimited capacity. Requires a retrieval step.
Episodic memory: logs of past conversations or actions. The agent can retrieve "what did I do last time this situation came up?"
Semantic memory: distilled facts and knowledge. "User prefers concise answers." "The project deadline is June 15."
Working memory: the active "scratchpad" — intermediate results the agent is tracking during a multi-step task.
Most production agents combine in-context + external memory. Full episodic and semantic memory is an active research area.
Planning patterns
ReAct (Reason + Act): the model alternates between reasoning steps ("I need to find the current price of X") and action steps (call search tool). Each observation feeds the next reasoning step. Simple, effective, widely used.
Chain of Thought (CoT): force the model to "think aloud" before answering. "Let me work through this step by step." Dramatically improves performance on complex reasoning tasks.
Plan-and-Execute: generate a complete plan first, then execute each step. Better for long-horizon tasks. Harder to adapt if early steps fail.
Tree of Thoughts: explore multiple reasoning paths simultaneously, evaluate them, prune bad ones. Computationally expensive but effective on hard problems.
Orchestration frameworks
You could build agents from scratch using raw API calls. Or you could use a framework:
LangChain: the original. Massive ecosystem, lots of integrations, sometimes overly complex for simple tasks.
LlamaIndex: stronger focus on data ingestion and RAG. Good for document-heavy agents.
CrewAI: multi-agent orchestration — multiple agents with different roles collaborating on a task.
OpenAI Assistants API: managed agents with built-in tool calling, code interpreter, and file retrieval. Less flexible but faster to ship.
Recommendation: start without a framework. Raw API calls with your own orchestration loop teaches you what's actually happening. Add a framework when the complexity justifies it.
What breaks in production agents
Tool failures: APIs go down, rate limits hit, unexpected response formats. Build retry logic and graceful degradation.
Context overflow: long-running agents accumulate context. You need strategies for compressing or summarizing history.
Infinite loops: agents can get stuck calling the same tool repeatedly. Add step limits and loop detection.
Hallucinated tool calls: models sometimes make up tool names or parameters that don't exist. Validate all tool calls before execution.
Cost: agents make multiple LLM calls per task. A task that takes 10 back-and-forth steps at $0.01/call adds up. Track token usage per agent run.
The best agent is the simplest one that solves the problem
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.