How You Actually Build an AI Agent From Scratch!

It’s not magic; it’s not millions of dollars (okay, sometimes it is); it’s just smart software.
And today, you’re going to see how to actually build an AI agent step-by-step, with code that makes sense and real-world logic — not just a messy pile of Python.
Let’s make this simple.
An agent is just a piece of software that can:
Not all agents are equal. Some are dumb (like, “turn left when you see wall” dumb). Others are smart (chat with humans, give advice, drive a car).
There are 4 kinds:
2. Model-based → Have an internal map of the world
3. Goal-based → Chase a goal like “book cheapest flight”
4. Utility-based → Choose what gives them best value
✅ Example:
When you ask Siri, “What’s the weather?” — it figures out you want weather info (Goal-based agent).
Before we even think about being smart, let’s get the basics ready:
You’ll need Python installed, and these libraries:
pip install requests termcolor python-dotenv
Also, make sure you have a local API (like Ollama) running.
Or you could connect to OpenAI if you want something more out-of-the-box.
This is like setting up the kitchen before you start cooking. Don’t skip it.
The model service acts as the “brain” of the agent. It receives user input, processes it through an AI model, and returns a response.
Here is the implementation of a basic model service class:
from termcolor import coloredimport osfrom dotenv import load_dotenvimport requestsimport jsonload_dotenv()class OllamaModel:
def __init__(self, model, system_prompt, temperature=0, stop=None):
self.model_endpoint = "http://localhost:11434/api/generate"
self.temperature = temperature
self.model = model
self.system_prompt = system_prompt
self.headers = {"Content-Type": "application/json"}
self.stop = stop
def generate_text(self, prompt):
payload = {
"model": self.model,
"format": "json",
"prompt": prompt,
"system": self.system_prompt,
"stream": False,
"temperature": self.temperature,
"stop": self.stop
} try:
request_response = requests.post(
self.model_endpoint,
headers=self.headers,
data=json.dumps(payload)
)
request_response_json = request_response.json()
response = request_response_json['response']
response_dict = json.loads(response)
return response_dict
except requests.RequestException as e:
return {"error": f"Error in invoking model! {str(e)}"}
✅ In short:
The OllamaModel
class sends user prompts to the model API and returns a parsed JSON response, forming the thinking layer of the AI agent.
Your agent isn’t going to be much good without tools. Tools are just small functions that help it do things.
To enable actionable behavior, the agent requires a set of predefined functions or “tools.”
Let’s give it two basic tools:
def basic_calculator(input_str):
# Does math like add, divide, multiply try: input_str_clean = input_str.replace("'", "\"").strip().strip("\"")
input_dict = json.loads(input_str_clean)
num1 = input_dict['num1']
num2 = input_dict['num2']
operation = input_dict['operation']
except (json.JSONDecodeError, KeyError) as e:
return str(e), "Invalid input format."operations = { 'add': operator.add,
'subtract': operator.sub,
'multiply': operator.mul,
'divide': operator.truediv,
'power': operator.pow }
if operation in operations:
try:
result = operations[operation](num1, num2) return f"\n\nThe answer is: {result}."
except Exception as e:
return str(e), "\n\nError during operation."
else:
return "\n\nUnsupported operation."def reverse_string(input_string):
# Flips a string backwards
reversed_string = input_string[::-1]
return f"The reversed string is: {reversed_string}"
✅ Real world:
2. Reverser → Flip a string
Tools are reusable utility functions that perform specific tasks, which the agent can invoke based on user requests.
Okay cool, but how will the agent know what tools it has?
Once tools are defined, they must be organized systematically.
The Toolbox class manages tool registration and retrieval.
class ToolBox:
def __init__(self):
self.tools_dict = {}def store(self, functions_list):
for func in functions_list:
self.tools_dict[func.name] = func.doc return self.tools_dict
def tools(self):
tools_str = ""
for name, doc in self.tools_dict.items():
tools_str += f"{name}: \"{doc}\"\n"
return
tools_str.strip()
The Toolbox keeps track of available tools, making it easier for the agent to decide which operations to use.
This is the fun part.
The Agent class thinks, decides, and acts. It orchestrates the full pipeline: thinking, selecting a tool, and executing an action.
agent_system_prompt_template = """You are an agent with access to a toolbox. Given a user query, determine which tool to use.If no tool is needed, just respond.Here is a list of tools:{tool_descriptions}"""
class Agent:
def __init__(self, tools, model_service, model_name, stop=None):
self.tools = tools
self.model_service = model_service
self.model_name = model_name
self.stop = stop def prepare_tools(self):
toolbox = ToolBox()
toolbox.store(self.tools)
return toolbox.tools()
def think(self, prompt):
tool_descriptions = self.prepare_tools()
agent_system_prompt = agent_system_prompt_template.format(tool_descriptions=tool_descriptions)
model_instance = self.model_service( model=self.model_name, system_prompt=agent_system_prompt,
temperature=0,
stop=self.stop )
return model_instance.generate_text(prompt)
def work(self, prompt):
agent_response_dict = self.think(prompt)
tool_choice = agent_response_dict.get("tool_choice")
tool_input = agent_response_dict.get("tool_input")
for tool in self.tools:
if tool.name == tool_choice: response = tool(tool_input) print(colored(response, 'cyan'))
return
print(colored(tool_input, 'cyan'))
✅ In short:
if __name__ == "__main__":
tools = [basic_calculator, reverse_string]model_service = OllamaModel model_name = "llama3.1"
stop = "<|eot_id|>"
agent = Agent(tools=tools, model_service=model_service, model_name=model_name, stop=stop) while True:
prompt = input("Ask me anything: ")
if prompt.lower() == "exit":
break
agent.work(prompt)
User Asks Question
↓
Agent Thinks (Which Tool?)
↓
Agent Picks Best Tool
↓
Executes Tool
↓
Shows Result
You don’t need a billion-dollar lab to build something smart.
You just need some Python, a model, a few tools, and the logic you just saw.
If you really wanted, you could now build your own version of a mini-Siri or a SlackBot that answers smarter.
Building an AI agent from scratch isn’t just a technical exercise — it’s becoming a key skill for companies that want to stay ahead.
Instead of relying on generic, prebuilt models, businesses that know how to design custom AI agents can:
Across industries like healthcare, finance, logistics, and retail, AI agents are already transforming the way companies operate.
They automate repetitive tasks, improve decision-making with real-time insights, power predictive maintenance, and deliver highly personalized customer experiences.
As AI adoption grows, organizations that invest in building strong in-house AI capabilities; starting with agent design, will lead innovation instead of chasing it.