Most of this post has been written by an AI, but it’s for a good cause.

You cannot really avoid AI nowadays. In order to “use it properly” vs “not being used” by it, my brain demands a minimal level of understanding of how it works. At some point I started exploring ways of running your own models and I tried Ollama and LM Studio, and the big questions came up:

• how do you “run” an AI model?
• which components are involved?

So I asked Claude a set of questions, and the answers are reported below.

This conversation was generated with Claude (claude.ai) back in March 2026. It covers the fundamentals of how large language models work, inference engines, training, agents, MCP, and the message role system.

How do you “run” an AI model?

Q: How do you “run” an AI model? In which programming/markup language and/or file format is it built?

At its core, “running” an AI model means executing a mathematical computation — specifically a massive series of matrix multiplications and other operations — to take some input (your prompt) and produce an output (a response).

Here’s what actually happens step by step:

1. Tokenisation — Your text is broken into tokens (roughly word fragments). “Running” becomes [“Run”, “ning”], for example.
2. Embedding — Each token is converted into a vector: a long list of numbers that represents its meaning in high-dimensional space.
3. Forward pass — That vector travels through the model’s layers (in transformer-based models, these are called attention layers and feed-forward layers). Each layer applies learned weights — essentially billions of numbers baked in during training — to transform the data.
4. Sampling — The final layer outputs a probability distribution over all possible next tokens. The software picks one (using strategies like temperature sampling or greedy decoding), appends it, and repeats until the response is complete.

All of this requires a lot of RAM (to hold the model’s weights in memory) and ideally a GPU (which is very good at parallel matrix math). Tools like LM Studio and Ollama handle loading the model into memory, managing the GPU/CPU resources, and exposing a simple interface so you don’t have to think about any of that.

What language/format are AI models built in?

There are two distinct things here: the training code and the model file itself.

Training & inference code is almost universally written in Python, using frameworks like:

• PyTorch (most common for research and modern LLMs)
• TensorFlow / JAX (also popular, especially at Google)

The model file — the actual saved weights — has a few common formats:

LM Studio and Ollama predominantly use GGUF files because the underlying inference engine they rely on — llama.cpp — is written in C++ and is extremely optimised for running quantised GGUF models efficiently on consumer hardware.

The short version: you “run” a model by loading its weight file into memory and doing a forward pass of matrix math for every token generated. The weights live in a file (usually GGUF for local use), the inference engine is C++, and the training framework is Python.

How do you generate the weights with training?

What are weights, conceptually?

Weights are just numbers — billions of them — stored in matrices. Each weight represents the “strength” of a connection between neurons in the network. Before training, they’re essentially random. The entire point of training is to adjust them until the model becomes useful.

The Training Loop

Training is essentially a repeated cycle of four steps:

1. Forward Pass

You feed the model a piece of text (e.g. “The cat sat on the”) and it predicts the next token. With random weights at the start, it’ll guess something nonsensical.

2. Calculate the Loss

You compare the model’s prediction to the correct answer (e.g. “mat”) using a loss function — a mathematical formula that produces a single number representing how wrong the model was. A high loss = very wrong. The most common loss function for LLMs is cross-entropy loss.

3. Backpropagation

This is the key algorithm. It works backwards through every layer of the network, calculating how much each individual weight contributed to the error. This gives you a gradient for each weight — essentially an arrow saying “adjust this weight in this direction by roughly this much.”

This works thanks to the chain rule from calculus, applied automatically by frameworks like PyTorch. You never calculate these gradients by hand — PyTorch’s autograd engine does it for you.

4. Gradient Descent (Weight Update)

You nudge every weight slightly in the direction that reduces the loss. The size of the nudge is controlled by the learning rate — a critical hyperparameter. Too large and training is unstable; too small and it takes forever.

The most common optimiser used for LLMs is Adam (Adaptive Moment Estimation), which is a smarter version of basic gradient descent.

Then you repeat this millions of times across an enormous dataset.

The Data

Modern LLMs are trained on truly vast amounts of text — think a significant fraction of the entire internet, books, code repositories, scientific papers, and more. The dataset for a frontier model is measured in trillions of tokens.

The Phases of Training

The Hardware Reality

Pre-training a large model requires thousands of GPUs (typically NVIDIA H100s) running in parallel, weeks to months of continuous computation, and hundreds of millions to billions of dollars in compute costs for frontier models.

The one-sentence summary: training is just repeatedly showing the model examples, measuring how wrong it is, working backwards to figure out which weights caused the error, and nudging them slightly in the right direction — billions of times over.

Are there other engines than llama.cpp?

Yes, quite a few! llama.cpp gets a lot of attention because it democratised local AI, but it’s one of several inference engines, each with different strengths.

CPU/Consumer-focused

• llama.cpp — The most popular for local use. Written in C++, excellent quantisation support, runs on almost anything including Apple Silicon and even CPUs without a GPU.
• Ollama — Actually built on top of llama.cpp, it just adds a nicer model management layer and a local API server.

GPU-focused / High Performance

• vLLM — Probably the dominant engine for serving LLMs in production at scale. Written in Python/CUDA, it introduced PagedAttention, a clever memory management technique that dramatically increases throughput when serving many users simultaneously.
• TensorRT-LLM — NVIDIA’s own inference engine, highly optimised specifically for their GPUs.
• DeepSpeed Inference — Microsoft’s engine, strong at handling very large models across multiple GPUs.

Framework-native

• Hugging Face Transformers — Not optimised for raw speed, but incredibly flexible and the de facto standard for research and experimentation.
• ExLlamaV2 — A community favourite for enthusiasts, very fast on NVIDIA GPUs with its own quantisation format (EXL2).

Apple-specific

• MLX — Apple’s own machine learning framework, optimised specifically for Apple Silicon’s unified memory architecture.

Browser / Edge

• WebLLM — Runs models directly in the browser using WebGPU.
• ONNX Runtime — Microsoft’s cross-platform runtime, designed to run models on edge devices, mobile, and embedded systems.

Rough use-case guide

• Running locally on your laptop → llama.cpp / Ollama / MLX (on Mac)
• Serving an API in production → vLLM
• Research and experimentation → Hugging Face Transformers
• Maximum speed on NVIDIA hardware → TensorRT-LLM or ExLlamaV2
• Running in a browser → WebLLM

LM Studio and MLX models — which engine is used?

LM Studio is primarily built around llama.cpp as its core engine. However, it has added support for MLX models as an additional backend, meaning it can invoke the MLX framework to run those models.

• When you load a GGUF model, it hands it to its built-in llama.cpp engine
• When you load an MLX model, it delegates to the MLX framework to do the actual inference

Think of LM Studio like a universal media player — it doesn’t re-implement every codec itself, it just knows which engine to call for which file type.

Why offer both?

The flow: from UI to model and back

The flow is slightly more layered than a direct function call:

You (type text)
      ↓
   UI layer          ← LM Studio's chat interface, or Ollama's terminal
      ↓
   API server        ← Both apps run a local HTTP server (e.g. localhost:11434 for Ollama)
      ↓
   Inference engine  ← llama.cpp, MLX, etc.
      ↓
   Model weights     ← The actual GGUF/MLX file loaded in memory
      ↓
   (response travels back up the same stack)

Why HTTP?

• Universality — Every programming language knows how to make an HTTP request, giving interoperability for free.
• OpenAI API compatibility — Ollama and LM Studio mimic OpenAI’s API format, so any app already built for ChatGPT can be redirected to your local model by just changing one URL.
• Process isolation — The UI and the engine run as separate processes.
• Not actually heavy locally — HTTP over localhost (loopback interface) is extremely fast; data never leaves your machine.

How the API server talks to the engine

• Direct library call — The engine (e.g. llama.cpp) is often compiled as a library and the API server calls its functions directly in memory.
• stdin/stdout pipes — Sometimes the engine runs as a separate subprocess and the server communicates via stdin/stdout.
• Unix sockets / named pipes — A faster alternative to HTTP for local inter-process communication.

Tokenisation

The app doesn’t feed your raw text into the engine — it first gets tokenised by the inference engine before entering the model, and the output tokens get decoded back to text before being displayed.

Streaming

The response doesn’t come back all at once. The model generates one token at a time, and the engine streams each token back up through the API to the UI as soon as it’s produced — which is why you see responses appearing word by word.

Model format on disk and in memory

On disk

GGUF (for local use) is a binary file format containing everything in a single file: - Model metadata (architecture, context length, tokeniser vocabulary) - All the weight tensors, usually quantised — compressed from 32-bit or 16-bit floating point numbers down to smaller representations (8-bit, 4-bit, even 2-bit).

Safetensors / PyTorch .bin store weights at full or half precision (float32 or float16), without quantisation. Much larger files, but lossless.

What quantisation actually means

When loaded into memory

Dequantisation on the fly — The engine loads the quantised weights from disk, but before doing the actual matrix multiplications it dequantises them back up to float16 in small chunks — block by block as computation proceeds.

In RAM vs VRAM — The weights end up split across VRAM (GPU memory, actively being computed on, in float16) and RAM (system memory, the rest in quantised form, waiting to be paged in).

The KV Cache — An additional in-memory structure that stores intermediate attention computations for all previous tokens so they don’t need to be recomputed on every new token. This can take up significant memory for long conversations.

Disk (GGUF)                    Memory
─────────────────              ──────────────────────────────────
Metadata          →            Loaded once, kept in structs
Quantised weights →            Dequantised to float16 on the fly
Tokeniser vocab   →            Loaded into a lookup table
                               + KV Cache built up as you chat

The KV Cache — what it actually is

The KV cache has nothing to do with caching systems like Redis, Memcached, or any database. It’s a completely different concept specific to transformer models.

What K and V stand for

In transformer attention layers, for every token the model processes, it computes three vectors: - Q (Query) — “what am I looking for?” - K (Key) — “what do I contain?” - V (Value) — “what do I actually output?”

What the KV cache actually is

It’s simply a tensor stored in memory — a big matrix that accumulates the K and V vectors for every token as they’re computed, so they never need to be recomputed.

Token 1 generated  → KV cache: [KV₁]
Token 2 generated  → KV cache: [KV₁, KV₂]
Token 3 generated  → KV cache: [KV₁, KV₂, KV₃]
...

It lives in VRAM or RAM, right alongside the model weights — just another block of floating point numbers managed directly by the inference engine. No external system involved.

This is also why context length has a memory cost. A longer conversation = a bigger KV cache = more memory needed.

Comparison with Redis

What are AI agents?

An agent is a system that wraps an LLM in a loop, giving it the ability to: - Take actions (call tools, browse the web, run code, read files) - Observe results of those actions - Decide what to do next based on those results - Repeat until the task is done

The core loop

User goal enters here
      ↓
LLM thinks: what should I do?
      ↓
LLM picks a tool to call
      ↓
Tool executes (real world action)
      ↓
Result fed back to LLM
      ↓
LLM thinks: am I done?
   yes ↓        no → loop again
Final answer returned

What are tools?

Tools are just functions the agent can invoke — web search, read/write files, execute code, query a database, call an external API, send an email, browse a webpage, etc.

The LLM is given descriptions of available tools, and it outputs structured requests to use them (function calling or tool use). The agent framework intercepts those requests, runs the actual function, and returns the result.

Agent frameworks

Minimal agent in Python

import anthropic

client = anthropic.Anthropic()

def search_web(query):
    return f"Results for: {query}"

tools = [
    {
        "name": "search_web",
        "description": "Search the web for current information",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search query"}
            },
            "required": ["query"]
        }
    }
]

messages = [{"role": "user", "content": "What is the latest news about AI?"}]

while True:
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1000,
        tools=tools,
        messages=messages
    )

    if response.stop_reason == "end_turn":
        print(response.content[0].text)
        break

    if response.stop_reason == "tool_use":
        tool_call = next(b for b in response.content if b.type == "tool_use")
        result = search_web(tool_call.input["query"])
        messages.append({"role": "assistant", "content": response.content})
        messages.append({"role": "user", "content": [
            {"type": "tool_result", "tool_use_id": tool_call.id, "content": result}
        ]})

Infinite loop risk in agents

Yes, it’s a real concern. An agent can get stuck in:

• True infinite loop — keeps calling tools in circles, never deciding it’s done
• Oscillation — alternates between two states forever
• Hallucinated tool calls — keeps calling the same tool hoping for a better result
• Task decomposition explosion — keeps planning instead of acting

Guards against it

MAX_STEPS = 20
steps = 0

while True:
    if steps >= MAX_STEPS:
        return "Agent stopped: max steps reached"
    steps += 1
    # ... rest of loop

Other safeguards include timeouts, token budgets, tool call deduplication, and explicit system prompt instructions to prefer finishing over continuing.

The dangerous case isn’t an infinite loop per se — it’s an agent that takes many expensive or irreversible real-world actions before anyone notices it went off the rails. This is why limiting what tools an agent has access to is just as important as limiting how many steps it can take.

Agent code in JavaScript and Ruby

JavaScript

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

function searchWeb(query) {
  return `Results for: ${query}`;
}

const tools = [
  {
    name: "search_web",
    description: "Search the web for current information",
    input_schema: {
      type: "object",
      properties: {
        query: { type: "string", description: "Search query" }
      },
      required: ["query"]
    }
  }
];

async function runAgent(userMessage) {
  const messages = [{ role: "user", content: userMessage }];
  const MAX_STEPS = 20;
  let steps = 0;

  while (true) {
    if (steps >= MAX_STEPS) {
      console.log("Agent stopped: max steps reached");
      break;
    }
    steps++;

    const response = await client.messages.create({
      model: "claude-opus-4-5",
      max_tokens: 1000,
      tools,
      messages
    });

    if (response.stop_reason === "end_turn") {
      const finalMessage = response.content.find(b => b.type === "text");
      console.log("Agent answer:", finalMessage?.text);
      break;
    }

    if (response.stop_reason === "tool_use") {
      const toolCall = response.content.find(b => b.type === "tool_use");
      const result = searchWeb(toolCall.input.query);
      messages.push({ role: "assistant", content: response.content });
      messages.push({
        role: "user",
        content: [{ type: "tool_result", tool_use_id: toolCall.id, content: result }]
      });
    }
  }
}

runAgent("What is the latest news about AI?");

Ruby

require "anthropic"

client = Anthropic::Client.new

def search_web(query)
  "Results for: #{query}"
end

tools = [
  {
    name: "search_web",
    description: "Search the web for current information",
    input_schema: {
      type: "object",
      properties: {
        query: { type: "string", description: "Search query" }
      },
      required: ["query"]
    }
  }
]

def run_agent(client, tools, user_message)
  messages = [{ role: "user", content: user_message }]
  max_steps = 20
  steps = 0

  loop do
    if steps >= max_steps
      puts "Agent stopped: max steps reached"
      break
    end
    steps += 1

    response = client.messages(
      model: "claude-opus-4-5",
      max_tokens: 1000,
      tools: tools,
      messages: messages
    )

    if response["stop_reason"] == "end_turn"
      final = response["content"].find { |b| b["type"] == "text" }
      puts "Agent answer: #{final["text"]}"
      break
    end

    if response["stop_reason"] == "tool_use"
      tool_call = response["content"].find { |b| b["type"] == "tool_use" }
      result = search_web(tool_call["input"]["query"])
      messages << { role: "assistant", content: response["content"] }
      messages << {
        role: "user",
        content: [{ type: "tool_result", tool_use_id: tool_call["id"], content: result }]
      }
    end
  end
end

run_agent(client, tools, "What is the latest news about AI?")

Comparison

Using MCP instead of hardcoded tools

With hardcoded tools, you manually write the tool definition and the function that executes it. With MCP, the MCP server advertises its tools automatically and also executes them.

Hardcoded:                      MCP:
──────────────────              ──────────────────────────────
Agent defines tools             Agent discovers tools from server
Agent executes tools            Agent asks server to execute tools
Everything in one script        Tools live in separate processes

MCP servers communicate over either stdio (the agent spawns the server as a subprocess) or HTTP/SSE (the server runs separately and the agent connects over the network).

Python with MCP

import anthropic

client = anthropic.Anthropic()

mcp_servers = [
    {
        "type": "url",
        "url": "https://mcp.brave.com/sse",
        "name": "brave-search"
    }
]

messages = [{"role": "user", "content": "Search for the latest AI news"}]

while True:
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1000,
        messages=messages,
        mcp_servers=mcp_servers  # tools come from here, no tools= needed
    )

    if response.stop_reason == "end_turn":
        print(response.content[0].text)
        break

    messages.append({"role": "assistant", "content": response.content})

Local MCP server (stdio transport)

mcp_servers = [
    {
        "type": "stdio",
        "command": "npx",
        "args": [
            "@modelcontextprotocol/server-filesystem",
            "/home/user/documents"
        ],
        "name": "filesystem"
    }
]

Why MCP is powerful

• Reusability — the same MCP server can be used by any agent, any framework, any language
• Separation of concerns — tool logic lives in the server, agent logic in your script
• Ecosystem — growing library of pre-built MCP servers (GitHub, Slack, Gmail, databases, web search)
• Standardisation — tools built for one agent framework work with any other that supports MCP

It’s turning tools from code you write into services you connect to.

The role field in messages

The role field labels each message so the model can distinguish between different participants in the conversation.

The three roles

• user — Represents the human side: your messages, tool results fed back in, anything coming “from outside” the model.
• assistant — Represents the model’s own previous responses, shown to maintain coherence across turns.
• system — A special role with instructions to the model rather than part of the dialogue (personality, constraints, output format). In Anthropic’s API this is a separate top-level parameter.

Why history needs to be sent every time

The model has no memory between API calls. Every call is stateless. To simulate a conversation, you reconstruct the entire history each time:

# Turn 2 — you send the ENTIRE history:
messages = [
    {"role": "user",      "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Paris"},
    {"role": "user",      "content": "And what is its population?"}
]

The strict alternation rule

Messages must strictly alternate between user and assistant roles:

# ✅ Valid
[user, assistant, user, assistant, user]

# ❌ Invalid — two user messages in a row
[user, assistant, user, user, assistant]

In agents

Tool results go in user turns because conceptually they come from “outside” — the external world responding to the model’s action:

messages = [
    {"role": "user",      "content": "Search for AI news"},
    {"role": "assistant", "content": [
        {"type": "text",     "text": "I'll search for that"},
        {"type": "tool_use", "id": "123", "name": "search_web", "input": {...}}
    ]},
    {"role": "user",      "content": [
        {"type": "tool_result", "tool_use_id": "123", "content": "Here are the results..."}
    ]},
]