Inside an AI model: from tokens to reasoning — Erkan Malcok

When you send a sentence to an AI model, what happens between your words and its answer?

A useful starting point is:

Text → token IDs → numerical representations
     → attention and neural computation
     → next-token scores → select a token
     → append it to the sequence → repeat

This is a teaching diagram, not a complete blueprint for every model.

Keep three layers distinct: architecture describes the machinery; training adjusts its learned parameters; the application supplies interfaces, tools and memory. ChatGPT is an application. Architectural comparisons concern the particular model running inside it.

What you will learn

• Explain the roles of tokens, vectors and attention.
• Distinguish expert routing from additional reasoning computation.
• Separate parameter learning from adaptation within a conversation.
• Compare model disclosures without treating missing detail as evidence of absence.

The central idea

A language model generates text by repeatedly computing a distribution over possible next tokens and selecting one. Attention and other neural operations produce the representations used in that computation. Training determines the learned parameters; the current conversation supplies context.

Follow this basic process first. The optional release comparison later examines expert routing, context compression and connections between layers without treating disclosure as a performance ranking.

A worked example

From a sentence to representations

Consider:

I deposited money at the bank.

A tokeniser converts the text into integer identifiers. Tokens may represent words, fragments or punctuation. Their numerical identifiers do not express degrees of meaning.

An embedding maps each identifier to a learned vector. Individual dimensions should not be assumed to represent neat human concepts.

The model also represents token position. This helps distinguish “The dog chased the cat” from “The cat chased the dog”. Position may be incorporated into representations or attention calculations; implementations vary. The initial token embedding is the lookup vector before any position-dependent operations.

Suppose bank receives the same token ID in these sentences:

I deposited money at the bank.

I sat beside the river bank.

Its initial token embedding is the same, but later representations can differ with context.

Attention combines information from accessible token positions. Queries and keys determine weights; those weights combine value vectors. In causal attention, a position cannot attend to later positions.

For one simplified attention head, suppose the following three positions receive these invented weights and value vectors:

deposited: 0.5 × [2, 0]
money:     0.4 × [0, 3]
the:       0.1 × [1, 1]

Combined value:
0.5[2, 0] + 0.4[0, 3] + 0.1[1, 1] = [1.1, 1.3]

[1.1, 1.3] is one attention head’s intermediate output, not a meaning label or the final next-token prediction. The example assigns all the weight to three earlier positions for simplicity; real attention also has other accessible positions, including the current position. These numbers illustrate an operation, rather than measured behaviour inside a particular model. Vaswani et al. (2017) (opens in a new tab).

From representations to generated text

Across successive Transformer layers, attention mixes information between positions, while feed-forward networks transform each position’s representation. Residual connections carry representations forward by adding them to a sub-layer’s output; normalisation helps control their scale. Precise arrangements vary between architectures.

At the output, scores become probabilities over possible next tokens. A decoding procedure selects a token, adds it to the sequence and repeats until a stopping condition is reached. It may choose the highest-probability token or sample from a distribution. This is a conventional autoregressive teaching model, rather than a complete account of every serving implementation. Vaswani et al. (2017), sections 3.1–3.4 (opens in a new tab).

For the completed input “I deposited money at the bank.”, suppose the final position produces this invented distribution. Assume Today and Then are each single tokens here:

Greedy decoding selects Today, the highest-probability individual token, and appends it:

I deposited money at the bank. Today

The model now computes a new distribution using the extended sequence. These probabilities are not calculated from [1.1, 1.3]: between that intermediate head output and prediction sit further neural operations and the output projection. The table illustrates the final selection step, not a measurement of any released model.

What reasoning means here

In this article, reasoning means solving tasks that require several dependent steps. It is an operational description of task performance, not a claim about consciousness or proof of understanding.

Consider: “Three notebooks cost £4 each. You pay with £20. How much change should you receive?”

Cost:   3 × £4 = £12
Change: £20 − £12 = £8

Check the intermediate cost, then check that £12 + £8 = £20. The second calculation depends on the first. A generated explanation can still contain mistakes, so these checks matter; visible steps alone do not establish how a model computed its answer internally.

Training intended to improve reasoning changes learned parameters. Additional computation at inference gives a trained model more opportunity to work on a particular task. OpenAI reports improvements from both in its o1 research; they are distinct processes. OpenAI (2024b), introductory discussion and learning curves (opens in a new tab).

A common misconception

Routing is not reasoning

In a mixture of experts (MoE) model, a router selects expert networks used to process each token. The model also performs shared operations. MoE routing selects computation. Additional reasoning computation allows more intermediate work before answering. Neither mechanism, by itself, establishes better reasoning or generalisation.

Gemini’s Deep Think, OpenAI’s reasoning models and Claude’s supported thinking modes should therefore be compared using their documented behaviour and matched evaluation conditions. Their implementations need not be identical. OpenAI (2024b) (opens in a new tab), Anthropic (n.d.) (opens in a new tab), Google DeepMind (2026a) (opens in a new tab).

Learning is not only one training stage

Pre-training develops representations through prediction objectives. Supervised fine-tuning trains on example responses. Reinforcement learning adjusts parameters using a reward signal associated with generated responses. Reasoning training may form part of post-training, rather than a separate stage every model follows.

DeepSeek’s R1-Zero experiment began with an already pre-trained model and applied reinforcement learning without initial supervised fine-tuning. The report describes emerging reasoning behaviours, alongside readability problems and language mixing. Final R1 used a broader pipeline, including initial examples. DeepSeek-AI (2025b), section 2 (opens in a new tab).

OpenAI also documents reinforcement learning for reasoning. The defensible distinction is between particular training methods and their disclosure, rather than “one model learns while another merely predicts”.

During ordinary inference, the trained model generates a response using its current parameters. Additional thinking computation does not itself mean those parameters are being updated. OpenAI (2024b) (opens in a new tab).

Conversation adaptation is not parameter learning

Your corrections can influence subsequent responses because they enter the conversation context. That does not, by itself, update the model’s trained parameters.

Training:
Training material and feedback → parameter updates

Conversation:
Your examples and corrections → current context → adapted response

Application memory can retain information for later use. Retaining that information is different from retraining the underlying model.

Try it yourself

Compare these new sentences:

For baseball practice, she picked up the bat.

Inside the cave, we spotted the bat.

Assuming bat has the same token ID, what remains unchanged and what can change? Under causal self-attention, which token positions can the bat position attend to? Could words added after bat change its representation in this causal model? Would the contextual change persist into an unrelated conversation? Explain why.

A second system accepts spoken questions and reads its answers aloud. Does that establish that its language model directly processes audio? What evidence would you need?

Two further questions:

• Does activating fewer parameters establish better reasoning? Explain what evidence would be needed.
• A model card omits expert counts. What can you legitimately conclude?

Answer

The initial token embedding remains the same under that assumption. Position-dependent operations and attention can produce different representations. In causal self-attention, the bat position can attend to itself and preceding token positions, subject to any additional attention restrictions; it cannot attend to later positions. The clues “baseball practice” and “cave” both precede bat. Appending later words does not change the representation at that earlier position in this causal model. Vaswani et al. (2017), section 3.1 (opens in a new tab).

Those contextual representations belong to processing these inputs. They do not, by themselves, persist into an unrelated conversation or update the embedding parameters. A separate memory or training mechanism would be a different claim.

A voice interface is insufficient evidence of native audio processing: it could use separate speech recognition and text-to-speech systems. Look for model-specific documentation. OpenAI’s GPT-4o description explicitly distinguishes an earlier speech pipeline from joint text, vision and audio processing. OpenAI (2024a) (opens in a new tab).

Fewer active parameters describe computation, not established reasoning quality. Compare performance on matched, independently checkable tasks, controlling versions, tools and inference budgets.

If expert counts are omitted, conclude only that they are not specified in that source. This does not establish that the model lacks experts, nor that no other source discloses them.

Use it in practice

For architectural claims, record the exact release, source and disclosure limit. “Not specified in the cited source” does not mean absent.

For benchmark comparisons, match model versions, reasoning budgets, tool access and attempts. Use unfamiliar tasks with independently checkable answers.

For learning, evaluate the interaction:

Ask me to predict first. Give one hint at a time. After my attempt, identify the misconception and ask a changed question that checks my understanding.

For programming, work through prediction → explanation → execution → debugging → a changed example.

A convincing explanation is not proof of correctness. Judge the lesson by whether you can explain and solve the changed example independently.

How particular models differ — optional

The basic generation process is enough to follow the core lesson. This section adds release-specific mechanisms and a bounded comparison of public disclosures reviewed on 1 October 2026.

Caching previous computation

During conventional autoregressive decoding, a KV cache stores earlier tokens’ attention keys and values so the model can reuse them when computing later tokens. It avoids recomputing those values at every step; it is not a store of conversation memories shared across applications. DeepSeek’s V3 report discusses caching in its attention design. DeepSeek-AI (2025a), section 2.1.1 (opens in a new tab).

These representations are model-specific. KV-Lingo studies learned translators between selected frozen model pairs, rather than reusing one model’s cache unchanged in another. Treat this as a separate research direction, not a universal serving feature. Castin et al. (2026), abstract (opens in a new tab).

Expert routing, context and connections

In a mixture of experts (MoE) model, a router selects expert networks for each token. Those experts perform part of the neural computation; the model also has shared operations. Active parameter counts therefore describe more than an expert count.

DeepSeek’s reports give concrete examples of three different design choices:

• Multi-head Latent Attention (MLA) in V3 compresses key/value representations into a smaller latent representation, reducing what must be cached. This is not the same as discarding whole token positions. DeepSeek-AI (2025a), section 2.1.1 (opens in a new tab).
• Compressed Sparse Attention (CSA) in V4 compresses the cache along the sequence dimension and applies sparse attention. Heavily Compressed Attention (HCA) uses heavier compression with dense attention over the compressed representations. DeepSeek-AI (2026a), p. 3, Model properties (opens in a new tab).
• Manifold-constrained hyper-connections (mHC) in V4 constrain residual mixing to stabilise information flow between layers. They address a different problem from selecting experts or compressing context. DeepSeek-AI (2026a), p. 3, Model properties (opens in a new tab).

Expert routing selects computation. Context compression changes stored representations. Connections between layers govern information flow.

Selected public disclosures by release

Each row concerns a named release and the specified source, not every model from its provider. Disclosure does not independently establish effectiveness. “Not specified in the cited source” describes a limit of this review, not proof that a mechanism is absent or undisclosed everywhere.

• OpenAI (2023), section 2, Scope and Limitations (opens in a new tab)
• OpenAI (2024a), opening description and Model capabilities (opens in a new tab)
• OpenAI (2024b), introductory discussion and learning curves (opens in a new tab)
• Anthropic (2026), Claude Opus 4.6 Summary Table: Modalities and Model architecture and training methodology (opens in a new tab)
• Google DeepMind (2026a), pp. 2–3, Model Information and Model Data (opens in a new tab)
• Google DeepMind (2026b), Model Information: Inputs, Outputs and Architecture (opens in a new tab)
• DeepSeek-AI (2025a), abstract, sections 2.1.1–2.1.2 and 5 (opens in a new tab)
• DeepSeek-AI (2025b), section 2 (opens in a new tab)
• DeepSeek-AI (2026a), p. 3, Model properties (opens in a new tab)
• DeepSeek-AI (2026b), introductory release description (opens in a new tab)

Related reading

• LLM fluency is not reasoning
• Fluent answers are not understanding

Optional further reading

Cheng et al. (2026), Conditional Memory via Scalable Lookup (opens in a new tab) introduces Engram: learned lookup for local token patterns alongside neural computation. This extends the architectural discussion beyond the introductory pipeline. Its inclusion must be verified separately for each released model; it is not application conversation memory.