What is Attention?

Think of attention as spotlighting the right words in a sentence.
For example, take this sentence:

"The cat was being chased by the dogs."

If someone asks: "Who was chasing?"
You instantly focus on “dogs”. That’s attention — focusing on the relevant part of the sentence to make sense of it.

Attention uses 3 characters to compute the “attention” sort thing. Those 3 things are Queries, Values , Keys

• Query (Q) - Its like a question each word asks as in “Who should I pay more attention to understand myself better ?“

• Key (K) - Its like a tag each word carries about what it offers , per say “Here is the info i hold.”

• Value (V) - Its like the actual information that can be shared as in “Here’s my content, if u need it”

Explained here with a simple Example !

Let’s use the sentence:

“The cat sat on the mat.”

Focus on the word “sat”.
Now imagine “sat” becomes a Query. It asks,

“Who can help me understand what I mean here?”

All the other words — “The”, “cat”, “on”, “the”, “mat” — are turned into Key + Value pairs.

Now, “sat” will:

1. Compare its Query with the Keys of all other words (using dot product).

2. Get attention scores: Higher if the Key matches the Query.

3. Pass the scores through Softmax — to turn them into weights like:

4. Use these weights to mix the Values of those words — giving “sat” a context-aware meaning.

cat -> 70%
mat -> 20%
others -> 10%

Why is it called Self-Attention ?

Because every other word goes through this process - comparing itself with all others (including itself). This mutual attention gives the model a deep layered understanding of how words relate.

That’s the core idea behind self-attention — the building block of Transformers.

After all language is more than just words, right !
Its also about structure, order and context. That’s where the Positional Encoding comes in.

🧠Why Transformers need Positional Encoding

Transformers are designed to look up at all words at once. This makes it incredibly powerful and but also makes them blind to word order unless we do something about it.

Imagine this sentence below

The cat chased the mouse

Now scramble the words

Mouse the cat chased the

Both sentences are created via same words , but one is meaningful and other one is meaningless and makes no sense .
Without knowing who comes first ,the model won’t understand the message.

🧩What is Positional Encoding

Positional encoding is technique to inject order while processing text in parallel by model.
Its like whispering to the model :

“Hey, this word is the first, this one’s the second, this one’s near the verb, and this one’s at the end.”

It does not change the words, just adds the sense of position to each one.

Let’s talk about Context

Consider the word “bank”.

• He sat by the bank and watched the sunset.

• He deposited his cash at the bank.

Same word ,but different meaning right !

How does the model figure this out ?

The word bank in each sentence is surrounded by different neighboring words, in different positions. Positional encoding helps GPT combine :

• Where the word is

• What’s around it

This gives the contextual meaning to the model.

📦Types of Positional Encoding (No Math, Promise)

• Sinusoidal Encoding : Adds a pattern to each word based on its position — kind of like each word dancing to a different rhythm. This helps the model to feel the “beat” of every sentence.

• Learnable Encoding : The model learns what position feels like. (Used in modern models)

🧠Final Takeaway

Positional Encoding gives words a sense of place — like a tag saying “I’m first,” “I come after the subject,” or “I’m the closing word.”

🧩 Step 1: Breaking Down the Prompt

When you type a prompt, the first thing that happens is the text is broken into smaller pieces. This isn't just for fun — it helps the model:

• Handle longer inputs

• Understand unseen or misspelled words

• Maintain flexibility with different word forms

These smaller units are called tokens, or more technically, subword units. This tokenization step is handled by a separate program called a tokenizer, which uses a variety of algorithms to break down words based on different rules and contexts.

If you're curious about how tokenizers work in detail, here’s a great resource by HuggingFace given below:

Tokenizer in Transformers

🔄 Step 2: Tokens → IDs → Embeddings

Once the text is tokenized, each token is mapped to a unique number called a token ID, based on the model’s internal vocabulary.
You can think of this vocabulary as a giant lookup table that maps each known token to a number.

These token IDs are then converted into high-dimensional vectors using something called an embedding matrix. These vectors — called embeddings — represent the tokens in a format that the neural network can understand and work with.

🧠 Step 3: From Generic to Contextual Meaning

nitially, token embeddings are context-independent — they mean the same across different sentences. For example:

• S1: Bird flies high

• S2: I am high

The word “high” would have the same embedding in both cases, even though the meaning clearly changes based on context.

That’s where the real magic begins.

Through self-attention and feedforward layers techniques inside the Transformer, the model begins to refine these embeddings. It learns how each token relates to others in the sentence — and updates the embeddings to reflect the true, contextual meaning.

🧬 Step 4: Deep Understanding

These refined embeddings are passed through multiple layers of the neural network, each extracting increasingly complex and abstract features. As the data flows through the layers, the model uncovers:

• Sentence structure (syntax)

• Emotions or tone (sentiment)

• Intent, reasoning, and relationships

All of this happens in a fraction of a second — resulting in the model generating coherent and meaningful responses.

🔚 Wrapping Up

So next time you enter a prompt, remember: behind the scenes, it's not just "text in, text out."

What starts as a simple sentence turns into a rich, high-dimensional representation that lets GPT understand you with remarkable intelligence.

Stay tuned for my next blog on Positional Encoding — how Transformers keep track of word order without using recurrence.

Until then, take care and peace ✌️