An embedding maps a token ID to a learned numerical vector. These coordinates let the network operate on discrete text, but individual dimensions do not have reliable human-readable labels.
A token ID chooses one row from a learned embedding matrix. DistilGPT-2 stores 768 values in that row, then transformer layers mix information across positions.
From token IDs to a table of numbers
A tokenizer returns an integer for each token. The model cannot multiply words directly, so an embedding layer retrieves a row from a matrix of trained parameters. The values of that row were learned during model training rather than manually assigned by a dictionary writer.
For DistilGPT-2, the original token embedding matrix has 50,257 rows and 768 columns. Looking up one token returns exactly 768 floating-point coordinates. A token ID is not the same thing as its 768-number vector.
Static embeddings and contextual representations are different
A static token embedding is always the same row when the same ID is selected from an unchanged checkpoint. Once the transformer combines positions through attention and other layers, the resulting hidden state can change depending on the surrounding words.
The word 'bank' in a sentence about money and the word 'bank' in a sentence about a river need contextual processing to help distinguish the use. You cannot infer a single fixed definition from one coordinate of its initial embedding.
Comparing vectors with cosine similarity
Cosine similarity measures the angle between two nonzero vectors: their dot product divided by the product of their lengths. A score near +1 means the vectors point in a similar direction, around 0 means their directions are approximately perpendicular, and near −1 means they point in opposite directions.
That geometric relationship is not a verified synonym detector. Two token embeddings can be close for reasons that are not a straightforward semantic relationship, particularly without considering context and the model's training objective.
Inspect the original checkpoint without downloading everything
Our viewer reads the chosen token's original FP32 embedding row from the public DistilGPT-2 safetensors checkpoint using validated HTTP byte-range requests. It can display all 768 coordinates and compare two selected vectors.
The browser inference model uses quantized ONNX weights, so its internal values may differ from those original 32-bit floats. The viewer labels this precision difference rather than claiming to show the exact runtime bytes.
Two numbers aren't enough to describe the model
Token ID → [0.21, −0.48, 0.03, …] (illustrative coordinates)
The full DistilGPT-2 lookup contains 768 learned values. You can inspect the actual values by choosing a token in the real model lab.
What this does not prove: The three written example coordinates are made up for explanation; only the connected inspector returns checkpoint values.
Compare real embedding vectors ↗Questions and clear answers
Does every embedding number correspond to a concept?
No. Coordinates are learned together. You should not label an individual dimension 'happiness' or 'truth' without specific, independently validated evidence.
Are embeddings generated from scratch on every prompt?
The pretrained lookup table remains fixed during ordinary inference; context-dependent hidden states are newly computed from the particular input.
Are the website's displayed weights real?
Yes, when the checkpoint lookup succeeds, the inspector reads original FP32 token embedding values. Its educational placeholders are separately labeled as illustrative.
Research & verificationLearned vector representations
Publication review: Peer-reviewed conference publication; relevant to vector representations, not an audit of these particular GPT-2 weights.
Independent verification: We read actual FP32 token-embedding rows from the publicly hosted DistilGPT-2 checkpoint using byte ranges. No third party has validated this site's extraction procedure.
Original weight file (public metadata) ↗Peer review of a paper does not certify this demo.