Vector Embeddings: How Computers Understand the Meaning of Words and Concepts

1. Architectural Foundations & Mechanics

For decades, traditional computer systems handled human language using discrete symbolic representations. In a classic database or early search engine, the word "physician" might be assigned the identifier 4012, while "doctor" was assigned 9821. To a relational database or a string parser, these numbers were arbitrary labels. The system had no innate mechanism to deduce that an individual searching for a "pediatric physician" would find "children's doctor" completely relevant.

The Failure of Sparse Categorical Encoding

Early computational linguistics attempted to bridge this gap using one-hot encoding. In a dictionary of size $|V|$, every term is represented by a vector of length $|V|$ containing all zeros, save for a single 1 at the word's designated index. While straightforward, one-hot vectors suffer from two catastrophic structural flaws:

1. Extreme Dimensional Sparsity & Memory Waste: If an enterprise dictionary contains 100,000 unique terms, every single word requires a 100,000-dimensional vector consisting of 99,999 zeros and a single one. Storing these vectors requires massive memory allocations for data structures that carry almost no information per byte.
2. Mathematical Orthogonality: In linear algebra, two vectors whose dot product equals zero are perpendicular, or orthogonal: $\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^{|V|} u_i v_i = 0$ Because every one-hot vector has its single 1 at a distinct index, the inner product between any two distinct words is always precisely zero. To an algorithm processing one-hot representations, "cat" is mathematically as distant from "feline" as it is from "bulldozer" or "quantum". There is zero geometric overlap, zero proximity, and no inherent concept of meaning.

Dense Embeddings: Geometry as Semantics

Dense vector embeddings eliminate this orthogonality by projecting vocabulary items into a continuous, lower-dimensional space—typically between $d = 256$ and $d = 4096$ dimensions. Rather than filling thousands of slots with zeros, every single dimension in a dense embedding holds a continuous floating-point number (e.g., in 16-bit brain floating point, bfloat16, or 32-bit single precision, fp32):

"doctor" → [0.412, -0.189, 0.821, -0.045, 0.612, …, 0.093]
"physician" → [0.408, -0.195, 0.814, -0.052, 0.605, …, 0.089]

By placing these numbers along continuous axes, similar concepts end up with similar values across many dimensions. The fundamental breakthrough of modern representation learning—originating with distributional hypothesis frameworks like Word2Vec and GloVe, and evolving into transformer architectures—is that words that appear in similar linguistic contexts occupy nearby points in vector space.

Furthermore, these dense spaces exhibit remarkable spatial properties: conceptual relationships correspond to directional vectors. In well-trained embedding spaces, the spatial vector pointing from "man" to "woman" is virtually parallel to the vector pointing from "king" to "queen". Calculating arithmetic over these vectors translates directly into reasoning over concepts:

&Vector(king) - &Vector(man) + &Vector(woman) ≈ &Vector(queen)

Static vs. Contextual Embeddings: The Transformer Evolution

Early dense vector architectures were static: each word in the vocabulary possessed exactly one fixed vector stored in an embedding matrix $W_E \in \mathbb{R}^{|V| \times d}$. This created a major linguistic stumbling block known as polysemy. For instance, consider the word "bank" in two different sentences:

• "The fisherman sat along the river bank."
• "The customer deposited money into the bank."

In a static model, the word "bank" was forced to occupy a single average coordinate in vector space, muddling geography and finance together. Modern transformer models resolve this through multi-head self-attention mechanisms. The model begins with an initial lookup from the base embedding matrix $W_E$, adds positional encodings, and then passes the vectors through stacked transformer layers. Within each layer, attention heads dynamically weigh each word against every other word in the sentence:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

By the time the representation reaches the final layer, the contextual embedding for "bank" has been mathematically steered toward geological concepts in the first sentence and financial institutions in the second.

Silicon Reality: Hardware Memory Hierarchies & Kernel Launches

While the mathematics of vector embeddings involves elegant geometry, executing these operations in production involves navigating physical silicon constraints. Modern machine learning inference pipelines run on hardware accelerators such as NVIDIA Hopper or Blackwell GPUs and Google TPUs. Understanding how vector embeddings move through these chips is essential to operating them efficiently.

A modern GPU contains a strict hierarchy of memory tiers, each with distinct latency and bandwidth profiles:

• SRAM / Shared Memory / Registers (On-Chip): Blazing speed (>15 TB/s aggregate bandwidth), ultra-low latency (<10 nanoseconds), but severely limited capacity (often tens to hundreds of megabytes per chip).
• High Bandwidth Memory (HBM3 / HBM3e, On-Package): High capacity (80 GB to 192 GB per GPU), wide bus width, high throughput (2.0 to 4.8 TB/s), but substantially higher latency (~100 to 200 nanoseconds).
• Host RAM / PCIe System Memory: Massive capacity (hundreds of gigabytes to terabytes), but comparatively low bandwidth (PCIe Gen 5 provides ~64 GB/s unidirectional per 16x slot), with transfer latencies measured in microseconds.

During an embedding lookup, the CPU or GPU kernel receives a batch of token IDs and reads corresponding rows from the large embedding table $W_E$. When batches are small or queries arrive sequentially, reading these disparate rows from memory involves random access across high-bandwidth memory.

Furthermore, in high-throughput serving systems, individual CUDA kernel launches create non-trivial CPU driver overhead. When launching thousands of small memory lookups and matrix multiplications per second, the time taken by the CPU to serialize and schedule commands into the GPU command queue can exceed the actual execution time on the tensor cores. To combat this serialization latency, modern inference runtimes capture static sequences of operations into CUDA execution graphs (cudaGraph_t). By instantiating the entire compute and lookup topology into a single launchable graph, the host eliminates CPU driver round-trips and executes embedding projections directly on the accelerator.

2. Mathematical Modeling & Resource Bounds

Operating vector embedding systems at production scale requires strict mathematical modeling of geometric proximity, memory footprints, arithmetic intensity, and query entropy dynamics.

Geometric Similarity Metrics

Once concepts are mapped into vectors, systems measure conceptual similarity using one of three standard distance metrics. Let $\mathbf{u}, \mathbf{v} \in \mathbb{R}^d$ represent two dense vectors:

1. Cosine Similarity: Measures the cosine of the angle between two directional vectors, completely ignoring differences in vector length. It ranges from $-1$ (opposite direction) through $0$ (orthogonal) to $+1$ (identical direction): $\cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|_2 \|\mathbf{v}\|_2} = \frac{\sum_{i=1}^d u_i v_i}{\sqrt{\sum_{i=1}^d u_i^2} \sqrt{\sum_{i=1}^d v_i^2}}$ If vectors are strictly pre-normalized during inference such that $\|\mathbf{u}\|_2 = 1.0$, cosine similarity simplifies directly to the inner product: $\cos(\theta) = \mathbf{u} \cdot \mathbf{v}$.
2. Dot Product (Inner Product): Retains both angular orientation and magnitude. Used extensively in retrieval systems where vector magnitude encodes term importance, frequency, or confidence: $\langle \mathbf{u}, \mathbf{v} \rangle = \sum_{i=1}^d u_i v_i$
3. Euclidean Distance ($L_2$ Norm): Measures the physical straight-line distance between two vector coordinates in the $d$-dimensional space: $D_{L_2}(\mathbf{u}, \mathbf{v}) = \|\mathbf{u} - \mathbf{v}\|_2 = \sqrt{\sum_{i=1}^d (u_i - v_i)^2}$

Arithmetic Intensity and the Roofline Model

A central metric in systems performance engineering is Arithmetic Intensity ($\mathcal{I}$), defined as the ratio of computational work (floating-point operations, or FLOPs) performed per byte of data transferred across the memory bus:

Under the Williams et al. Roofline Model, hardware performance is bounded by two distinct ceilings:

Where $P_{\text{peak}}$ represents the hardware's theoretical compute ceiling (e.g., 989 TFLOPS of FP16/BF16 tensor core compute on an NVIDIA H100 SXM5), and $B_{\text{mem}}$ represents the physical memory bandwidth (3.35 TB/s on an H100).

The "knee" or ridge point where a workload transitions from memory-bandwidth-bound to compute-bound is calculated as:

Embedding operations and autoregressive key-value lookups involve transferring hundreds of thousands of weight bytes while performing only a minimal number of multiplications and additions per parameter. With typical arithmetic intensities hovering between $1$ and $15 \text{ FLOPs/Byte}$, embedding generation and decoding phases operate deep within the memory-bound regime. Accelerating these workloads requires minimizing memory traffic and maximizing cache reuse rather than adding raw tensor math capacity.

Resource Formulations: Embedding Tables and Key-Value Memory

To provision memory accurately on host and accelerator devices, engineers calculate exact byte consumption for both static embedding layers and dynamic sequence caches.

For a static vocabulary of size $|V|$ projected into a vector space of dimension $d$ with precision byte size $b_{\text{prec}}$ (where FP32 = 4 bytes, FP16/BF16 = 2 bytes, INT8 = 1 byte):

For example, a vocabulary $|V| = 128,000$ with an embedding dimension $d = 4096$ stored in 16-bit precision requires:

When embedding vectors enter transformer attention blocks across long conversations, the Key-Value (KV) cache memory dominates accelerator capacity. For a transformer containing $L$ layers, with key-value head count $n_{\text{heads}}$, head dimension $d_{\text{head}}$, batch size $B$, and sequence length $S$:

The factor of $2$ accounts for storing separate tensors for Keys and Values. For a model with 32 layers, 8 key-value heads, head dimension 128, processing a batch of 16 requests at sequence length 4,096 in 16-bit precision:

Entropy Dynamics: Low-Entropy vs. High-Entropy Divergence

Modern production pipelines increasingly use speculative execution (draft-then-verify architectures) to accelerate vector generation and token processing. A smaller "draft" embedding model predicts $K$ candidate tokens in parallel, and a larger "target" model evaluates all $K$ candidates in a single parallel verification pass, eliminating sequential round-trips to memory.

However, the practical throughput of these pipelines diverges heavily based on the Shannon Entropy ($\mathcal{H}$) of the query distribution:

If a query has low entropy (e.g., standard code synthesis, structured JSON formatting, or common boilerplate phrases), the token probability distribution is sharp. The speculative draft acceptance rate $\alpha$ consistently exceeds $0.85$ to $0.95$. The expected speedup ratio $\mathbb{E}[S]$ scales according to:

Where $\gamma$ is the number of speculative tokens evaluated per pass and $c$ is the ratio of draft verification cost to target forward pass cost. Under high acceptance ($\alpha \to 0.9$), systems regularly realize $2.5\times$ to $3.2\times$ real-world speedups.

Conversely, during high-entropy queries (creative narrative, out-of-distribution reasoning, diverse foreign vocabularies), probability distributions flatten. The draft acceptance rate drops to $\alpha < 0.40$. Because the target model must repeatedly reject candidate vectors and fall back to sequential generation, speculative overhead actually decreases overall system throughput below baseline, introducing severe tail-latency spikes ($P_{99}$). Production deployments must employ dynamic admission control based on entropy metrics to gate speculative verification on and off.

3. Memory Allocation Strategies: Contiguous vs. Segmented Blocks

How vector embeddings and their associated context tensors are stored in physical GPU DRAM directly governs system stability. Early inference engines allocated memory using monolithic, contiguous buffers sized for the theoretical maximum sequence length of each request. Modern enterprise systems instead employ segmented block allocation (often implemented via paged memory managers like PagedAttention).