Modern GPU Architecture: How Thousands of Tiny Cores Accelerate Parallel Workloads

Executive Summary & Key Takeaways

A central processing unit (CPU) is engineered like an Olympic sprinter operating an agile delivery van: it features a handful of complex cores with massive cache hierarchies, branch predictors, and out-of-order execution pipelines designed to race through intricate, sequential decision trees with minimal latency. A graphics processing unit (GPU), by contrast, operates like a transcontinental cargo rail network. It does not attempt to steer through winding city side streets at breakneck speeds. Instead, it deploys thousands of smaller, synchronized arithmetic logic units (ALUs) working along parallel tracks. When moving massive volumes of uniform data—such as matrix multiplications in machine learning models or pixel shaders in graphics pipelines—this throughput-oriented architecture outpaces sequential processors by orders of magnitude.

However, harnessing thousands of concurrent execution lanes exposes a fundamental physical bottleneck: raw computational throughput regularly outstrips the memory bus. In modern workloads such as large language model inference, performance is rarely bound by arithmetic units alone. Memory bandwidth saturation, warp divergence, and allocation fragmentation dictate real-world latency. By inspecting how hardware components—such as Streaming Multiprocessors, warps, high-bandwidth memory (HBM), and software memory managers—interact under varying operational entropy, engineers can design systems that maximize hardware saturation without triggering catastrophic tail-latency spikes.

• Throughput Over Latency: Modern GPUs trade deep single-thread execution hardware (speculative branch predictors, massive reorder buffers) for thousands of lightweight ALUs managed by hardware warp schedulers that hide memory stalls through zero-overhead context switching.
• The Memory Bandwidth Wall: Autoregressive inference and sparse tensor kernels frequently operate at low arithmetic intensity (FLOPs per byte), causing compute units to sit idle while awaiting data from high-bandwidth memory (HBM).
• Speculative Acceleration: Batched candidate verification (speculative decoding) shifts memory-bound token generation into compute-bound matrix operations, loading model weights once to evaluate multiple sequence steps in parallel.
• Entropy-Driven Divergence: High-entropy generation targets degrade speculative acceptance rates, triggering warp branch divergence, execution rollback overhead, and uncoalesced memory reads that can drop execution speed below standard non-speculative baselines.
• Segmented Allocation Eliminates OOM: Paged, non-contiguous block allocation algorithms (like PagedAttention) eliminate external memory fragmentation and over-allocation penalties, enabling predictable high-concurrency throughput.

1. Architectural Foundations & Mechanics

To understand why modern graphics processors excel at large-scale parallel processing, one must first contrast their silicon real estate allocation with traditional general-purpose CPUs. A high-end server CPU allocates roughly 70% to 80% of its die area to control logic, out-of-order execution windows, branch predictors, and multi-megabyte L2 and L3 caches. The CPU's primary design goal is minimizing instruction latency for a single thread of execution. It spends silicon transistors anticipating which instructions will run next, reordering them to avoid pipeline stalls, and caching memory references close to the execution cores.

A modern GPU discards almost all of this speculative complexity. Because parallel workloads exhibit high data parallelism—meaning the exact same operation must be applied to millions of distinct data points—the GPU dedicates the vast majority of its silicon area directly to floating-point and integer ALUs. Latency is not hidden through deep predictive caching; it is hidden through extreme concurrency.

The Streaming Multiprocessor (SM) and Warp Scheduling

The fundamental building block of an NVIDIA GPU is the Streaming Multiprocessor (SM). In modern architectures such as Hopper (GH100) or Ada Lovelace (AD102), a GPU contains between 128 and 144 SMs. Each SM contains:

• Multiple sub-cores (processing blocks), each equipped with its own instruction dispatch unit and warp scheduler.
• Distinct sets of execution units: FP32 cores, FP64 cores, INT32 cores, and specialized Tensor Cores designed for mixed-precision matrix multiply-accumulate operations.
• A massive physical Register File (typically 64K 32-bit registers per SM, totaling over 256 KB).
• A high-speed unified L1 Data Cache / Shared Memory scratchpad (typically 128 KB to 228 KB per SM).

GPUs execute instructions using the Single Instruction, Multiple Threads (SIMT) execution model. Threads are not managed individually by hardware schedulers. Instead, the hardware groups 32 parallel threads into a basic execution unit known as a warp.

When a GPU kernel runs, the warp scheduler issues a single instruction across all 32 lanes of the warp simultaneously. If an instruction requires a memory read from high-bandwidth device memory (taking 200 to 400 clock cycles to traverse the crossbar and memory controllers), the warp scheduler does not stall the execution pipeline. Because the thread states and register allocations for dozens of active warps are held statically in the massive register file, the scheduler switches execution context to an alternate, ready warp in a single clock cycle with zero register spilling overhead.

Memory Hierarchy and Coalescing

The GPU memory hierarchy consists of distinct tiers, each featuring trade-offs between capacity, latency, and throughput:

• Registers: Located directly inside the ALU pipelines. Throughput exceeds 30 TB/s aggregate per chip with sub-cycle access latency, but register capacity is strictly partitioned across resident warps.
• Shared Memory (SRAM): Software-managed scratchpad memory shared across all threads within a single thread block. Latency is roughly 20 to 30 clock cycles, providing upwards of 10 to 15 TB/s aggregate bandwidth.
• L2 Cache: A centralized cache (typically 50 MB to 96 MB on modern datacenter chips) shared across all SMs. It services requests that miss the local L1 cache with latency ranging from 60 to 100 cycles.
• High-Bandwidth Memory (HBM3/HBM3e / GDDR6): Off-chip dynamic RAM connected via silicon interposers and wide bus interfaces (e.g., 5120-bit wide memory interfaces in H100). Bandwidth reaches 3.35 TB/s to 4.8 TB/s, but access latencies span 200 to 400 clock cycles.

To achieve high throughput across off-chip memory buses, GPUs enforce memory coalescing. Memory controllers service global memory requests in aligned 32-byte or 128-byte transaction segments. When the 32 threads of a warp access 32 consecutive 4-byte floating-point numbers within an aligned 128-byte region, the hardware merges those 32 separate logical reads into a single 128-byte physical transaction.

Conversely, if threads calculate stride patterns or non-contiguous addresses that point to distinct, unaligned memory sectors, the memory subsystem must issue up to 32 independent memory transactions to satisfy a single instruction. This uncoalesced memory access pattern degrades effective memory bus efficiency to a small fraction of peak theoretical bandwidth.

Warp Divergence: The Cost of Branching

Because all 32 threads in a warp share a single instruction issue pipeline, conditional branching (`if/else` control flow) requires careful handling. If thread 0 through 15 evaluate a branch condition as `true`, while threads 16 through 31 evaluate it as `false`, the warp experiences warp divergence.

The hardware handles this divergence by serializing the execution paths. It executes the `true` path first, activating an execution mask that disables writebacks from lanes 16 through 31. Once that branch completes, the hardware switches the execution mask to disable lanes 0 through 15 and executes the `false` path. Total execution latency becomes the sum of both paths rather than the maximum of the two, cutting arithmetic execution efficiency by half.

CUDA Graphs: Mitigating Host Launch Overhead

When executing thousands of rapid, fine-grained kernels—such as verification checks during speculative token generation—the host CPU quickly becomes the system bottleneck. Each traditional CUDA kernel launch (`cudaLaunchKernel`) requires the CPU driver to format the launch parameters, push them through the OS driver stack, and write to the GPU's hardware command queue via PCI Express. This process introduces 3 to 10 microseconds of CPU serialization latency per kernel.

CUDA Execution Graphs address this host overhead by shifting execution orchestration directly onto the device. The application records an entire directed acyclic graph (DAG) of kernels, memory copies, and synchronization events once. The driver validates the graph and instantiates it into a static device-side execution schedule. Subsequent runs require only a single CPU doorbell write (`cudaGraphLaunch`), reducing host CPU launch overhead from dozens of microseconds down to sub-microsecond levels.

2. Mathematical Modeling & Resource Bounds

Understanding GPU workload acceleration requires modeling the relationship between floating-point computation and memory transfer rates. The primary analytical tool for this analysis is the Williams Roofline Model.

The Roofline Model & Operational Intensity

The operational intensity ($I$), expressed in FLOPs per byte, measures the number of floating-point operations performed for every byte transferred from global device memory:

$I = \frac{\text{Work (FLOPs)}}{\text{Data Movement (Bytes)}}$

A GPU's operational ceiling is defined by two hardware constants:

1. Peak Compute Throughput ($P_{\text{compute}}$), measured in TFLOPS ($10^{12}$ operations/second).
2. Peak Memory Bandwidth ($B_{\text{memory}}$), measured in TB/s ($10^{12}$ bytes/second).

The hardware ridge point ($I_{\text{ridge}}$) defines the exact threshold where an application transitions from being memory-bandwidth-bound to compute-bound:

$I_{\text{ridge}} = \frac{P_{\text{compute}}}{B_{\text{memory}}}$

Consider an NVIDIA H100 SXM5 GPU operating on 16-bit floating-point (FP16) tensor operations without structural sparsity:

• $P_{\text{compute}} = 989.4\text{ TFLOPS} = 989.4 \times 10^{12}\text{ FLOPs/s}$
• $B_{\text{memory}} = 3.35\text{ TB/s} = 3.35 \times 10^{12}\text{ Bytes/s}$

Calculating the ridge point:

$I_{\text{ridge}} = \frac{989.4 \times 10^{12}}{3.35 \times 10^{12}} \approx 295.34\text{ FLOPs/byte}$

If a kernel executes fewer than $295.34$ mathematical operations for each byte loaded from off-chip HBM3 memory, the execution units spend clock cycles stalled on the memory crossbar. The attainable throughput ($T_{\text{attainable}}$) is strictly bounded by the memory subsystem:

$T_{\text{attainable}} = \min\left(P_{\text{compute}},\, I \times B_{\text{memory}}\right)$

The Autoregressive Memory Wall vs. Batched Verification

During standard autoregressive large language model decoding with a batch size of $B=1$, generating a single token requires reading all parameter weights $W$ from global memory into the SM register files to perform a matrix-vector product (GEMV). For a 70-billion parameter model stored in 16-bit precision:

• $\text{Model Size} = 70 \times 10^9 \text{ parameters} \times 2 \text{ bytes} = 140 \times 10^9 \text{ bytes} \approx 140\text{ GB}$
• $\text{Compute per Token} \approx 2 \times 70 \times 10^9 = 140 \times 10^9\text{ FLOPs}$
• $\text{Operational Intensity } I = \frac{140 \times 10^9\text{ FLOPs}}{140 \times 10^9\text{ Bytes}} = 1.0\text{ FLOP/byte}$

Because $I = 1.0 \ll 295.34$, standard autoregressive decoding utilizes less than 0.35% of the H100's tensor compute capacity. Theoretical token generation time is completely dominated by the time required to stream weights from memory:

$t_{\text{step}} = \frac{140\text{ GB}}{3.35\text{ TB/s}} \approx 41.79\text{ milliseconds}$

Speculative execution changes this balance. A small draft model (or candidate generator) predicts a batch of $K$ candidate tokens ahead. The large target model then verifies all $K$ tokens simultaneously in a single forward pass. Rather than running $K$ sequential matrix-vector operations, the GPU executes a matrix-matrix multiplication (GEMM) of dimension $[M \times K] \times [K \times N]$:

$I_{\text{speculative}} \approx K \times I_{\text{autoregressive}} = K \times 1.0\text{ FLOPs/byte}$

By verifying $K=5$ tokens in parallel, the GPU streams the 140 GB of model weights across the memory bus once, multiplying operational intensity by $5\times$ without increasing weight-read memory bandwidth consumption.

KV Cache Memory Consumption Formulation

The Key-Value (KV) cache preserves past attention states to prevent recomputing attention keys and values for preceding tokens. For a multi-head transformer with $L$ layers, $N_{\text{heads}}$ key-value heads, head dimension $D_{\text{head}}$, batch size $B$, sequence length $P$, and data precision size $S_{\text{bytes}}$:

$\text{Bytes}_{\text{KV}} = 2 \times B \times L \times N_{\text{heads}} \times D_{\text{head}} \times P \times S_{\text{bytes}}$

The multiplier $2$ accounts for separate storage of Key and Value projection matrices. On an 8-GPU tensor-parallel cluster running a 70B model ($L=80$, $N_{\text{heads}}=8$ per GPU under Grouped-Query Attention, $D_{\text{head}}=128$, $S_{\text{bytes}}=2$ for FP16), a sequence of length $P=4096$ with batch size $B=16$ requires:

$\text{Bytes}_{\text{KV}} = 2 \times 16 \times 80 \times 8 \times 128 \times 4096 \times 2 = 21,474,836,480\text{ bytes} \approx 21.47\text{ GB per GPU}$

Entropy-Driven Throughput Divergence

Speculative verification acceleration depends on the probability distribution entropy of the target generation task. Let the token sequence distribution be $X$, with Shannon entropy defined as:

$H(X) = -\sum_{i=1}^{V} P(x_i) \log_2 P(x_i)$

Where $V$ is vocabulary size. The empirical token acceptance rate $\alpha \in [0, 1]$ represents the probability that the large target model confirms a candidate token proposed by the speculative mechanism:

• Low-Entropy Regime ($H(X) \to 0$): Observed in structured code generation, predictable JSON formatting, and deterministic factual queries. The draft mechanism achieves high acceptance rates ($\alpha \ge 0.85$). The expected number of accepted tokens $\mathbb{E}[N]$ per verification round with speculation depth $K$ follows:

  $\mathbb{E}[N] = \frac{1 - \alpha^{K+1}}{1 - \alpha}$

  When $\alpha = 0.90$ and $K=5$, $\mathbb{E}[N] \approx 4.68$ tokens accepted per single model weights memory sweep.
• High-Entropy Regime ($H(X) \gg 3.5$): Observed in open-ended creative prose, diverse dialogue, and high-temperature sampling ($T \ge 1.0$). Draft acceptance plummets ($\alpha \le 0.30$). For $\alpha = 0.20$ and $K=5$, $\mathbb{E}[N] \approx 1.25$ tokens. Because the system incurs the latency of draft generation, state synchronization, and verification rollbacks, effective generation speed drops below standard autoregressive decoding.

Hardware Resource Calculator

The following runnable Python function models the execution bounds of GPU workloads, evaluating operational intensity, memory bandwidth bottlenecks, and entropy-dependent speculative speedups.

from dataclasses import dataclass
import math

@dataclass(frozen=True)
class GPUHardwareProfile:
    name: str
    peak_fp16_tflops: float  # Dense TFLOPS
    memory_bandwidth_tb_s: float  # TB/s
    hbm_capacity_gb: float

@dataclass(frozen=True)
class WorkloadProfile:
    parameter_count_billions: float
    precision_bytes: int  # 2 for FP16, 1 for FP8
    context_length: int
    batch_size: int
    layers: int
    kv_heads: int
    head_dim: int

def calculate_hardware_bounds(
    gpu: GPUHardwareProfile,
    workload: WorkloadProfile,
    speculation_depth_k: int,
    acceptance_rate_alpha: float
) -> dict[str, float]:
    """
    Calculates operational intensity, execution time bounds,
    and effective throughput across standard and speculative execution regimes.
    """
    # 1. Hardware Ridge Point (FLOPs / Byte)
    peak_flops = gpu.peak_fp16_tflops * 1e12
    memory_bandwidth_bytes_s = gpu.memory_bandwidth_tb_s * 1e12
    ridge_point = peak_flops / memory_bandwidth_bytes_s

    # 2. Model Memory Footprint (Weights)
    weight_bytes = workload.parameter_count_billions * 1e9 * workload.precision_bytes
    
    # 3. KV Cache Footprint per Token Step (Bytes)
    kv_cache_bytes = (
        2 * workload.batch_size * workload.layers * workload.kv_heads
        * workload.head_dim * workload.context_length * workload.precision_bytes
    )

    # 4. Standard Autoregressive Step (K=1)
    flops_per_token = 2.0 * workload.parameter_count_billions * 1e9 * workload.batch_size
    standard_intensity = flops_per_token / weight_bytes
    standard_memory_time_s = weight_bytes / memory_bandwidth_bytes_s
    standard_compute_time_s = flops_per_token / peak_flops
    standard_step_latency_s = max(standard_memory_time_s, standard_compute_time_s)

    # 5. Speculative Verification Step (K > 1)
    speculative_flops = flops_per_token * speculation_depth_k
    speculative_intensity = speculative_flops / weight_bytes
    spec_compute_time_s = speculative_flops / peak_flops
    # Weight read occurs once across the batch of K tokens
    spec_step_latency_s = max(standard_memory_time_s, spec_compute_time_s)

    # 6. Expected Accepted Tokens via Acceptance Rate Alpha
    if math.isclose(acceptance_rate_alpha, 1.0):
        expected_accepted_tokens = float(speculation_depth_k + 1)
    else:
        expected_accepted_tokens = (
            1.0 - math.pow(acceptance_rate_alpha, speculation_depth_k + 1)
        ) / (1.0 - acceptance_rate_alpha)

    # Effective latency per token generated
    effective_speculative_token_latency_s = spec_step_latency_s / expected_accepted_tokens
    speedup_multiplier = standard_step_latency_s / effective_speculative_token_latency_s

    return {
        "hardware_ridge_point_flops_per_byte": round(ridge_point, 2),
        "standard_operational_intensity": round(standard_intensity, 4),
        "speculative_operational_intensity": round(speculative_intensity, 4),
        "total_weight_memory_gb": round(weight_bytes / (1024**3), 2),
        "kv_cache_footprint_gb": round(kv_cache_bytes / (1024**3), 2),
        "standard_token_latency_ms": round(standard_step_latency_s * 1000, 3),
        "speculative_token_latency_ms": round(effective_speculative_token_latency_s * 1000, 3),
        "expected_tokens_per_round": round(expected_accepted_tokens, 2),
        "effective_speedup_factor": round(speedup_multiplier, 2)
    }

if __name__ == "__main__":
    h100 = GPUHardwareProfile(
        name="NVIDIA H100 SXM5",
        peak_fp16_tflops=989.4,
        memory_bandwidth_tb_s=3.35,
        hbm_capacity_gb=80.0
    )
    
    llama_70b_tp8 = WorkloadProfile(
        parameter_count_billions=70.0 / 8.0,  # 8-way Tensor Parallel slice
        precision_bytes=2,
        context_length=4096,
        batch_size=1,
        layers=80,
        kv_heads=8,
        head_dim=128
    )

    # Scenario A: Low entropy (e.g., Code Verification, Alpha=0.88)
    low_entropy_results = calculate_hardware_bounds(
        gpu=h100,
        workload=llama_70b_tp8,
        speculation_depth_k=5,
        acceptance_rate_alpha=0.88
    )

    # Scenario B: High entropy (e.g., Creative Writing, Alpha=0.25)
    high_entropy_results = calculate_hardware_bounds(
        gpu=h100,
        workload=llama_70b_tp8,
        speculation_depth_k=5,
        acceptance_rate_alpha=0.25
    )

    print("--- Low-Entropy Regime (Predictable Code/Structured Data) ---")
    for metric, val in low_entropy_results.items():
        print(f"  {metric}: {val}")

    print("\n--- High-Entropy Regime (Open Dialogue/Creative Writing) ---")
    for metric, val in high_entropy_results.items():
        print(f"  {metric}: {val}")

3. Production Trade-offs & Benchmarks

Configuring high-throughput GPU serving infrastructure involves architectural trade-offs between memory layout, allocation granularity, and concurrency behavior. A critical choice centers on how memory is allocated for dynamic sequences: contiguous virtual memory allocations versus segmented physical block allocators.

Contiguous vs. Segmented Block Allocation

Traditional inference engines allocate memory for the KV cache by reserving a contiguous block of virtual memory sized to the maximum possible sequence length (e.g., 4096 or 8192 tokens) for every incoming request. Modern memory engines (such as PagedAttention) segment the KV cache into fixed-size physical blocks (typically holding 16 or 32 tokens) linked via dynamic lookup tables, identical to virtual memory paging in operating system kernels.