Mixture of Experts (MoE) Explained: Architecture, Routing & Why It Matters for AI Testing

Ask most engineers how a trillion-parameter model runs fast enough to answer in two seconds, and you’ll get some version of “an insane amount of GPUs.” That’s true, but it’s not the real answer. The real answer is that most of those trillion parameters never touch your prompt at all. GPT-4, Mixtral, DeepSeek-V3, and Grok don’t activate every parameter for every token — they activate a small, dynamically-chosen slice of themselves, every single time. The architecture that makes this possible is called Mixture of Experts, and if you’re testing anything built on a frontier model in 2026, you’re already testing a MoE system whether your test plan accounts for it or not.

A dense model activates 100% of its parameters for every token. A Mixture-of-Experts model routes each token through a gating network to only 2 of 8 experts, activating roughly 12.5% of expert parameters per token. DENSE vs MIXTURE-OF-EXPERTS Dense Model IN Feed-Forward Layer OUT 100% params active Mixture of Experts IN Router (Gate) Expert 3 Expert 4 Expert 7 Expert 2 Expert 1 Expert 5 Expert 6 Expert 8 OUT ~12.5% of experts active (2 of 8) Blue = uniform dense activation · Rose & mint = this token’s 2 routed experts · Grey = present, skipped for this token

The Short Answer

Mixture of Experts (MoE) is a neural network design that replaces one large, always-on layer with many smaller “expert” sub-networks plus a lightweight router that decides, token by token, which experts actually get to process that token. Instead of every parameter contributing to every prediction, only a small fraction — typically 2 out of 8, or a handful out of dozens — activates per token. The rest of the model exists on disk and in memory, but sits idle for that particular piece of text.

• It decouples parameter count from compute cost. A MoE model can have 10x more total parameters than a dense model while spending roughly the same FLOPs per token, because most of those parameters never activate at once.
• A router (gating network) makes the routing decision. It scores every expert for every token and picks the top-K highest-scoring ones — usually K=1 or K=2.
• This is why trillion-parameter models are commercially viable. Mixtral, DeepSeek-V3, Grok, and (by most credible reporting) GPT-4 all use sparse MoE layers to get frontier-scale knowledge without frontier-scale inference bills.
• Sparsity is a trade, not a free lunch. You gain compute efficiency and pay for it in routing complexity, load-balancing overhead, and a genuinely new category of bugs.
• For QA, MoE changes what “the same input” means. Batch composition can influence which expert a token routes to under capacity limits — a dense-model mental model of determinism doesn’t hold the same way.

Attribute	Dense Model	Mixture of Experts
Parameters active per token	100%	Typically 10–25% (e.g. 2 of 8 experts)
Total parameter count vs. compute cost	Tightly coupled — more params = proportionally more FLOPs	Decoupled — total params can scale far ahead of per-token FLOPs
Inference cost at a given quality bar	Higher, for equivalent capability	Lower per token, at the cost of more total memory/VRAM
Training stability	Well-understood, simpler to tune	Harder — needs load-balancing losses to avoid expert collapse
Determinism characteristics	Output depends only on the input (given fixed sampling)	Can depend on batch composition under hard expert-capacity limits
Memory footprint	Matches active compute closely	Must hold all experts in memory even though few activate per token
Known production users	LLaMA 2, most sub-70B open models	Mixtral, DeepSeek-V3/V3.1, Grok-1, GPT-4 (widely reported)

How Mixture of Experts Actually Works

Every MoE layer has three moving parts: a set of experts, a router that scores them, and a combination step that merges what the selected experts produce. Understanding each one is what separates “I’ve heard of MoE” from being able to reason about why a MoE-backed system just misbehaved.

Experts Are Not “Topic Specialists” — That’s a Popular Myth

The name “expert” invites a tempting but wrong mental model: an expert for coding, an expert for poetry, an expert for math. In practice, research on Switch Transformer and Mixtral’s own released routing analysis shows experts specialize on much shallower statistical patterns — punctuation, verb tenses, specific token types, syntactic structure — not clean human-legible domains. Two experts can both fire heavily on the same sentence for reasons that have nothing to do with “topic.” Don’t design tests around the assumption that Expert 4 is secretly your model’s math brain; that assumption doesn’t survive contact with the actual routing logs.

The Gating (Router) Network

The router is a small learned layer — often just one matrix multiplication followed by a softmax — that takes each token’s hidden representation and outputs a score for every expert in that layer. Conceptually:

router_logits = token_hidden_state @ W_gate        # one score per expert
router_probs  = softmax(router_logits)
top_k_experts = argmax_k(router_probs, k=2)        # e.g. top-2 routing

The router is trained jointly with the rest of the model — it isn’t hand-coded, and nobody explicitly tells it what each expert should specialize in. Specialization, to the extent it exists, emerges purely from gradient descent trying to minimize loss.

Top-K Routing and Sparse Activation

Once the router scores every expert, only the top-K are actually invoked — the rest contribute nothing to that token’s forward pass. Mixtral 8x7B uses top-2 routing across 8 experts per layer (so ~2/8 = 25% of expert parameters activate per token, per layer). DeepSeek-V3 uses a much finer-grained scheme — 256 routed experts with 8 selected per token, plus always-on shared experts — which is part of why it achieves strong quality at a lower activated-parameter count (37B active out of 671B total) than you’d expect from its total size.

Expert Capacity and Load Balancing

Here’s the part almost nobody outside ML infra teams knows, and it’s the single most important fact for testers: experts have a capacity limit — a maximum number of tokens they’ll accept per batch, set for hardware efficiency. If an expert’s top-scored tokens exceed its capacity in a given batch, the overflow tokens get dropped from that expert and either routed to a lower-preference expert or passed through with no expert transformation at all (a residual “skip”). This means, in principle, the exact same token can be handled differently depending on what else is in the batch with it — a property dense models simply don’t have.

To prevent a small number of experts from dominating (which wastes capacity on unused experts and creates a rich-get-richer training dynamic called expert collapse), MoE training adds an auxiliary load-balancing loss that penalizes uneven routing distribution. Get this wrong at training time, and you ship a model where 2 of 8 experts absorb 80% of tokens while the rest barely train — a quality problem no amount of prompt engineering fixes.

How a Single Token Actually Moves Through an MoE Layer

A token embedding is scored by the gating network against all experts, the top 2 experts by score are selected and weighted, each processes the token in parallel, and their weighted outputs are summed to produce the final output. TOKEN ROUTING, STEP BY STEP Token hidden state 1 Gating Network 2 E1 E2 E3 E7 E8 Expert 3 weight 0.62 3 Expert 7 weight 0.38 Σ weighted sum 4 Output to next layer 5 Weights come from the router’s softmax scores — rose (top pick) contributes more than mint (second pick).

A Brief History: From “Outrageously Large” to Trillion-Parameter Production Systems

MoE isn’t a 2024 invention — it’s a 2017 idea that finally found hardware and training recipes good enough to matter at scale.

Year	Model / Paper	Organization	Total Params	Active Params / Token	Why It Mattered
2017	“Outrageously Large Neural Networks” (Sparsely-Gated MoE)	Google Brain	Up to 137B	~4–8B	First to show sparse MoE could scale LSTMs 1000x with modest compute increase
2021	Switch Transformer	Google	1.6T	~billions (top-1 routing)	Simplified routing to top-1; first trillion-parameter language model
2021	GLaM	Google	1.2T	~97B	Beat GPT-3 quality using ~1/3 the training energy
2023	Mixtral 8x7B	Mistral AI	46.7B	~12.9B	First widely-used open-weight MoE; matched/beat much larger dense models
2024	Mixtral 8x22B	Mistral AI	141B	~39B	Extended the same recipe to frontier-adjacent scale, fully open weights
2024	Grok-1	xAI	314B	~86B (top-2 of 8)	Released open-weight, confirmed production MoE at frontier scale
2024–2025	DeepSeek-V3 / V3.1	DeepSeek AI	671B	~37B	Fine-grained 256-expert routing; frontier quality at strikingly low active-param count
2023–present	GPT-4 / GPT-4-class models	OpenAI	Undisclosed	Undisclosed	Never officially confirmed, but widely reported (including by third-party technical analyses) to use a MoE architecture

The pattern across every row: total parameter count keeps climbing far faster than active parameter count. That gap is the entire economic argument for MoE, and it’s why nearly every company racing toward trillion-dollar AI infrastructure bets — Google, Microsoft/OpenAI, Meta, xAI, Mistral, DeepSeek — has shipped or is actively building MoE architectures rather than simply scaling dense models further.

Why Trillion-Dollar AI Companies Bet on MoE

The economics are blunt: inference cost scales with active compute, not total parameter count. A dense model that matches DeepSeek-V3’s 671B total parameters would need to run all 671B for every single token — at massive GPU-hour cost, per request, forever. DeepSeek-V3 gets comparable frontier quality while only ever running ~37B parameters per token. Multiply that difference across billions of daily API calls, and MoE isn’t an academic curiosity — it’s the difference between an inference bill that scales linearly with model quality and one that scales sub-linearly.

There’s a second reason, less discussed but just as real: MoE lets a company keep growing “knowledge capacity” (total parameters, and therefore the breadth of what the model has room to encode) largely independently from growing “reasoning compute per response” (active parameters). That decoupling is exactly what you want when the bottleneck shifts from “can we afford to train a bigger model” to “can we afford to serve a bigger model to hundreds of millions of users.”

The Testing Blind Spot: What QA Teams Miss With MoE Systems

None of the above is purely academic if your team tests a product built on a MoE-backed LLM — which, if you’re testing anything wired to a frontier API in 2026, you almost certainly are. Dense-model testing habits carry over, but they’re not sufficient. Here’s what actually changes.

Non-Determinism From Expert Routing

Most testers already know LLM outputs vary run to run because of sampling temperature. MoE adds a second, less obvious source of variation: under hard expert-capacity limits, which expert a token gets routed to — or whether it gets dropped and skipped — can depend on what other tokens are in the same batch. Two functionally identical requests sent at different times, landing in different batches with different co-tenants, can theoretically receive slightly different expert assignments. This is invisible from the API surface and won’t show up in a single-request smoke test; it surfaces as low-frequency, hard-to-reproduce output drift that looks like a flaky test until you understand where it’s coming from.

Load-Balancing Bugs and the “Dead Expert” Problem

If you’re evaluating or fine-tuning an open-weight MoE model (Mixtral, DeepSeek, or a custom MoE), a poorly-tuned load-balancing loss can leave some experts dramatically under-trained — a “dead expert” that any token routed to it gets a low-quality transformation from. This shows up as quality that’s inconsistent in a pattern-less way across otherwise-similar inputs, because which expert a given input happens to hit is not something your test cases control directly.

Token Dropping Under Capacity Limits

When an expert exceeds its per-batch capacity, some production MoE implementations drop the overflow tokens entirely rather than reassigning them — that token effectively skips transformation at that layer. Under high load (exactly the conditions your load tests should be simulating), this can degrade output quality in ways a low-traffic functional test suite will never catch, because token dropping is a function of concurrent batch load, not of any single request in isolation.

Evaluating Output Quality Across Experts, Not Just Across Prompts

Traditional regression testing assumes: same input in, same output class out, forever, until you change something. With MoE, “you” aren’t the only thing that can change the routing — a provider’s silent backend update to expert weights, capacity factors, or even batch-scheduling logic can shift which experts specific token patterns hit, producing a regression with no corresponding change in your own prompts or code. Golden-dataset regression testing and LLM-as-judge scoring (covered in more depth in our AI testing strategy guide) become less “nice to have” and more “the only way you’ll catch this class of regression at all.”

A Practical MoE Testing Checklist

You can’t inspect expert routing through most commercial APIs — OpenAI, Anthropic, and most hosted providers don’t expose which expert handled which token. That doesn’t mean you’re powerless; it means your test strategy has to work around the black box instead of through it.

• Test under realistic concurrent load, not just isolated requests. Capacity-related dropping and routing variance are load-dependent — a test suite that only ever sends one request at a time will never exercise this failure class.
• Run the same prompt N times and measure output distribution, not a single output. Track variance over time, not just at release — a stable distribution that drifts week over week is a signal worth alerting on.
• Build a golden dataset with expected quality bands, not exact-match expected outputs. Exact string matching is already wrong for any LLM; for MoE specifically, treat wider output variance as expected baseline behavior, not noise to suppress.
• Re-run your evaluation suite on every model-version pin change — and periodically even when you haven’t changed the pin. Providers update backend weights without always bumping a version string you control.
• Log full request/response pairs with timestamps. When a quality regression does show up, correlating it against provider status pages / changelogs is often the only way to distinguish “our prompt changed” from “their model changed.”
• If you self-host an open-weight MoE model, monitor per-expert utilization in production. A skewed utilization histogram is an early warning sign of the same “dead expert” quality problems you’d otherwise only discover downstream, in output quality metrics.

Mixture of Experts isn’t a footnote in how frontier models are built anymore — it’s the default architecture for anything competing at the trillion-parameter tier, and understanding it changes what “testing an AI feature” actually means. The next time a test flakes on an LLM-powered feature with no code change in sight, routing-level non-determinism is worth ruling out before you write it off as “just how AI is.”

Bookmark