A model whose feed-forward layers are split into many expert networks, with a router that sends each token to only a few of them. Total parameters can grow without a matching rise in compute per token, so an MoE model gets more capacity for a given inference cost.
Card 1 of 292. Answer: A model whose feed-forward layers are split into many expert networks, with a router that sends each token to only a few of them. Total parameters can grow without a matching rise in compute per token, so an MoE model gets more capacity for a given inference cost.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
In each MoE layer a small gating network scores the experts for every token, picks the top k, often 1 or 2, and combines their outputs weighted by the router's scores. Attention and embeddings stay shared. Shazeer and colleagues brought sparsely gated experts to large networks in 2017, and the Switch Transformer (Fedus and colleagues, 2021) routed each token to a single expert. An auxiliary load-balancing loss stops the router from sending most tokens to a few favorite experts.
Example: Mixtral 8x7B has 8 experts per layer and routes each token to 2. It holds 46.7 billion parameters in total, not 56 billion, because only the feed-forward blocks are duplicated, and it uses about 12.9 billion per token, so it runs at roughly the speed of a 13B dense model. DeepSeek-V3 takes this further with 671 billion parameters, 37 billion active per token.
The catch: every expert has to sit in memory, so memory follows total parameters while compute follows active ones, and spreading experts over GPUs adds communication. Training can be unstable. And experts tend to specialize in token types and syntax more than in topics; Mixtral's authors found little evidence of experts by subject.
