Model parallelism is a distributed training and inference technique where a single neural network is split across multiple devices so that different parts of the model compute different portions of the forward and backward passes.
What is Model Parallelism?
When a model is too large to fit on one GPU or accelerator, you can partition it across devices. In tensor model parallelism, individual tensor operations are split across devices, for example splitting the hidden dimension of matrix multiplications. In pipeline model parallelism, layers are assigned to different devices and microbatches are streamed through the pipeline. These approaches can be combined, and often run alongside data parallelism, where multiple replicas process different batches. Model parallelism requires communication between devices, so its performance depends on interconnect bandwidth and latency. It also influences how checkpoints are stored, how optimizer states are sharded, and how activation memory is managed.
Where it is used and why it matters
Model parallelism is used for training and serving large language models, diffusion models, and large multimodal transformers that exceed single-device memory limits. It matters because it enables scaling model size, which can improve capability, but it increases systems complexity. Poor partitioning can lead to idle devices, communication bottlenecks, or numerical mismatches across partitions. Strong observability and careful configuration are needed to avoid wasting expensive accelerator time.
Types
- Tensor parallelism: split matrix operations across devices, synchronize partial results.
- Pipeline parallelism: split layers across stages, run microbatches to fill the pipeline.
- Sequence parallelism: shard sequence dimensions to reduce activation memory and communication.
FAQs
- How is model parallelism different from data parallelism?
Data parallelism replicates the whole model on each device, while model parallelism splits one model across devices. - What is the main overhead?
Cross-device communication for activations, gradients, and partial tensor results is the primary overhead. - Do I still need gradient synchronization?
Yes, when model parallelism is combined with data parallelism, gradients must be reduced across replicas. - Is model parallelism used in inference too?
Yes, large models can be sharded across devices for inference, but latency depends on communication and pipeline bubbles.
