When your model doesn't fit on one GPU, you can split it across multiple ones. There are several ways to do this and picking the wrong one can make adding a second GPU actually slower than using one.

Tensor parallelism

Basically split each model layer across GPUs, ie. each GPU holds a slice of every layer and they communicate mid-computation. Here every GPU works on every token, but it needs fast interconnect like NVLink to work well. Over PCIe (or worse, over network) the communication overhead eats the gains.

Pipeline parallelism

split the model by layers instead, eg. if you have 2 GPUs, GPU 0 gets the first half of the layers, GPU 1 gets the second half, and data flows through like a pipeline. This works fine over PCIe, but one GPU is always idle while the other computes.

How to pick?

The topology of your system decides which strategy works. For Nvidia hardware, if you run nvidia-smi topo -m it will show you how your GPUs are interconnected. for example, two A100s over NVLink move data faster than the same GPUs over PCIe, because NVLINK provides a direct link between GPUs without going through the PCIe Bus.