Recent experiments on atom loss in ultra-cold Fermi gases all show a maximum at a magnetic field below Feshbach resonance, where the s-wave scattering length is large (close to inter-particle distance) and positive. These experiments have been performed over a wide range of conditions, with temperatures and trap depths spanning over three decades. Different groups have come up with different explanations, among them the emergence of Stoner ferromagnetism. Here, we show that this maximum is a consequence of two major steps. The first is the establishment of a population of shallow dimers, which is the combined effect of dimer formation through three-body recombination, and the dissociation of shallow dimers back to atoms through collisions. The dissociation process will be temperature dependent, and is affected by Pauli blocking at low temperatures. The second is the relaxation of shallow dimers into tightly bound dimers through atom-dimer and dimer-dimer collisions. We have constructed a simple set of rate equations describing these processes. Remarkably, even with only a few parameters, these equations reproduce the loss rate observed in all recent experiments, despite their widely different experimental conditions. Our studies show that the location of the maximum loss rate depends crucially on experimental parameters such as trap depth and temperature. These extrinsic characters show that this maximum is not a reliable probe of the nature of the underlying quantum states. The physics of our equations also explains some general trends found in current experiments.