The Quadrupole magnetic flow sorter (QMS) has been developed to separate immunomagnetically labeled cells from unlabeled cells at high throughput. In order to make QMS separation performance predictable and controllable, a theoretical model of the magnetic sorting process in the QMS was constructed based on (1) the magnetophoretic mobility distribution of the cell sample, (2) the annular channel geometry, (3) the magnetic field strength and (4) the adjustable flow conditions of the QMS. To test this model, breast cancer cells (HER-2/neu) were immunomagnetically labeled and then mixed with white blood cells for separation by the QMS. The magnetophoretic mobility distribution was measured by the Cell Tracking Velocimetry (CTV) technique developed in our lab. The good agreement between prediction and experimental results demonstrated the reliability of the model and the promise of a theoretical approach to optimizing conditions for QMS operation.