Cancer stem cells (CSCs) have been identified in primary breast cancer tissues and cell lines. The size of CSC population varies a lot among cancer tissues and cell lines but is associated with aggressiveness of breast cancer. In this study, we develop a mathematical model to explore the key factors which control the size of CSC during tumor cell growth both in vitro and in vivo. Our mathematical model and experimental data suggest that there is a negative feedback mechanism to control the balance between CSC and non-stem cancer cells. We further calculate how feedback sensitivities and robustness can be regulated by different intrinsic and extrinsic factors.