Sex steroids, also known as sex hormones are synthesized naturally by the gonads (ovaries or testes). The two main classes of sex steroids, androgens and estrogens, are crucial hormones for the proper development and function of the body; they regulate sexual differentiation, the secondary sex characteristics, and sexual habits. They are also well known to influence many common cancers, especially hormonally driven cancers such as breast and prostate cancer. In this report, we review the association of sex steroids with cancer and the potential use of endocrine manipulation in cancer therapy as well as the limitations and challenges faced in this field.