Infertility is a condition whereby pregnancy does not occur despite having unprotected sexual intercourse for at least one year. The main reason could originate from either the male or the female, and sometimes, both contribute to the fertility disorder. For the male, sperm disorder was found to be the most common reason for infertility. In this paper, we proposed male infertility analysis based on automated sperm motility tracking. The proposed method worked in multistages, where the first stage focused on the sperm detection process using an improved Gaussian Mixture Model. A new optimization protocol was proposed to accurately detect the motile sperms prior to the sperm tracking process. Since the optimization protocol was imposed in the proposed system, the sperm tracking and velocity estimation processes are improved. The proposed method attained the highest average accuracy, sensitivity, and specificity of 92.3%, 96.3%, and 72.4%, respectively, when tested on 10 different samples. Our proposed method depicted better sperm detection quality when qualitatively observed as compared to other state-of-the-art techniques.