We aimed to examine the relationship between Apolipoprotein E ε4 (APOE*4) carriage on cognitive decline, and whether these associations were moderated by sex, baseline age, ethnicity, and vascular risk factors. Participants were 19,225 individuals aged 54-103 years from 15 longitudinal cohort studies with a mean follow-up duration ranging between 1.2 and 10.7 years. Two-step individual participant data meta-analysis was used to pool results of study-wise analyses predicting memory and general cognitive decline from carriage of one or two APOE*4 alleles, and moderation of these associations by age, sex, vascular risk factors, and ethnicity. Separate pooled estimates were calculated in both men and women who were younger (ie, 62 years) and older (ie, 80 years) at baseline. Results showed that APOE*4 carriage was related to faster general cognitive decline in women, and faster memory decline in men. A stronger dose-dependent effect was observed in older men, with faster general cognitive and memory decline in those carrying two versus one APOE*4 allele. Vascular risk factors were related to an increased effect of APOE*4 on memory decline in younger women, but a weaker effect of APOE*4 on general cognitive decline in older men. The relationship between APOE*4 carriage and memory decline was larger in older-aged Asians than Whites. In sum, APOE*4 is related to cognitive decline in men and women, although these effects are enhanced by age and carriage of two APOE*4 alleles in men, a higher numbers of vascular risk factors during the early stages of late adulthood in women, and Asian ethnicity.