The Apolipoprotein E ε4 (APOE ε4) haplotype is the strongest genetic risk factor for late-onset Alzheimer's disease (AD). The Translocase of Outer Mitochondrial Membrane-40 (TOMM40) gene maintains cellular bioenergetics, which is disrupted in AD. TOMM40 rs2075650 ('650) G versus A carriage is consistently related to neural and cognitive outcomes, but it is unclear if and how it interacts with APOE. We examined 21 orthogonal neural networks among 8,222 middle-aged to aged participants in the UK Biobank cohort. ANOVA and multiple linear regression tested main effects and interactions with APOE and TOMM40 '650 genotypes, and if age and sex acted as moderators. APOE ε4 was associated with less strength in multiple networks, while '650 G versus A carriage was related to more language comprehension network strength. In APOE ε4 carriers, '650 G-carriage led to less network strength with increasing age, while in non-G-carriers this was only seen in women but not men. TOMM40 may shift what happens to network activity in aging APOE ε4 carriers depending on sex.