Background Little is known regarding whether arterial stiffness and atherosclerotic burden are each independently associated with brain structural changes. Simultaneous assessments of both arterial stiffness and atherosclerotic burden in associations with brain could provide insights into the mechanisms of brain structural changes. Methods and Results Using data from the SESSA (Shiga Epidemiological Study of Subclinical Atherosclerosis), we analyzed data among 686 Japanese men (mean [SD] age, 67.9 [8.4] years; range, 46-83 years) free from history of stroke and myocardial infarction. Brachial-ankle pulse wave velocity and coronary artery calcification on computed tomography scans were measured between March 2010 and August 2014. Brain volumes (total brain volume, gray matter, Alzheimer disease signature and prefrontal) and brain vascular damage (white matter hyperintensities) were quantified using brain magnetic resonance imaging from January 2012 through February 2015. In multivariable adjustment models including mean arterial pressure, when brachial-ankle pulse wave velocity and coronary artery calcification were entered into the same models, the β (95% CI) for Alzheimer disease signature volume for each 1-SD increase in brachial-ankle pulse wave velocity was -0.33 (-0.64 to -0.02), and the unstandardized β (95% CI) for white matter hyperintensities for each 1-unit increase in coronary artery calcification was 0.68 (0.05-1.32). Brachial-ankle pulse wave velocity and coronary artery calcification were not statistically significantly associated with total brain and gray matter volumes. Conclusions Among Japanese men, higher arterial stiffness was associated with lower Alzheimer disease signature volumes, whereas higher atherosclerotic burden was associated with brain vascular damage. Arterial stiffness and atherosclerotic burden may be independently associated with brain structural changes via different pathways.