A recent genome-wide association meta-analysis showed a suggestive association between alcohol intake in humans and a common single nucleotide polymorphism in the ras-specific guanine nucleotide releasing factor 2 gene. Here, we tested whether this variant – associated with lower alcohol consumption – showed associations with brain structure and longitudinal ventricular expansion over time, across two independent elderly cohorts, totaling 1,032 subjects. We first examined a large sample of 738 elderly participants with neuroimaging and genetic data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI1). Then, we assessed the generalizability of the findings by testing this polymorphism in a replication sample of 294 elderly subjects from a continuation of the first ADNI project (ADNI2) to minimize the risk of reporting false positive results. The minor allele – previously linked with lower alcohol intake – was associated with larger volumes in various cortical regions, notably the medial prefrontal cortex and cingulate gyrus in both cohorts. Intriguingly, the same allele also predicted faster ventricular expansion rates in the ADNI1 cohort at 1- and 2-year follow up. Despite a lack of alcohol consumption data in this study cohort, these findings, combined with earlier functional imaging investigations of the same gene, suggest the existence of reciprocal interactions between genes, brain, and drinking behavior.