The different brain anatomy of men and women is both a classic and continuing topic of major interest. Among the most replicated and robust sex differences are larger overall brain dimensions in men, and relative increases of global and regional gray matter (GM) in women. However, the question remains whether sex-typical differences in brain size (i.e., larger male and smaller female brains) or biological sex itself account for the observed sex effects on tissue amount and distribution. Exploring cerebral structures in men and women with similar brain size may clarify the true contribution of biological sex. We thus examined a sample of 24 male and 24 female subjects with brains identical in size, in addition to 24 male and 24 female subjects with considerable brain size differences. Using this large set of brains (n = 96), we applied a well validated and automated voxel-based approach to examine regional volumes of GM. While we revealed significant main effects of sex, there were no significant effects of brain size (and no significant interactions between sex and brain size). When conducting post hoc tests, we revealed a number of regions where women had larger GM volumes than men. Importantly, these sex effects remained evident when comparing men and women with the same brain size. Altogether, our findings suggest that the observed increased regional GM volumes in female brains constitute sex-dependent redistributions of tissue volume, rather than individual adjustments attributable to brain size.