The origin of the Moon's large-scale topography is important for understanding lunar geology, lunar orbital evolution and the Moon's orientation in the sky. Previous hypotheses for its origin have included late accretion events, large impacts, tidal effects and convection processes. However, testing these hypotheses and quantifying the Moon's topography is complicated by the large basins that have formed since the crust crystallized. Here we estimate the large-scale lunar topography and gravity spherical harmonics outside these basins and show that the bulk of the spherical harmonic degree-2 topography is consistent with a crust-building process controlled by early tidal heating throughout the Moon. The remainder of the degree-2 topography is consistent with a frozen tidal-rotational bulge that formed later, at a semi-major axis of about 32 Earth radii. The probability of the degree-2 shape having both tidal-heating and frozen shape characteristics by chance is less than 1%. We also infer that internal density contrasts eventually reoriented the Moon's polar axis by 36 ± 4°, to the configuration we observe today. Together, these results link the geology of the near and far sides, and resolve long-standing questions about the Moon's large-scale shape, gravity and history of polar wander.