Questioner: Given that the arms race was a disaster for the Soviet Union economically and of little advantage militarily, why did the USSR engage in it?
What it did to them economically is exactly what Khrushchev predicted, and presumably what JFK and his advisers had in mind when they turned down Khrushchev’s call for sharp mutual reduction of offensive military forces, and his significant unilateral steps in that direction: stagnation, and an end to significant socioeconomic progress, in fact decline relative to the vastly richer West.
I think one can make an argument, like yours, that the USSR should have simply let itself fall far behind the US in offensive military capacity, and hope for the best. With their experience, and what they saw happening right then — with the enormous “missile gap” (in US favor), the purposeful humiliation at the missile crisis, JFK’s military buildup and sharp increase in intervention and support for military dictatorships — it’s not too surprising that they chose the suicidal course of trying to match the US in military terms.
The far more interesting question, both as seen from Mars and (crucially) as seen from here, is why the US leadership, with the applause of elites, continued massive arms build-up, and is now hysterically worshipping the class A mass murderer and torturer who drove it farther forward in the 80s.