Meshech Chochmah in Parshat Ki Tisa (the section of text that follows Terumah in Exodus) speaks at length of the connection between the Torah's permitting certain symbols and Israel's having erred with the Golden Calf. The Israelites mistakenly viewed Moses as a middleman to God, and in his absence they sought a concrete crutch to replace him. Moses therefore cried out loudly, "Do you think that I am your key to talking with God? Heaven forbid! I am only a man like you. The Torah does not depend upon me, and had I never appeared, the Torah would be just as it is now."

(As proof of this, during the 38 years that God was incensed with Israel until the generation born in Egypt passed away, God never spoke to Moses in terms of endearment. --Rashi on Deuteronomy 2:17)

One must not imagine that the Tabernacle and the Temple are inherently holy. Their holiness derives only from God's residing among His children, and if those children break their covenant, the holiness of the Beit HaMikdash and Mishkan will be removed. They will be like everyday items destroyed by vandals. Thus, when Titus entered the Temple with a harlot, he was not harmed, for its holiness had already been removed.