Quite frankly, I hate this phrase. Yet people say it to me all the time and will continue to say it, so I have to learn how to live with it. So do you. Most of the time, it simply is not worth the effort to explain to someone who (a) won't even remember and (b) doesn't care.
For those who don't know, why is "temple" such an annoying phrase? From the orthodox perspective (and the conservative one, from what I understand), our position is that the only "temple" is the one with a capital T in Jerusalem that was destroyed and will be rebuilt in the future.
"Temple" was a revolutionary idea created very early in the reform movement. And, to be honest, it's one of the most appropriate positions that the reform movement has ever taken. It's exactly on point with their philosophy. The idea is that since the Second Temple was destroyed in 77 CE, the Temple continues to exist through individual Jews, and the Temple can be recreated wherever those Jews assemble and live Jewishly. From their perspective, it's removing the intermediaries and going straight to G-d. If you like analogies, I think of the Protestant Reformation and the opposition to the authority of the Pope and priests.
Of course, that argument would make you think that the orthodox think that the Temple is necessary today as an intermediary. We don't, though I admit I'm not as well versed with the orthodox arguments as the reform one. However, my personal understanding is that the Temple is someone very specific and irreplaceable (except by another one, of course). I think there is such a difference between the closeness between the Jewish people now and at the time the Temple stood, and that our synagogues can't compare. In essence, I find it presumptuous to compare the two. (Though being mostly vegetarian and a vegetarian/vegan sympathizer, I have my own concerns about how I would feel about the resurrection of animal sacrifices - but that's another discussion for another day.)
As you might imagine, this is difficult to explain to people who don't feel the same way. Certainly more than you would generally like to do in casual conversation. So...I let it slide.
As a side point, I often hear orthodox and reform Jews refer to a "conservative temple." My understanding is that the conservative movement's position is identical to the orthodox position. Therefore, please avoid referring to conservative synagogues as "temples."
Following that tangent, something I don't understand is why, if the orthodox are so opposed in principle to the use of the word "temple," why do they say "reform temple" and "conservative temple"?? Besides not liking it when other people say it, I don't ever use it to refer to other synagogues of any group. Of course, if you hear the word "temple" from an orthodox mouth, you can be almost certain that it was said with a sneer! In that sense, it's very similar to the use of the word "goy" (See Adventures in Semantics: Goy v. Non-Jew).