Beged ish is Hebrew for "a man's garment." In short, men and women aren't supposed to wear the clothing of the other sex. It's not tznius. The arguments in "polite conversation" tend to be about women wearing pants. Of course, aside from being a d'oraisa prohibition, there are also issues of tznius.
The source is Deuteronomy/Devarim 22:5, "A masculine accouterment shall not be upon a woman, nor shall a man wear a woman's garment, for all those who do these thing are an abomination to Hashem, your G-d."
The problem is how to define gendered clothing. It certainly can change over time, based on community custom. However, in our modern world, how do we define community? And how do we deal with a change in the times? These are questions that rabbis disagree on vehemently. Even more complicated, rabbis may allow an item of clothing as permissible under one theory but prohibited under another halachic theory. For instance, my understanding is that Rav Moshe (Rabbi Moshe Feinstein, z"tl-if you don't know him, you should) held that women's pants are not beged ish, but the crotch split violates tznius. So even when you get around a beged ish issue, there are tznius issues to consider. On the other end of the spectrum, some groups would (loudly) emphasize that pants are beged ish per se. Therefore, they are not permissible in any context, even as pajamas in the privacy of your own home with no men around.
This is an area where my favorite quote applies: "Everyone who is more lenient than you is a heretic. Everyone more stringent than you is a fanatic." Whenever you have this conversion, unless you agree, the other person is going to think you're a heretic or fanatic, and you will feel the opposite.
Yet there's a twist here: what about all that cross-dressing on Purim? Didn't I see the rabbi dressed as a mermaid?? (True story: I met my first rabbi's 2 year old son on Purim, and he was dressed as a girl. For several weeks, I thought the rabbi had a daughter.) Apparently, cross-dressing is allowed on Purim because everyone knows that the intent isn't to be lewd; we presume it is for the joy of Purim.
The most interesting beged ish question I encountered: I heard swords are beged ish. Does this mean that guns are beged ish today?
As a last thought, my "research" brought me to this gem of a quote: "Modesty is a characteristic that has no Hashkafa."