Quite honestly, you're probably going to look like a nutter whatever you do. But that's ok. You can minimize your public nuttiness with a few tips. This topic always makes me think of plane trips, so perhaps this colors my discussion. Feel free to add your tips in the comments!
First, seek a private location, if available. There's not much you can do in some circumstances, such as a plane with the seatbelt light still on. However, if available, there are airport chapels, the back of the plane, and other quiet corners.
Half the time, if you don't shuckle (rock) back and forth, you just look like someone who reads while moving his or her lips.
Use your cell phone! Pretend you're talking on your phone or a bluetooth and suddenly, you look no more crazy than anyone else. (But you will look crazy to someone like me who never remembers that bluetooths exist.)
But what about bowing during the Amidah? For instance, if I'm on a plane, I make less of a bow in my seat than I normally would. The people sitting by me inevitably realize I'm doing "something weird," but it generally doesn't draw attention from others.
And now, the pièce de resistance, tefillin! How can you possibly avoid looking like a nutter when forced to don tefillin in a public place? Surprise: You can't! If you have tips, I'd love to hear them, but I have yet to see it done without confusing everyone else. And at some point, some "proud American" is going to think you're a terrorist or something equally un-American. Don't think you're safe because you're flying to or from a large Jewish community: I remember a news article approximately 2-3 years ago where a teenager donned tefilin on a flight between Chicago and NYC, and the flight attendants tackled him!