I have friends who enjoy anime (even a lot) and friends who live anime. I must confess a few of my friends pluck my last nerve from time to time.
I have one friend who constantly talks in supposed Japanese (she got really upset when another friend who is actually majoring in Japanese pointed out that much of what she said was not correct) She comes into the room like a grammatically incorrect Japanese-speaking hurricane, and enters any conversation by releasing her verbal floodgate of latest anime/yaoi news. I could be talking about my cat dying and she would enter the room and find a way to incorperate Naruto into the conversation, while spewing all the commonly used words in Fanfiction in Japanese (Ex, kawaii). She only listens to music if it is Japanese, and shuns anything from another nationality. She plays DDR at the mall arcade, hogging the game mercilessly and ticking off everyone waiting, even though she is horrible at it. She also writes stories all day about every anime charactor ever being gay whether or not they actually are and does this instead of anything constructive in school. It wouldn't bother me if she didn't talk about it ALL THE TIME, because to each thier own, untill I have to hear about it.
Honestly she is usually really hard to be around. I have a couple other friends that are like her on a much lower level. And around here I see a lot of people like her at conventions.
Two of my other friends are dating, and they have no job, don't do school, and live with eachother with one of the girl's parents. They can talk about other things besides anime, just barely, but all they do together is write yaoi fanfiction all day. ALL DAY. Before they dated their love was like a highschool manga, they both had a crush on eachother but totally couldn't just tell eachother, so they both told me, all day every day.
I enjoy anime, I read fanfiction, and if wallet permits, I go to conventions. I don't really consider myself an otaku per se, but I like a fair amount of anime (though I can be picky). My first "crush" was on an anime charactor. A lot of people at my college are really into anime (I am in the anime club with many of them), but they're people beyond that -- you could hate anime and still be friends with them. But a few of my friends are human embodiements of anime, they seriously live it, to the point to where you can't even really talk to them unless it somehow relates to anime or at the very least Japan.
My point is, the annoying friends I listed, are the kind of people that make the rest of us who love anime look terrible. As someone else in this thread said, people NOTICE the girl running around dressed like Saske screaming at the top of her lungs about "Naruto and Saske making out being so KAWAII!", people don't notice the rest of -- the majority of -- us anime fans who are social, productive, and active in various ways. Thus people associate anime watching with people like that. It's normal when you don't know about/understand something to associate it with extreme stereotypes, however irritating it is.