Downvotes are a really poor thing to be upset about. It's just feedback. If you don't listen to feedback you don't adjust.
Not only that, but not all downvotes are equal. There are people here who if they downvoted me it'd be closer to an upvote than anything. On the other hand, there are some people where a downvote signifies a significant level of disapproval, and, if I respect their perceptions normally, this is a clear sign that I must check before rek.
Downvotes are the bee's knees. They allow people to signal disapproval without diverting the conversation into a slog fest. Likewise, upvotes are good too. It's a system of fluid feedback, and while yes it can mean people become a little too invested in their karma ratios, on the whole it organically rewards 'good' behaviour while signalling bad. People here go to effort for upvotes, and that in turn has enriched our community with their productivity.
My advice to you batman is to try to understand the reason you are being voted down, and then decide whether you care about what it is they're communicating. Personally I tend to downvote not because of disagreement with their stated opinion, but because of how they're saying it. I'm very happy for people to disagree so long as they're not asshats about it.