you should be very afraid.
And being afraid will help you accomplish what? Nothing at all. Instead of being afraid, you should start doing your research. It's good that OP did.
To OP:
I know that you've been hearing a lot of things about people having their wallets hacked. But it's all because of their own negligence. It's possible that their computer was already hacked even before they have generated their wallet. It's possible that someone (in his home) has access to his computer and is also a bitcoin user and they got the bitcoins. Remember, in your own house is most likely where you'd find the thief (in life in general, not just in bitcoins).
If you are not hacked by now, then probably you are not hacked at all and you're just overthinking things. Because if you were the hacker, you wouldn't wait for a little long when you hacked a bitcoin address because the person might sell out some of his coins. You're not going to wait for that person to have more bitcoins, you just hack it right away.
If you got hacked, you can't do anything. The only indication that you got hacked is when your bitcoins are already gone, right? You and I know that bitcoin's transactions are not reversible and that doesn't change even when you have a lot of bitcoins.