I feel like it's worth worrying about if you keep your bitcoins on hot wallets. If we are talking about a cold wallet, then without the physical pressing of the button it is impossible to steal your bitcoins. Personally, I use the hardware wallet Ledger Blue.
The more money stored in your wallet , the higher security standard you have to set. That's the rules , you'll automatically do it for yourself to swipe the insecure feeling up.
Own an offline and hardware wallet are necessary without third party , you know the next level about that. Don't be scared but be more careful.