Being your own bank has it's downsides, one of which is physical attack.
Even without being your own bank and using fiat currencies, you could still be physically attacked and robbed, the robbers can also force you to transfer money to them, it happens, just that in such cases it is easier to track the criminal, than when they use BTC.
However, some people who suffer $5 wrench attacks are boastful about the amount they have in BTC, bringing attention to themselves before they get eventually attacked, so you can avoid this kind of attack by simply keeping quiet. Another way is to extend your seed phrase with a passphrase, this gives you the opportunity of plausible deniability, but make sure you load the base wallet with a good amount, but way less that what you actually have, so the thieves can take it and leave.