If I have only one wallet, I can remember my only wallet's mnemonic seed phrase with my brain and memory. Likely it will be my best choice if I am in such hard situation for privacy and fund safety while I know that memory is not trusted. If I choose memory as my backup, I have to rehearse wallet recovery by brain every day.
Having your human brain memory as only backup is a recipe for desaster, sooner or later. It's true that you should rehearse regularly and possibly every day. But let me ask: how do you verify that your rehearsal from memory is correct without a written backup?
I have a feeling that memory verification without a written backup is quite a gamble, isn't it? Yes, I can imagine ways without it, but are they safe? I tend to say, no! And therefore, the need for a written backup makes "storage" in human memory somewhat obsolete. At least, I know for myself, that I don't want to rely on my own brain's memory. Too much experience with having forgotten stuff that I thought I wouldn't forget. Human memory is a fragile thing...
On the topic and the thread is already too long and stuff has been repeated over and over again and what comes to my mind may have been stated already (I only went briefly over first posts and few in between).
I don't want to judge anybody's family. I hope your own blood gets along with each other and there's enough respect to private stuff of family members. I can imagine that's sadly not always the case.
Your hardware wallet doesn't store your coins, it protects only the keys which enable movement of coins. Typically a hardware wallet is protected by a pin or passphrase. If only you know that access code, then basically: problem solved.
It has been mentioned to pay extra attention to your (redundant!) mnemonic recovery words backup(s). Those should not be accessible easily and maybe only family members you can trust 100% should know where they are. Again, I don't want to judge your family. Only you can assess who should know what or if anybody should know something at all.
I think, there's always some way to hide secrets that it's utterly hard to find for others or very unlikely to be found by those who shouldn't find it. Be creative but remember to not let the foot gun go off on yourself.
Every family member has the right for private space, to remain untouched by other family members. I wouldn't want to live under the same roof if that is no consensus within the family members. Of course, easier said, than actually lived...