Exactly! Especially now that sometimes, even the closest members of your family can betray you for the sake of money. But it really depends on how much you trust them. For me, its okay if they have the slightest idea that Im into crypto or have crypto-related activities, but they will never know the other details, particularly the important ones like the value of my assets, passwords, or seed phrases. Writing down some important details and keeping them in the safest place in my room works for me because my family members arent nosy when it comes to everyones private and personal things. You should try that too!
The challenge I'm seeing here might not be trust issues. This kind of storage system brings point of failure. If something happen to that hardware wallet, you can't point finger, you can't suspect one person because you never can tell exactly who did it. To avoid this kind of complications, your best interest is to remove the hardware wallet from the place and keep in a place where nobody will be able to access, you don't know who will try to take the Bitcoin from the wallet.
Sometimes we think people are ignorant of the things we do until they shock us with things we never expected. The guy might even be fool that nobody know what is a hardware wallet in his home until one person steal everything from the wallet most especially people that are closer to him who might know his passwords and pins.