If hardware wallet manufacturers and resellers had the same quality standards than other security-relevant devices (e.g. ATMs or privacy phones), then they should be considered safe. Unfortunately probably they aren't in the same quality category, according to the last incidents.

The challenge is probably that their devices have to be cheap (and thus quality control can't be that extensive like in an ATM, it would be too expensive) but the attack target (the Bitcoins a hacker could steal) is very attractive, potentially similar to an ATM's content.

Anyway I wonder if stock hardware (PCs, phones etc.) couldn't be targeted in a similar way, like with spy chips. In the current market it wouldn't be worth it probably (if a malicious company installs such chips on their devices, probably only a low percentage of their customers even would use crypto) but if Bitcoin became very popular, this is perhaps a real danger in the low price device segment. Thus, trust will always be an issue, regardless of the device, and even paperwallets are only 100% safe if you do the key generation math with pen and paper ...

So in the end, IMO there is always a very small chance that you are using a device with compromised hardware. And thus, hardware wallets are probably not significantly less safe than other devices. Personally, I do have some "bad feeling" about them, and thus I don't use them, but that may be paranoia ...