You would lost your coin, but if someone have access to both hardware wallet and your finger at same time, that someone also could steal your Bitcoin another way such as intimidation or violence.
If you think about it, fingerprint authorization is not a bad thing provided that it is implemented properly and used carefully. To increase the security of your funds, developers may allow to register different fingers for different accounts, it will be akin to use of passphrases, but this approach will be more natural and understandable for average users. For example, the first account could be tethered to your thumb and will serve as "duress wallet". You can put a small amount of bitcoins on that account and if there is an attack on you, you can send "all" of your bitcoins to the robber. In the same way, you can have different accounts for different needs using different fingers and toes. But you should carefully choose your fingers, because, for example, if you use your middle toe to unlock the wallet, the attacker can easily guess that you are trying to trick him.