As bitcoin is completely anonymous and decentralized more users think that bitcoin is used for illegal activities. On this basis people avoid it and stand away from it, though it has highest price than any other currency that have existed.
Every currency is used for both legal and illegal purposes. It depend on the activity and the people. At times for legal works, money is used for illegal purposes. It does not require government's authority. It does not happen only with BTC.
I believe is that lack of places to spend bitcoin, In my country i can hardly buy anything from bitcoin, mostly i can just store and exchange and nothing beyond that. If i could live without fiat i would do that, but only thing stopping me from using it is unavailability to spend. I blame this on media and governments. From media we get that everything can be hacked, and we all know the governments part.