In my opinion is the association with the dark web and illegal dealings happening that uses bitcoin as a medium of exchange, that keeps people away from BTC. Lots of my friends even think that BTC itself is illegal to own and use, which is definitely not true in my country.
Perhaps also the price fluctuations and the lack of understanding about the concept of BTC. Public figures like Peter Schiff think that BTC is a huge bubble that is destined to pop. And many investors view BTC as a risky investment.
What people thinks about bitcoin does not affect our community that much. They only say that because they do not have enough knowledge about bitcoin. All they know is that it's an online currency. People like to look at the negative side of something just to sound intelligent, so they kind of just mention bitcoin being used in darknet which is true and that leads them to just ditch their interest with bitcoin.
prohibition of the government that may stop people using bitcoin
I am pretty sure that the government can NOT affect much the growing community of bitcoin. It will be foolish if the government ban bitcoin in their country just because the government supports fiat money. Bitcoin is supposed to be used anonymously so if ever the government prohibits bitcoin in their country, a user only needs a VPN to look like he's from another country and he'll always remain anonymous.