The person who is selling gold for bitcoins will still sell those bitcoins on an exchange for legacy currency so he can re-stock his bullion supply and pay his expenses/cost of living, so the net effect is the same. Until that gold seller can re-stock his inventory by buying gold at wholesale prices using only bitcoin, and pay his rent, taxes and put food on his table using only bitcoin, the bitcoins will still need to be exchanged for legacy currency by someone in the chain.
But there will be a definite lag in the system before the value of Bitcoin takes a hit. What I'm trying to say is that the longer that lag is drawn out the less impact it will have on the market.
