The rich already have their swiss bank accounts, tax consultancy services, good investment opportunities etc. Bitcoin offers nothing for them.
As far as good investment opportunities go, the "rich" care more about preservation of capital than about explosive growth. Thus, if the Bitcoin price stabilizes or grows steadily against USD, and if USD experiences moderate to heavy inflation, then Bitcoin might look very good.
This is assuming that the exchanges OTC markets can handle a large influx of capital -- on the order of ~$1B.
Right now on MtGox, if I were to spend $1M, I would buy up the order book from 6.75 all the way to ~$15, leaving a potential gap straight back down to 6.75 (and an instant loss of ~50% of my value).
Contrast this with the foreign exchange markets that handle $ trillions per day and rarely move more than 1% per day.
So I guess my bottom line point is that once there is larger money on both sides of the book, BTC will look good to large-scale investors.