I never had good luck with oil -- heat transfer coefficient is 3 to 4 times worse for oil, it's messy, it wicks, is more viscous (doesn't flow as well) etc., etc. Oil is also a pain to circulate well in the tank itself, especially around heatsinks designed for forced air, and don't forget that it wicks. And it wicks. Drop a wire into the tank, and have a puddle on the floor in the morning. 4realz. Anything dipped in oil becomes a pain to rework, no matter how much solvent you use (trust me, it will get on the board even if the board itself is not submerged. And it *does* have a smell. There's fun stuff i've only read about -- boiling point ~30 - 40 C, but it costs way too much. Something like that.
It just needs to be properly planned and executed. The wicking is easily overcome by gravity and wick stops. Usually stuff you plan to submerge, you don't plan to bring out unless it fails. You can get it in bulk, like $2 a gallon or something, even less in some places. As for flow, inexpensive aquarium powerheads do the trick with ease. Mineral oil, and variant oils, are used in high end server racks all over the world. Just needs to be done well.
I wonder if that would happen - heat dissipation from the circuits -> oil -> glas should be sufficient if you use a fishtank. oil has a heat dissipation of ~0,13 (a fifth of water) and glas is at around 0,76. If you take the surface of a fishtank to be around 1 m² the temperature should even out at max. 65°C if your room temperature is around 20°C
It still needs a method of transfer. Air bubbles, generous surface movement, fans blowing on the glass, heat exchanger such as radiator/etc. Otherwise with units like these, it'll cook itself. Without it, the heat doesn't dissipate, it collects in the bath. You can put a mild CPU (think single, maybe dual core, few years old) in there, but that's it. Not ASIC chips. You'd need a generous thermal exchange plan.
Oil is great if you plan it out. Super quiet and neat, and easier than watercooling systems (when dealing in bulk large pieces of equipment like this). For just a few chips I might consider water cooling. If I was serous about something other than air cooling for many units, I'd just design a simple submerge tank for oil.
I've always wanted to do a mineral-oil to salt water pool cooling datacenter hahaha. I have a 45,000 gallon pool which can easily be used to absorb heat from some racks, just need to have the oil heat exchanger inside of a closed circuit of the pool's system. Plenty of thermal capacity, and it'll warm the pool up for the crisp winter use.
