I like what you have on the site currently.

The forex method might be the standard for exchanges but these are used by traders all the time.

A standard rule for any user interface is "Don't make me think!"

Like most people on here, I'm an IT nerd so maths, rates and calculation make sense to me but I work with artists all day long and their math skills tend to be poor. My first thought would be how they would respond to the three options. When people change up money for holidays they tend to think of how many euros or dollars etc they're going to need but they also might want to think about it in terms of their home currency but I think they rarely think in terms of rates and the notion that they're either buying/seling euros etc doesn't even come in to it.

My preference would be to put in the number of GBP I want to sell and for the interface to calculate the resultant BTC at the current exchange rate and then to be able to adjust the rate and/or the BTC you're wlling to receive. IMO, That's a more natural way of thinking about it. I guess this is effectively all three methods at once - you can change any parameter and it will recalculate the other params based upon which one was changed. How that's stored in the DB is irrelevant to the user but I guess the forex format of storage is probably the best.

I guess the UI might look something like...

Qty1 * currency_seletor = rate * Qty 2 * currency_selector_2

If you update either quantity then the rate is recalculated. If you change the rate then Qty2 is recalculated. Put that with some nice Ajax currency graphics and you're on your way. It would also be good if it let you know if your order matches any in the orderbook.

Surely this is a solved problem in the finance industry?

I think I like 3 the best but I'm still not entirely sure what the problem with it is.

Your calculations end up with an amount of BTC and an amount of GBP.

The BTC amount is taken from one person and given to the other, and the GBP amount is taken from the 2nd and given to the 1st.  It's a zero sum game as far as you are concerned.  So where's the 'absorb that cost' bit?

When the division isn't exact, one person loses less than 0.01 uBTC and the other gains it.  I don't see how the exchange is absorbing anything.

That's my point. It's totally insignificant as far as I'm concerned, and the exchange operator could absorb that cost if need be.

I think what Genjix might be saying is that it would be possible to round the result in the favour of both sides of the trade so they both gain a fraction of a 'satoshi', and the exchange loses a satoshi on each trade.

Currently one side gains a fraction and the other side loses a fraction, but it would be possible (and very cheap) for both sides to gain.  Then you could claim to be the only exchange with negative commission!