I'm curious if you know how a peer to peer exchange work or if you have used a decentralized one. If you have not use one, I will advice you use one before you jump into conclusion. You don't need an intermediary in a decentralized p2p, the system is designed to work without any interference. The seller gives Bitcoin and it's lock waiting for the buyer to pay and once transactions is confirmed by the seller, Bitcoin is release to the seller to claim, it's simple.
To the buyer you meant to say here.
One simple way of interpreting this is simple;
Seller: Bitcoin is locked in escrow by the seller
Buyer: The buyer sends bitcoin to the seller
Seller: Payment is received from the buyer and confirmed
Escrow: Buyer receives the bitcoin from escrow
When there is an intermediary, it's no longer decentralized, that's a middleman which Bitcoin was never a friend. Sadly, everyone prefered centralized exchanges for doing this and it's not even because there is anything special about centralized exchanges, the users refused to used dex to boost liquidity. Most Dex are having liquidity fragmentation, only the privacy ones are fair.
There is another reason for reliance on centralized exchange. Not all direct p2p market place is fully working as a decentralized exchange. How the decentralized escrow mechanism of a DEX is being designed is worthy of being considered when choosing a DEX exchange to use. Eliminating escrow service means accepting likely counterparty risk and DEX doesnt mean it is safe.