As I stated you're seeing an apparent Sun (not the Sun's actual position) from the perspective of an observer on the ground. The atmosphere creates a lensing effect due to refraction and, perspective along with convergence also effects the apparent position of the Sun or any celestial objects that are visible. The only time refraction doesn't come into play is when viewing an object directly above you at 90 degrees.
The next time you're swimming dive underwater and look up at objects on the surface, again it's not the most accurate simulation but you'll get an idea of how refraction works. Try to grab objects above the water, you'll find they're not where they appear to be.
There is a problem with your claim that the sun only "appears" to set because of refraction. The effect of the atmospheric refraction is the opposite of what you claim. The refraction by the atmosphere causes the sun to look
higher than it actually is, and not lower. Refraction by the atmosphere doesn't explain why the sun goes below the horizon.
However, refraction by the atmosphere
does explain why the sun flattens at sunset. It is because the bottom of the sun appears higher that it actually is.
Next time you are swimming, try your own suggestion. I think you might be surprised to find that the objects above the water are
lower than they appear.