There are a couple of good shops in the Gold Souk that only sell bullion. Mostly Pamp or Valcambi bars, and if you like silver they sell Emirates Silver 1Kg and 500g.
I've also seen small gold rounds, but they are I think aimed at Indian market and not something I recognize. I don't see many coins if any.
One is called Lakoo's and the other I can't remember right now. I'll check when I get home and update you.
They are much cheaper than at the airport, and there is no question about getting ripped off, all they do is sell bullion and I have never heard of any fakes. Not sure about the sovereigns, I've never seen them here.
There is a vending machine is at Emirates Palace in Abu Dhabi, I didn't know there was one at Burj Kaleefa, will have to go check it out.