I'm puzzled by that. $2-$5 over melt/spot is what people normally pay for bullion (sometimes less than $2). Are these the right numbers in your case? Given the system you described, I'd expect you to do much better than $2-$5 over spot. I thought the cash-for-gold places were giving you a special deal in order to sell their inventory quickly...
Some premiums over spot are fixed, as @goldpelican outlined. For example, the US Mint charges its authorized wholesale purchasers $2 per ASE over spot regardless of volume. (And that's with a minimum order of 25,000 ounces...) And there are only a handful of US Mint authorized purchasers, reducing competition. (The US Mint makes it very hard to buy bullion from them – you have to jump through all kinds of hoops, be a major player in the industry for a number of years, produce financial statements, etc. They don't just take your money for their product like a normal business – they've created a cartel.)
You missed the part where I said really cool bullion, like Australian series 1 and 2 silver lunars, Kangaroo 1 oz coins in the box with COA (before they started mass production), proof kookaburras, Britannia silver 4 coin proof sets, Birds of Australia from the late 80s into early 90s, etc.....