There's a couple of reasons bullion dealers don't take cards.
Firstly, there's the additional costs. Card payment costs to the merchant can range from 0.2% to 1.5%, which is fine if you're selling something with a 40% markup like a jeans or a sandwich press but not so fine if you're making 1% on several ounces of gold. Of course there's always the "just pass it on to the customer" argument but if 1% of $10,000 is $100 then the deal can easily go from pretty good to pretty average. It's generally not worth it for the dealer to introduce an extra layer of complexity into their order flow. There are usually other fees involved in addition to the per-transaction/value fee as well.
Secondly, the rules on charge-backs are complicated and "CNP" (card not present) transactions are particularly risky for the merchant. If you get a charge-back for an order with a 5% margin, you're doing the next 20 orders "for free" just to make up for the loss.
On one hand, charge-backs are great for consumers because they protect against dodgy merchants, but on the other hand they don't protect merchants from dodgy consumers. Even if the customer isn't dodgy, they've been conditioned to report any problems with purchases and just let their bank/card issuer take care of it, so in that respect people have become a bit lazy.
(I've had customers say "something went wrong with the payment" and showed me the receipt where it clearly states that the transaction has been automatically reversed. I say "it's fine, you weren't charged" and then a few days later I get a charge-back because they reported it "just in case" and then I have to spend time arguing with the payment processor and pointing out that I never received the funds being claimed so I'm not giving the money back when I never had it in the first place. Then we have to get into terminal log reports and a bunch of other time wasting stuff).
Charge-backs can occur up to 120 days after the transaction has taken place, so it's a long time to wait to find out if you actually really have that money or whether it's at risk.
Basically, it's just not worth the hassle for the dealer: cards are expensive, complicated and risky which is pretty much the exact opposite of what dealers are trying to achieve. It's bullion. Treat it as cash. You give me your cash, I give you my cash at the agreed rate and everyone's happy.