Ask yourself what is wrong with the world at the moment?
Now ask yourself if debt is any kind of answer?
Debt and interest is the root cause of all troubles and more debt is never the answer.
The only reason you would consider debt to fund physical PMs is if you have no savings, in which case you are probably already in debt. Doubling up on debt is nuts and isn't saving, it's speculation on short term gain.
Buying only what you can afford is always the answer. Get out of debt, then save.
All increases in buying pressure in Silver, including you borrowing to buy, will help add to the paper profits of other speculators and they will encourage people like you to get silver no matter what. However, this is a ponzi, with people who can not afford what they have, encouraging others to buy what they again can not afford just to drive the price higher so they can take some profit or feel good about their paper profits. Not unlike the Aussie housing market. But the more people like you say I want to borrow to buy physical silver "for the long term", the more people like me start to think it is time to get out because prices are going to fall when the "last fool" (a ponzi term, no disrespect) enters the market.
If you want to bet on the direction of silver, but need to liquidate in a hurry in a downturn, try an ETF (asx: ETPMAG) with a stop loss limit applied.
If you want the security of physical metal in your possession, buy only what you can afford and accumulate a position over the long term that you own outright.
Edit: Here the fact sheet on ETPMAG.
http://www.asx.com.au/documents/products/etfs_fact_sheet_physical_silver.pdf
Disclosure: I don't own ETPMAG, but the more speculators motivated by greed that enter the market, the more I am thinking about it with trailing stop losses on the way up.