Bad idea IMO.
Why? Because precious metal do not pay dividends, therefore there is no compounding with time, ZERO payout, ZERO return until you sell. This is a HUGE downside to metal.
Also, look at the last 40 years, silver has only peaked twice. Do you really want to bet that it'll happen again and that you can time it?
You are basically betting you can do that against a practically guaranteed compounded return by investing in an high yield index share fund or similar.
Given your age you should be taking advantage of compounding interest...