Your last paragraph hit the nail in the head. That is my biggest quibble to ANY magic energy source, that we will run out of recoverable minerals to build out our civilization from - no matter what we do.

Minerals are located in concentrated reserves, which once mined never return. And as we run out of cheap resources the energy needed to recover the not so accessible reserves rises exponentially - and we will soon find ourselves in a rat race where we need to mine ever more metals to build ever more nuclear reactors, renewables and such to keep up with the rising energy cost of mining.

We never realize though, that this is a race we cannot win: the energy demand of mining will soon spiral out of hand and lead to fall in mineral production, and fall in material standards of living as well.