All this comes down to the existence of a cheap (excluding environmental effects) resource in the form of coal and gas being available, thus bringing about the natural human tendency to want to use it rather than using something more costly.
Whilst there is certainly an environmental cost with coal, it's not a cash cost that consumers are paying today. So as with anything financial, there's the natural tendency to leave someone else to foot the bill when it arrives, safe in the knowledge that you won't be alive when most of it arrives.
Fossil fuels provide a massive leverage of human effort. Relatively few people are employed in order to produce a vast amount of energy, which then does the work of
billions of people even in countries with a relatively small workforce.
Here's some facts and figures to illustrate the point. I've used data for Tasmania since I have the figures, but the same principle applies in any developed society (with the exception of the point that Tas uses more hydro and wood, and less coal and gas, than is typical of most places).
Oil = equivalent to a workforce of 80 million people.
Hydro-electricity = equivalent to a workforce of 62 million people.
Natural gas = equivalent to 31 million workers.
Wood (used as fuel) = equivalent to 29 million workers.
Coal = equivalent to a workforce of 17 million.
Wind = equivalent to 7 million workers.
Landfill gas = equivalent to 200,000 workers.
Solar electricity (rooftops) = equivalent to 135,000 workers.
Total workforce = equivalent to 226 million human workers, of which only 0.1% are actual humans.
So we have Tasmania, population just over 500,000 people, with an effective workforce of 226 million. And only about 1% of the state's actual human workforce is employed in the energy industry in order to make this happen. Even the rotting garbage at a few rubbish tips in Hobart and Launceston contributes almost as much as the entire human workforce.
That's the sort of leverage I'm talking about and it's much the same everywhere. This is what makes our current lifestyle possible - we do 0.1% of the work ourselves, and get our almost free energy slaves to do the other 99.9%. Take that cheap energy away and everything falls in a heap real quick - there's no way that humans could do even 1% of what we get done with cheap energy today.
Now here's the problem. The energy sources we use presently don't require many people in order to operate. You have a few hundred workers running the hydro system producing the equivalent labour of 62 million. Or you have a few hundred workers at a coal-fired plant in another state doing much the same.
The trouble is that "alternative" energy sources typically offer less leverage of human effort. It takes more people to produce the same power in a reliable manner. We have to put more effort into it, thus diverting human labour away from something else and that leads directly to reduced living standards.
As for what could reasonably work as a solution, I'd be putting my efforts into geothermal, solar thermal, wind and hydro in the Australian context with geothermal being the big "game changer" if we can get dry geothermal working satisfactorily. It does need government backing in my opinion, as did coal, hydro and gas to get them up and running, and given the potential benefits it seems well worth putting some serious effort into it if the private sector doesn't get a move on.
Geothermal - we've got massive potential resources in several states within an acceptable distance of major loads. Since it could produce 24/7 baseload power, it's a real game changer.
Solar thermal - there are limits but certainly it's possible to get consistent power from the sun to supply a decent portion of the total load.
Wind - it's intermittent but the resource is pretty decent especially in SA and Tas, and combined with hydro it can indeed provide baseload energy.
Hydro - we could certainly build "another Snowy" if we wanted to, Australia isn't as lacking in hydro resources as seems to be the popular belief. The main role however would be to provide a storage and balancing means so as to integrate solar and wind at high levels into the grid and maintain a constant supply to consumers. There are many potential developments in Qld, NSW, Vic and Tas.
The trouble is, all of those are at present more expensive than coal. Brown coal in particular is incredibly cheap - the cost of mining a tonne of brown coal and running it through the boilers at the more modern power stations in Victoria is almost exactly the same as running water that falls free from the sky through hydro turbines in any existing scheme on a cost per unit of output basis. Running an existing coal plant is incredibly cheap when the fuel itself is worthless and everything is highly automated.
The crux of it is that coal is cheap and plentiful so we use it. If gold was cheap and plentiful then we'd make door knobs and especially electrical cables out of it. But since gold is scarce and expensive to extract, we're forced to accept the use of inferior materials for practically all applications. It's the same with coal - if there was very little coal on Earth and it cost a fortune to mine then we wouldn't be using it to generate electricity, having simply accepted the higher cost of using alternative methods. But since we do have cheap coal, we want to use it.
It's the same with anything. Once you have something cheap, in this case coal, humans don't want to lose it in favour of something that requires more effort and provides less leverage. Just like someone who earns $150 an hour doesn't want a pay cut to $25 and hour, but would be more than happy with $25 if the alternative was to earn nothing. That's essentially the problem we have - there are alternatives but they effectively involve a pay cut and most people tend not to be keen on that idea.