All I can really add to this is something my college geology professor said: When mother earth is done with us, she'll kill us.
Feel free to replace "mother earth" with whatever deity, calamity, or belief you wish.
My big gripe is what I call artificialization of material. We produce more corn than we can possibly consume, but in an effort to maintain the noble pursuit of farmers' jobs, the government buys the surplus and lets it rot somewhere. As a former dairy/corn/chicken farmer, I take some serious offense to the fact that there are billion-dollar corn and soybean farms supported by my tax dollars, but thousands of PA dairy farms are selling off cattle for hamburger because it costs more to produce milk than you get when you sell it. I think the current figures are about a 10-cent loss per gallon. Dairy farming in PA (and likely other places) is not only a non-profiting job, you pay to make milk.
Then I get even gripier (that's a real word) when everything in our economy has become globalized. I'm all for global trade, but we have made this odd structure where it is easiest and cheapest to (for instance) mine nickel in Canada, ship it to India for refining, then send it to China or Japan where it is plated onto circuit boards, then it's shipped to Korea where Hyundai puts it in a car ECM, then the car gets shipped to Argentina for sale.
Then I get gripiest (another real word) when we can't just eat food. We don't eat a green bean or a wild Salmon filet or a fresh local sausage. We eat canned beans with six ingredients on the label, a farm-raised, color-injected, inbred salmon, and a hot dog made from lips and buttholes that is more Nitrites and preservatives than it is meat.
We have engineered our food to the point where it isn't really food anymore. I think it likely contributes to illness, disease, and death. I'm not saying you'll die if you eat a Ballpark frank, but a lifetime of that stuff could certainly alter the population's average lifespan.
I think what I'm really saying is (to piggyback on the OP's 50% vs 95%) I think THAT is the big killer if we lose a significant part of our population. I think people can live without infrastructure easier than they can live without access to hot dogs, canned food, and farmed meats. As for me, I have a .308 and a farm with deer, 5 acres ready to plant, and a fire pit to cook. Bring on the apocalypse.