I tend to carry a minimum of tools with me on road rides. A 2 hr ride is "short". A long ride is ~8 hrs. In the small saddle bag is one tube, old metal tire levers, and a patch kit (replaced every few years because the glue dries out), with a Zefel HP frame pump. My Swiss army knife is usually with me as well, plus the cell phone. I usually keep my wallet and cell phone is sandwich bags to protect them from sweat or rain. My next road bike will probably be a touring bike since that tends to be the kind of road riding I do (long, solo rides with less concern about all-out speed).
Mtn biking is another matter entirely, but will depend on if I'm racing or just riding with friends.
My "ride kit" consists of a Vaude pack with a NLA Park tool pouch (chain tool, tire levers, dog-bone multi-wrench, spoke wrench), the SA knife, Leatherman, a folding hex key set, RD hanger, (2) tubes, patch kit, compact pump, emergency TP, a length of duct tape, snake-bite kit, more food than I plan to eat, 100 oz bladder and sometimes a couple of water bottles as well. I often carry a spare rear derailluer, too. Yes, my pack has been known to be quite... hefty...
The "race kit" uses a smaller pack with a 70 oz bladder, the Park pouch, SA knife, RD hanger, tube, patch kit, CO2 inflaters and reserve food. Gels are put under the leg elastic of my shorts with maybe a power bar in a jersey pocket.
My DH ride kit will vary depending on the ride. For lift-serve, a light pack with just water. For shuttling, usually a bit more, but generally just enough to get me back to the pick-up point and then back to my car where I'll usually have my full tool set, folding work stand and a tub of spares.