Canon has always used the "L" designation the way Nikon has used the "ED" designation: to indicate that those lenses are constructed with Flourite or Ultra-Low Dispersion lens elements. These elements are used to greatly reduce lateral chromatic abberation and longitudinal chromatic abberation, thereby improving lens performance.These materials are more expensive than "normal" optical glass and therefore make the lenses more expensive.
The biggest difference in your two categories of lenses is maximum aperture. Lenses with large maximum apertures are more expensive to manufacture at the same quality standards than a lens with a smaller maximum aperture. They require more elements, more exotic materials, and tighter construction tolerances than slower lenses. Extreme wide angle and super telephoto lenses bring additional issues to the table that make them expensive to make both fast and high performance. Fast lenses require larger elements as well as additional elements. "Bigger" and "more of them" equals increased size and weight.
Regarding price, the 24-70 f2.8/L and the 24-105 f/4.0L are within $100 of each other. The 17-40 f/4.0L is half the price of the 16-35 f/2.8L because of the maximum aperture and the difference in cost it takes manufacture a lens with that aperture and still maintain image quality standards. The same thing goes for the 70-200 f/4.0L and the 70-200 f/2.8L. You're paying to overcome physics.
I can't speak for lens hoods, but you will almost always only find a tripod collar on zoom telephotos and super telephotos where it makes sense to move the center of gravity away from the underside of the camera body.
As for comparisson, the link below address just about every lens you mentioned: