Why do simple containers not expand around floated contents?
One of the supposed 'frustrations' with CSS floats is a container's refusal to expand to encompass any floated elements within it:
Some text following the image
Note that the light grey border surrounds a paragraph containing both the image and text. There are several ways to 'expand' the container (e.g. overflow-auto) to achieve the desired effect:
Some text following the image
However, many web designers question the need to do this in the first place: why do float containers behave so strangely by default? I'd often wondered this too, until the answer became blindingly obvious. To reason about this behaviour, we must first understand what the float property was originally intended for.
Whilst floats have become increasingly common to achieve all sorts of advanced layout techniques — many centered around the reproduction of columns — the original design rationale is a far more mundane one: image layout.

The above image from the W3C demonstrates the 'classic' application of floats. An image is placed on the page, and text following it 'wraps around' that image, appearing immediately to the right, flowing down until there is space below the image in which to continue. This is effectively a reformulation of the bad old align="left" HTML property.
The following W3C image perfectly demonstrates why containers do not encapsulate their floats by default:

In this image, we have just two paragraphs: one containing an image and some text, the next containing just text. In most cases, we would want the second paragraph to float around the image, just as the previous text did.
Here's a real-life example:
A paragraph of text before the image
Some text following the image
Another paragraph of text
Another paragraph of text
Without the default behaviour, this type of effect would be impossible to achieve; an image could only ever float alongside content within its own paragraph:
A paragraph of text before the image
Some text following the image
Another paragraph of text
Another paragraph of text