If you isolate yourself completely from the surrounding acoustic
environment (eg in an anechoic chamber that absorbs all sound), you can
actually hear the Brownian motion of air molecules beating against your
tympanic membrane (eardrum). It sounds exactly like the ocean from a few
sand dunes away. I suspect that holding a sufficiently large shell, or
cup, or hand, against your ear has some of the same effect, isolating it
from the environment and effectively amplifying the background noise (by
making it more noticeable). It's not quite the same thing, so there
might be more involved, such as reverberation within the cavity of the
shell, but I think what you're hearing is basically noise in the absence
of signal.