Re: maximum memory limit

[Jamie Lokier <lk@tantalophile.demon.co.uk>]

Stephen C. Tweedie wrote:
> libc can easily mmap /dev/zero in chunks of multiple MB at a time if
> it wants to, and can then dole out that memory as if it was a huge
> piece of the heap without kernel involvement.

Several allocators already do that (including GCC's), but Glibc doesn't
AFAIK.

> You cannot have an arbitrarily growable heap, AND an arbitrarily
> growable stack, AND have the kernel correctly guess where to place
> mmaps.
> 
> One answer may be to start mmaps down near the heap boundary, and to
> teach glibc to be more willing to use mmap() for even small mallocs.
> That may break custom malloc libraries but should give the best
> results for code which uses the standard glibc malloc: it doesn't
> artificially restrain the stack or the mmap area.  Anything
> relying directly on [s]brk will be affected, of course.

There are lots of custom malloc libraries.  If you're going to teach
Glibc something anyway, why not add a new mmap flag?

One flag I think would be quite useful is MAP_NOCLOBBER|MAP_FIXED: there
are times when I'd like to be able to _try_ mapping something at a fixed
address, but fail if there is something mapped there already.  Currently
it's necessary to parse /proc/self/maps, and that's far from thread
safe.

The obvious use is for pre-relocated shared libraries, which can run at
any address but will load faster and share more pages if loaded at a
specific address.  (Especially if they're non-PIC).

As a natural extension, a map flag which says "try to map at the
supplied address, but if there is an object there search for a big
enough hole above the supplied address" would simultaneously be useful
for malloc optimisations like you're suggesting, and pre-relocated
shared libraries.  That would be MAP_NOCLOBBER without MAP_FIXED.

-- Jamie
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/