linux-user/mmap.c: fix integer underflow in target_mremap

Fixes: https://bugs.launchpad.net/bugs/1876373

This code path in mmap occurs when a page size is decreased with mremap.  When a section of pages is shrunk, qemu calls mmap_reserve on the pages that were released.  However, it has the diff operation reversed, subtracting the larger old_size from the smaller new_size.  Instead, it should be subtracting the smaller new_size from the larger old_size.  You can also see in the previous line of the change that this mmap_reserve call only occurs when old_size > new_size.

Bug: https://bugs.launchpad.net/qemu/+bug/1876373
Signed-off-by: Jonathan Marler <johnnymarler@gmail.com>
Reviewded-by: Laurent Vivier <laurent@vivier.eu>
Message-Id: <20200502161225.14346-1-johnnymarler@gmail.com>
Signed-off-by: Laurent Vivier <laurent@vivier.eu>
This commit is contained in:
Jonathan Marler 2020-05-02 10:12:25 -06:00 committed by Laurent Vivier
parent fd568660b7
commit 257a7e212d

View file

@ -708,7 +708,7 @@ abi_long target_mremap(abi_ulong old_addr, abi_ulong old_size,
if (prot == 0) {
host_addr = mremap(g2h(old_addr), old_size, new_size, flags);
if (host_addr != MAP_FAILED && reserved_va && old_size > new_size) {
mmap_reserve(old_addr + old_size, new_size - old_size);
mmap_reserve(old_addr + old_size, old_size - new_size);
}
} else {
errno = ENOMEM;