GHSA-xj96-63gp-2gmr: Pillow: Heap out-of-bounds write in `ImageFilter.RankFilter` via integer overflow in `ImagingExpand`

Summary Pillow's public rank-filter API can trigger a native heap out-of-bounds write when given a very large odd filter size. Minimal public API trigger: from PIL import Image, ImageFilter im = Image.new("L", (3, 3), 128) im.filter(ImageFilter.MedianFilter(4294967295)) ImageFilter.RankFilter.filter() calls image.expand(size // 2, size // 2) before rank-filter size validation. With size = 4294967295, the expansion margin is 2147483647 (INT_MAX). ImagingExpand() then computes the output dimensions with unchecked signed int arithmetic. On tested builds, this wraps to a tiny output image and the border-expansion loop writes past the allocation. This is reachable through documented public classes (RankFilter, MedianFilter, MinFilter, and MaxFilter). No private API, ctypes, or custom Python object is needed. Details Current src/PIL/ImageFilter.py: class RankFilter(Filter): def filter(self, image): if image.mode == "P": msg = "cannot filter palette images" raise ValueError(msg) image = image.expand(self.size // 2, self.size // 2) return image.rankfilter(self.size, self.rank) The expand() call is made before image.rankfilter(...). Current src/libImaging/Filter.c:ImagingExpand() does not check output-size overflow: if (xmargin < 0 && ymargin < 0) { return (Imaging)ImagingError_ValueError("bad kernel size"); } imOut = ImagingNewDirty( imIn->mode, imIn->xsize + 2 * xmargin, imIn->ysize + 2 * ymargin ); For a 3x3 image and xmargin = ymargin = INT_MAX, the computed output size wraps to 1x1 on tested builds. The following loop still uses the huge margin: for (x = 0; x < xmargin; x++) { imOut->image[yout][x] = imIn->image[yin][0]; } src/libImaging/RankFilter.c does contain checks that would reject this size: if (!(size & 1)) { return (Imaging)ImagingError_ValueError("bad filter size"); } if (size > INT_MAX / size || size > INT_MAX / (size * (int)sizeof(FLOAT32))) { return (Imaging)ImagingError_ValueError("filter size too large"); } But those checks are reached o