CompactHashMap/CompactHashSet/MapMakerInternalMap native deserialization triggers uncontrolled memory allocation

Summary

CompactHashMap, CompactHashSet, and MapMakerInternalMap implement readObject(ObjectInputStream) by reading an attacker-controlled elementCount or initialCapacity from the stream. On the very first insertion during or after deserialization (for the CompactHash* collections, inside allocArrays()) or immediately during deserialization (for MapMakerInternalMap), the implementation eagerly allocates arrays sized to that full, unvalidated value (up to CompactHashing.MAX_SIZE = 2^30-1, ~1.07 billion, for example) before the rest of the stream's actual content has been consumed. A 72-byte malicious serialized stream containing a single tiny element can force a multi-gigabyte allocation and crash a JVM with OutOfMemoryError, dynamically confirmed against the current release.

Details

CompactHashMap.readObject() (guava/src/com/google/common/collect/CompactHashMap.java):

private void readObject(ObjectInputStream stream) throws IOException, ClassNotFoundException {
  stream.defaultReadObject();
  int elementCount = stream.readInt();
  if (elementCount < 0) {
    throw new InvalidObjectException("Invalid size: " + elementCount);
  }
  init(elementCount);
  for (int i = 0; i < elementCount; i++) {
    K key = (K) stream.readObject();
    V value = (V) stream.readObject();
    put(key, value);
  }
}

elementCount is only checked for being non-negative; there is no upper bound and no relationship enforced between the declared count and how much data actually follows in the stream. init(elementCount) (line 251) just stores the value (clamped to [1, CompactHashing.MAX_SIZE], i.e. up to 2^30-1):

void init(int expectedSize) {
  Preconditions.checkArgument(expectedSize >= 0, "Expected size must be >= 0");
  this.metadata = Ints.constrainToRange(expectedSize, 1, CompactHashing.MAX_SIZE);
}

The real allocation is deferred to allocArrays(), which fires lazily on the first put() call — i.e. after only one key/value pair has actually been read off the wire:

int allocArrays() {
  ...
  int expectedSize = metadata;
  int buckets = CompactHashing.tableSize(expectedSize);
  this.table = CompactHashing.createTable(buckets);
  setHashTableMask(buckets - 1);
  this.entries = new int[expectedSize];
  this.keys = new Object[expectedSize];
  this.values = new Object[expectedSize];
  return expectedSize;
}

With expectedSize near CompactHashing.MAX_SIZE (2^30-1), this single call allocates a hash table plus three expectedSize-length arrays — multiple gigabytes — triggered by a stream that has, at that point, only supplied one small real element. CompactHashSet.readObject()/allocArrays() have the identical shape. CompactLinkedHashMap/CompactLinkedHashSet extend these classes and inherit the same unguarded path.

The problem with MapMakerInternalMap is similar, with a size read from the serialized data and stored in a MapMaker instance for later user during deserialization, when the MapMaker is passed back to MapMakerInternalMap and segments are allocated by calling newSegment, whose implementations (example) create Segment instances (example) that allocate arrays of the requested capacity through the newTable helper.

This is the same amplification shape as the already-fixed CVE-2018-10237 (AtomicDoubleArray/CompoundOrdering: attacker declares a large size, library eagerly allocates before validating against actual stream content). That fix rewrote AtomicDoubleArray.readObject() to grow an ImmutableLongArray.Builder incrementally per element actually read, rather than pre-allocating from the declared size — Guava's own HashBiMap.readObject() documents the same defensive principle explicitly in a comment: init(16); // resist hostile attempts to allocate gratuitous heap. CompactHashMap/CompactHashSet do not follow either of these two patterns already present elsewhere in the same codebase.

Reachability: All classes are top-level, implements ... Serializable, and require no special classpath setup beyond Guava being present (near-ubiquitous on JVM backends). A native Java deserialization attacker names the class directly in a hand-crafted stream — no particular Guava factory method needs to have been used by the victim application, exactly as with the precedent CVE.

PoC

Full source and run instructions in the attached poc-guava-compacthash-dos.zip (Maven project, guava:33.7.1-jre — the current latest release — as the only dependency). The PoC:

1. Uses reflection only to legitimately obtain a CompactHashSet instance (it is package-private; a real attacker does not need this step, since they craft the serialized bytes directly rather than calling Guava's own serialization code).
2. Serializes one set with 1 element and one with 2 elements, diffs the two byte streams to locate the exact offset of the elementCount field (byte-precise, not guessed).
3. Patches that one 4-byte field in the 1-element stream from 1 to 0x3FFFFFFF (CompactHashing.MAX_SIZE), leaving the rest of the 72-byte stream (one real "A" element) untouched.
4. Feeds the resulting 72-byte malicious stream into a plain new ObjectInputStream(...).readObject() call against the real, unmodified guava-33.7.1-jre.jar.

Actual run output:

### -Xmx8g (generous heap) ###
Located size field at offset 63, value=1 (expected 1)
Malicious stream size: 72 bytes
Declared elementCount: 1073741823 (~1.07 billion)
Real payload: 1 tiny string element ("A")
=== RESULT ===
Exception: java.lang.OutOfMemoryError: Java heap space
Time to allocate+fail: 4013 ms
Heap used before: 3 MB
Heap used after (at failure point): 4103 MB
JVM max heap (-Xmx): 8192 MB

### -Xmx128m (ordinary service-sized heap) ###
Located size field at offset 63, value=1 (expected 1)
Malicious stream size: 72 bytes
Declared elementCount: 1073741823 (~1.07 billion)
Real payload: 1 tiny string element ("A")
=== RESULT ===
Exception: java.lang.OutOfMemoryError: Java heap space
Time to allocate+fail: 24 ms
Heap used before: 4 MB
Heap used after (at failure point): 2 MB
JVM max heap (-Xmx): 128 MB

A 72-byte input forces ~4.1 GB of real allocation before failing on an 8 GB heap, and crashes a 128 MB heap in 24 ms — over 50,000x amplification between input size and forced allocation.

To reproduce: mvn -q compile dependency:build-classpath -Dmdep.outputFile=cp.txt && java -Xmx128m -cp "target/classes:$(cat cp.txt)" poc.Poc

Impact

Uncontrolled resource consumption (CWE-770 / CWE-400) via native Java deserialization: any application that runs ObjectInputStream.readObject() on data reachable by an attacker, and has Guava on its classpath (true for a large fraction of the JVM ecosystem), can be crashed with a single ~72-byte payload, regardless of what type the application itself expected to deserialize.

Severity

CVSS 3.1: AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:N/A:H — identical vector to CVE-2018-10237, the precedent this is a sibling of. Recomputed with the full formula: exploitability = 8.22×0.85×0.44×0.85×0.85 = 2.221; ISS = 1-(1×1×0.44) = 0.56; impact = 6.42×0.56 = 3.595; base = roundup(5.816) = 5.9 (Medium).

Affected

com.google.guava:guava, all editions (-jre/-android) — confirmed on the current latest release, 33.7.1-jre. CompactHashMap/CompactHashSet/CompactLinkedHashMap/CompactLinkedHashSet have had this same readObject/allocArrays shape for a long time (probably since they were added in Guava 23.6), and similarly for MapMakerInternalMap (which was introduced in Guava 4.0); not tested against older individual releases.

Suggested fix

Apply the same defense already used elsewhere in this codebase: either clamp the initial init() capacity to a small constant regardless of the declared elementCount (as HashBiMap.readObject() already does — init(16); // resist hostile attempts to allocate gratuitous heap) and grow incrementally as elements are actually read, or grow the backing arrays incrementally per iteration of the deserialization loop (as the fixed AtomicDoubleArray.readObject() does via ImmutableLongArray.Builder) instead of eagerly sizing to the full attacker-declared count on first insert.