Introduction

In Java, the memory used by a process is a composite of several components, each serving a unique purpose in the functioning of the application. This memory can be defined as follows:

Java process memory usage = Committed heap + committed non-heap + NUMBER_OF_THREADS * ${-Xss} + Native allocations

1. Committed Heap: This is the actual memory reserved by the JVM for the application. It includes all objects created by the application, including those currently in use and those that are ready to be garbage collected.

2. Committed Non-Heap: This is the memory used by Java to store metadata about the application, the JIT compiled code (code cache), loaded classes, and other non-heap data.

3. Thread Stacks: Each thread in a Java application has its own stack, which is used for storing the state of method calls and local variables. The size of each thread stack is determined by the -Xss JVM option. The total memory used by all thread stacks in the application is the product of the number of threads and the -Xss value. The default value in last 64 bits JVM versions is 1 MiB.

4. Native Allocations: This refers to memory allocated by the application through native code, outside of the JVM's control.

In Besu, most of the native allocations are related to RocksDB.

The problem

During our burn-in tests for the 23-7-3-RC release, it was observed that this version consumes more memory compared to prior releases.

This particular release comes with the GC flags enabled by default, which are designed to minimize Besu memory usage (-XX:G1ConcRefinementThreads=2 -XX:G1HeapWastePercent=15 -XX:MaxGCPauseMillis=100).

However, these flags are not present by default in the 23.7.2 release. The comparative memory usage between 23.7.2 and 23.7.3-RC can be seen in the screenshots provided below.

Our tests are conducted on Azure VM with 16 GiB RAM, and all nodes are rebooted every Monday. This is the reason for the observed drops in memory usage on the 4th, 11th, 18th, and 25th of September.

...

One reason for the discrepancy between the heap dump and the current heap usage depicted in the graphs is that the JVM performs a full garbage collection (GC) prior to generating a heap dump.

 it's observable that activating the new transaction pool has resulted in increased allocations in the heap. The PeerTransactionTracker consumes more space due to the seenTransactions hashmap tracking a larger number of transactions, with the maximum being MaxPeers * 100,000.

Additionally, it's noticeable that the 23.7.3-RC release has a larger number of transactions in the heap, with 76,439 transactions occupying 144 MiB, compared to the 23.7.2 release, which has 4,030 transactions taking up 23 MiB.

While we have identified a difference in memory usage between the 23.7.2 and 23.7.3-RC releases, the increase in heap usage, which resulted in a larger heap committed space, does not account for the entire discrepancy in memory usage.

3. Threads' Stack comparaison

Let's proceed to examine the variation in the count of threads. As depicted in the screenshot below, it's clear that the number of threads is identical between the two releases.

Image Modified

4. Native allocations compraison

...

The off-heap memory usage appears to be very similar between the two releases.

Conclusion

Based on the detailed analysis of each component of the Java process memory usage, it's evident that the difference in memory usage at the process level between the two releases is primarily due to the new allocations associated with the new transaction pool.

To summarize:

• The committed heap and non-heap memory usage are higher in the 23.7.3-RC release, primarily due to the new transaction pool.
• The number of threads, and consequently the memory usage associated with thread stacks, is identical in both releases.
• The off-heap memory usage is very similar between the two releases.

Therefore, the increased memory usage in the 23.7.3-RC release can be attributed to the new allocations related to the new transaction pool. This finding underscores the importance of careful memory management and optimization, particularly when introducing new features or changes to the system.