...
This particular release comes with the some GC flags enabled by default, which are designed to minimize Besu memory usage (-XX:G1ConcRefinementThreads=2 -XX:G1HeapWastePercent=15 -XX:MaxGCPauseMillis=100).
However, these flags are not present by default in the 23.7.2 release. The comparative memory usage between 23.7.2 and 23.7.3-RC can be seen in the screenshots provided below. To compare apples to apples, we enabled the same GC flags on 23.7.2.
Our tests are conducted on Azure VM with 16 GiB RAM, and all nodes are rebooted every Monday. This is the reason for the observed drops in memory usage on the 4th, 11th, 18th, and 25th of September.
...
The additional restarts visible in the screenshot are likely associated with manual restarts conducted for specific testing purposes.
Memory usage on 23.4.4
In Like for the case of the 23.7.1 release, the GC flags were activated on September 4th, with automatic restarts taking place on the 11th, 18th, and 25th of September. The memory usage pattern for the 23.4.4 release is quite similar to that of the 23.7.1 release, with a peak of approximately 8.6 GiB prior to the flag settings, and a much more stable usage below 7.5 GiB thereafter.
...
2. Heap committed comparaison
It's evident that the total memory usage ( comprising both heap and non-heap ) is more substantial in the 23.7.3-RC release compared to the 23.7.2 release. Consequently, the committed memory (heap + non-heap) is also significantly higher in the 23.7.3-RC release.
...
sudo jcmd PID GC.heap_dump /tmp/ameziane-test-regression-23-7-3-RC-2-heapdump-29092023-2.hprof
PID:
Dumping heap to /tmp/ameziane-test-regression-23-7-3-RC-2-heapdump-29092023-2.hprof ...
Heap dump file created [969758926 bytes in 5.839 secs] => 925MB
One reason for the discrepancy between the heap dump and the current heap usage depicted in the graphs is that the JVM performs a full garbage collection (GC) prior to generating a heap dump.
...
While we have identified a difference in memory usage between the 23.7.2 and 23.7.3-RC releases, the increase in heap usage, which resulted in a larger heap committed space, does not it account for the entire discrepancy in memory usage .?
3. Threads' Stack comparaison
...
The Besu dashboard does not provide a graph for Besu's off-heap usage. However, based on the formula provided in the introduction, I we have generated a graph using this formula:
Off heap memory = process memory - (committed head + committed non heap + NUMBER_OF_THREADS * 1024 * 1024) // Xss = 1 MiB = 1024 * 1024 Bytes
process_resident_memory_bytes{instance=~'$system',job='besu-node'}-(jvm_memory_bytes_committed{instance=~"$system", area="heap", job=~".*besu-node"}+ignoring(area) jvm_memory_bytes_committed{instance=~"$system", area="nonheap", job=~".*besu-node"}+(jvm_threads_current{instance=~"$system", job="besu-node"}*1024*1024))
This metric will be added to Besu Full dashboard.
The off-heap memory usage appears to be very similar between the two releases.
Conclusion
Based on the detailed analysis of each component of the Java process memory usage, it's evident that the difference in memory usage at the process level between the two releases is primarily due to the new allocations associated with the new transaction pool.
...
Therefore, the increased memory usage in the 23.7.3-RC release can be attributed to the new allocations related to the new transaction pool. This finding underscores the importance of careful memory management and optimization, particularly when introducing new features or changes to the system.