According to the logs, Fatal: Thread death due to error=java.lang.OutOfMemoryError: Java heap space, thread=Thread[autopilot ,server_core]
Fatal: Server may no longer be stable and should restart.
Local fix
The CEP Server has used up all of its memory. This does not mean that all available memory in the system has been used. The JVM starts with a specific maximum heap size. When it uses this much memory, then it is unable to allocate any additional memory. This maximum heap size is configurable, but of course cannot exceed the actual physical memory available in the system.
The size of heap space can be changed by using JVM options, -Xms and -Xmx:
- Xms: denotes starting size of heap.
- Xmx: denotes maximum size of heap in Java plus an additional option for GC stability.
Within the localhost directory there is the ATPNODE.lax file. Locate the line, “Additional JVM command line options,” and edit as follows:
lax.nl.java.option.additional=-server -Xms4096m -Xmx4096m -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=128 -XX:CMSInitiatingOccupancyFraction=50
This example may not be applicable to your java JRE and version and may need to be modified
Restart the CEP server. The hardware/software requirements are in Chapter 1.6 of the Core Services Installation Guide in the meshIQ Platform Documentation Library.
The typical formula for Java heap size is (number of facts + number of sensors) * factor.
Example: (100K facts + 10K sensor)* 4000 = 440MB of heap size is absolutely required. Typically you add an additional 40-50% of play on top of that and you get around 800-1G of heap space allocated for the JVM.