Информация:

Количество виртуальных машин = 4, каждая с экземпляром Tomcat 8.5. * В кластере Приложения = 5 военных приложений - 2 приложения UI и 3 веб-сервиса. Версия Java = java 1.8. * Configuratin = 2 LTMS и 2 веб-сервера Apache - 1 LTM поверх кластеров tomcat, который обрабатывает запросы веб-сервисов (200K + в день в будние дни). Еще один LTM поверх веб-сервера, который также находится на вершине экземпляров Tomcat, который обрабатывает запросы приложений пользовательского интерфейса (более 10 тысяч запросов в день). JVM parameters: all default with -Xms3072m -Xmx3072

Tomcat configuration:

 Connector port="xxxx"                 
       protocol="HTTP/1.1"  
           connectionTimeout="3000"  
           enableLookups="false"  
           redirectPort="yyyy"  
           maxThreads="80"   
Connector port="yyyy"  
          protocol="org.apache.coyote.http11.Http11NioProtocol"  
          redirectPort="yyyy"  
          secure="true"    
          scheme="https"  
          clientAuth="false"  
          sslProtocol="TLS"   
          sslEnabledProtocols="SSLv2Hello,TLSv1,TLSv1.1,TLSv1.2"  
          SSLEnabled="true"  
          maxThreads="70"  
          maxKeepAliveRequests="100"  
          keepAliveTimeout="5000"  
          connectionTimeout="10000"  
          keystoreFile="....."  
          keyPass="..."  
          keystorePass="..."  
          keyAlias="....."  
          truststoreFile="..."  
          truststorePass="..."  
          ciphers="......."        
   Connector port="zzzz"
     scheme="https"     
     protocol="AJP/1.3"     
     redirectPort="yyyy"     

Issue: We are having to recycle tomcat every once a week, and we do it during the weekend. IF not recycled, on the 7th or 8th day of uptime, minor GC time goes up to anywhere from 5 sec to 30 sec, if still not restarted then whenever major GC happens it takes a minute atleast causing several failed transactions. Up on checking the VM status on all 4 nodes, during this time, we see a lot of swapping happening. Memory utilization is under 55% the whole time, also cpu utilization is below 25%. Surprisingly, this happens during the weekend when there is very little to no load. We have not ever seen any OOM errors, so far it appears that heap tuning is a non-issue (I may be wrong). We also have same configuration in a production simulation environment where the load is not as much as production servers, and there is no such swapping/GC problems in that environment. Any insight or any advice on this would be great help. Please let me know if any other info is needed.