2 Replies Latest reply on Oct 21, 2017 5:51 AM by varathanran RSS
    aensminger Newbie

    Kepserver websocket forcibly closed

    We have about 8 or 10 Kepservers communicating to ThingWorx (and have for several months), and recently started having regular crashes(multiple times a day) of Tomcat in our production environment. It doesn't hang or get slow, Tomcat just stops abruptly. I noticed the crash log from Tomcat listed a boatload of blocked threads, I've attached a sample here. Through lots of trial and error (and much help from PTC support), we identified that it must be somehow related to our remote connections, because with the WSExecutionProcessinng subsystem turned off for several days we had no issues.

     

    We stopped the actual crashing by adding the JVM argument  "-XX:-UseAESIntrinsics" after deep googling and shots in the dark led us to find a lot of similarities with this issue. However, the underlying issue I think is still happening, as we still get daily errors in the Tomcat std_err log that look like the below.

     

    I'm running thin on things to investigate (but do still have a few), but has anyone here seen anything like this? Or potentially have insight? Been through most of the obvious stuff and working with support a few weeks on this and I feel we're running dry. Thanks in advance!

     

    example stomcat8-stderr log - What would cause this?:

     

    Oct 17, 2017 4:04:58 AM org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer doClose

    INFO: Failed to close the ServletOutputStream connection cleanly

    java.io.IOException: An existing connection was forcibly closed by the remote host

    at sun.nio.ch.SocketDispatcher.write0(Native Method)

    at sun.nio.ch.SocketDispatcher.write(Unknown Source)

    at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)

    at sun.nio.ch.IOUtil.write(Unknown Source)

    at sun.nio.ch.SocketChannelImpl.write(Unknown Source)

    at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:134)

    at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:370)

    at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:398)

    at org.apache.coyote.http11.upgrade.NioServletOutputStream.doClose(NioServletOutputStream.java:138)

    at org.apache.coyote.http11.upgrade.AbstractServletOutputStream.close(AbstractServletOutputStream.java:140)

    at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doClose(WsRemoteEndpointImplServer.java:143)

    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.close(WsRemoteEndpointImplBase.java:638)

    at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:118)

    at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:81)

    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:450)

    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:338)

    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:270)

    at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:570)

    at org.apache.tomcat.websocket.WsSession.onClose(WsSession.java:510)

    at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.close(WsHttpUpgradeHandler.java:183)

    at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.access$200(WsHttpUpgradeHandler.java:48)

    at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsReadListener.onDataAvailable(WsHttpUpgradeHandler.java:214)

    at org.apache.coyote.http11.upgrade.AbstractServletInputStream.onDataAvailable(AbstractServletInputStream.java:198)

    at org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDispatch(AbstractProcessor.java:96)

    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:663)

    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)

    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1517)

    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1474)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

    at java.lang.Thread.run(Unknown Source)