4 Replies Latest reply on Aug 15, 2017 10:00 PM by toml RSS
    toml Explorer

    Training failed when monitoring remote thing from kepserver

    Hi,

    I created a remotething in new composer by binding tag from kepservex,and added a anomly alert, then I go to mashup Anomaly_Dashboard to monitoring my newly created thing, but it always failed of training.

     

    Here is error log:

    2017-08-08 12:32:58.757+0800 [L: ERROR] [O: S.c.t.d.e.DSLProcessor] [I: ] [U: Administrator] [S: ] [T: http-nio-80-exec-12] Execution error in service script [AnomalyServices GetAnomalyAlertsByEntity] : TypeError: Cannot call method "GetPropertyDefinitions" of null

     

    Error in invoking service Things_AnomalyServices1, GetAnomalyAlertAttributes, Things, AnomalyServices, Services, GetAnomalyAlertAttributes. Error from the server: "EOF"

    Returning a FAILED state for TimedValue [timestamp=1502168642040, value=95.0]. ThingWatcherMessage [timestamp=2017-08-08T13:04:02.508, severity=ERROR, state=ThingWatcherInternalState [internal=OBTAINING_MODEL, external=TRAINING], messageCode=WAT1001E, messageText=Operation exception. {Throwable=[Trainer.TrainingJobErrorException: Training job with id 39 entered into an incomplete state [FAILED] unable to process models.  Error message: [ConnectException: Connection refused: connect]}]]

     

    Anomaly Alert [Thing4Anomaly2-Simulation_Examples_Functions_Ramp1-amomalydemo2] Failed when in [TRAINING] State. Thingwatcher Error Code :: [WAT1001E]  Message :: Operation exception. {Throwable=[Trainer.TrainingJobErrorException: Training job with id 39 entered into an incomplete state [FAILED] unable to process models.  Error message: [ConnectException: Connection refused: connect]}]

     

    Unable to persist entries because ForsetiClient[id=15785,tx=Transaction[eventId=26137,thread=1369:'ForkJoinPool-34-worker-12',status=STATUS_ACTIVE,resourceCount=1]] can't acquire ExclusiveLock{owner=ForsetiClient[id=15794,tx=Transaction[eventId=26145,thread=1444:'ForkJoinPool-34-worker-30',status=STATUS_ACTIVE,resourceCount=1]]} on NODE(5602), because holders of that lock are waiting for ForsetiClient[id=15785,tx=Transaction[eventId=26137,thread=1369:'ForkJoinPool-34-worker-12',status=STATUS_ACTIVE,resourceCount=1]]._ Wait list:ExclusiveLock[ForsetiClient[id=15794,tx=Transaction[eventId=26145,thread=1444:'ForkJoinPool-34-worker-30',status=STATUS_ACTIVE,resourceCount=1]] waits for [15785, 15793, 15794, ]]

     

    Error occurred during commit for the ThingworxPersistenceProvider persistence provider.

     

     

    Anyone can help? Thanks a lot.

      • Re: Training failed when monitoring remote thing from kepserver
        carko Creator

        Hello, Tom.

         

        We found some internal documentation which suggest trying the following:

         

        • Use the fully qualified hostname or ip address of the results micro service server in the results.destination.uri property in microservice.properties, for example
        results.destination.uri=results://myserver.mydomain.com:8096/results/models
        • Restart the training and results microservices
        • Then reset the alert using the RetrainAlert service or  DisableAlertsForProperty /  EnableAlertsForPropertyservices of the monitored Thing

         

        Can you try this and let us know the results?

         

        Thanks,

         

        -- Craig A.

          • Re: Training failed when monitoring remote thing from kepserver
            toml Explorer

            Hi Craig,

             

            Thanks for your help, here is my configruation of  microservices.properties

             

            dataset.webservices.url=

            dataset.app.id=

            dataset.app.key=

            results.destination.uri=results://192.168.31.202:8090/results/models

            file.storage.path=tempStorage

             

            I can access http://192.168.31.202:8090/results/models in browser at Thingworx server. And I already restart Tomcat of Thingworx, and restart the training and results microservices.  Then using RestartAnomalyAlert service to retrain,but I got the same error. I doubt something wrong with my database.


            It seems I can't use DisableAlertsForProperty or EnableAlertsForProperty services in  AnomalyServices, because it doesn't accept parameter of thing name. Also I can't use service of RetrainAnomalyAlert, it says:

            JavaException: com.thingworx.common.exceptions.InvalidRequestException: Attempted to retrain ThingWatcher [Thing4Anomaly2-Simulation_Examples_Functions_Ramp1-amomalydemo2] that is in an unsupported state [FAILED] (RetrainAnomalyAlert#38)


             

            Thanks & Best Regards

            Tom

              • Re: Training failed when monitoring remote thing from kepserver
                cmorfin Communicator

                Hi Tom

                 

                Could you provide some more details as below:

                - are you deploying the microservices as docker or jar file ?

                - could you provide the full command you execute to start the microservices or docker run command

                - could you send a screenshot of AlertProcessingSubsytem > Configuration page

                - are ThingWorx and microservices on the same machine or different ?

                - what is the OS of ThingWorx server and microservices server ?

                 

                Could you uplaod:

                - ThingWorxStorage/logs/ApplicationLog.log

                - Tomcat/logs from ThingWorx machine

                 

                Note to reset the alert you can maybe try the disableAllAlert and EnableAllAlert service instead for the time being.

                 

                Thanks

                Christophe

                  • Re: Training failed when monitoring remote thing from kepserver
                    toml Explorer

                    Hi Christophe,

                     

                    Thanks for your help. I believe my configuration is correct. Because the monitoring once working properly. And I just tried restart VMware(Thingworx installed in VMware), it is working again,don't know why.Although I still got the error messages which I putted in my original post. At least I can say those error dose not affect anomaly detection.

                     

                    Thanks & Best Regards

                    Tom