22 Replies Latest reply on Sep 6, 2016 9:50 AM by aanjan RSS
    fpassiatore Explorer

    Stream do not store any data

    Hello,

     

    I'm currently trying to set up a stream to store data.

     

    I created a service that store data into that stream and update the properties of the related thing.
    When I execute that service, the properties of the thing are updated but nothing appears in the stream.

    The service is not returning any error.

     

    I even tried to push data in the stream directly from the default service "AddStreamEntry" on the stream itself.
    And even if the service return no error, nothing is added in the stream.

     

    I'm wondering if it's not somehow related to the persistent provider (we are using Postgresql, and on a previous installation with default db everything was working fine)

     

    Any idea on what I am missing here?

     

    Thanks in advance

      • Re: Stream do not store any data
        ttielebein Collaborator

        Are there any errors in the logs? Look under Monitoring > Application (top right corner of home-screen). Also check in the Script logs. Let me know!

        • Re: Stream do not store any data
          aanjan Heavyweight Champ

          Francesco, is it a Stream or a ValueStream? Are your properties logged? Is the user you are using an admin level user?

            • Re: Stream do not store any data
              fpassiatore Explorer

              it's a simple stream, the properties are indeed logged and the user is admin yes

              Edit: I tested the valuesStream as well, and I can say that I have the exact same issue as with simple Streams (no data saved in the valuestream, nothing in the DB).

                • Re: Stream do not store any data
                  ttielebein Collaborator

                  How are you writing to the stream? Value Streams work with the "Logged" option on properties. To update streams, there must be some service somewhere doing this in Javascript. Can you provide that?

                   

                  Thanks!

                  Tori

                    • Re: Stream do not store any data
                      fpassiatore Explorer

                      Hello Tori,

                       

                      My thing's properties have the options "logged" and "persistent" enabled

                       

                      Here is the setup:

                       

                      I have something like 10 things corresponding to LoRa devices.

                      I have a "callback interface" that is used by those LoRa devices to update their own properties.

                      On the callback interface I have a service "UpdateData" that will, thanks to a (unique) deviceID received as a parameter, update the properties of the related thing and store those new properties in a stream.

                      The devices call that "UpdateData" service every 15 minutes.

                       

                      Here is the script of the service :

                      // define the log prefix

                      var logPrefix = me.name +"::LoRa.CallbackDataSet: ";

                       

                      //##################-- calling OpenWeather webservice --##################

                      var OWparams = {

                          url: "http://api.openweathermap.org/data/2.5/weather?lat="+Latitude+"&lon="+Longitude+"&units=metric&appid=**************************************" /* STRING */

                      };

                       

                      // result: JSON

                      var OWresult = Resources["ContentLoaderFunctions"].GetJSON(OWparams);

                       

                      if (OWresult != undefined)

                          var externalTemp = OWresult.main.temp;

                      else {

                          var externalTemp = 0;

                          logger.error(logPrefix + " Failed to retrieve the external temperature from OpenWeather.");

                      }

                      //---------------- end of calling of OpenWeather webservice ----------------

                       

                       

                      //#############-- try to find a thing that has the given deviceid --#############

                      var thingName = me.getLoRaDeviceFromDeviceID({deviceId: DeviceID});

                      if(thingName == "") {

                          logger.error(logPrefix + " Failed to find  thing provisioned for sigfoxId: " + DeviceID);

                          throw "Failed to find thing provisioned for sigfoxId: " + DeviceID;

                      }

                      //-------------------------------------------------------------------------------

                       

                      var dateNow = new Date();

                      //####################-- Setting params --####################

                      // location:LOCATION

                      var myLocation = new Object();

                      myLocation.latitude = Latitude;

                      myLocation.longitude = Longitude;

                      myLocation.elevation = Altitude;

                      myLocation.units ="WGS84";

                       

                      Things[thingName].Temperature = Temperature;

                      Things[thingName].ExtTemperature = externalTemp;

                      Things[thingName].BatteryLifetime = BatteryLevel;

                      Things[thingName].FridgeHumidity = Humidity;

                      Things[thingName].FridgeLocation = myLocation;

                      Things[thingName].RSSI = RSSI;

                      Things[thingName].SNR = SNR;

                      Things[thingName].SpreadFactor = SpreadFactor;

                      Things[thingName].LastCommunication = dateNow;

                      //-----------------------------------------------------------

                       

                      //####################-- Saving fridge value into stream --####################

                      var values = Things["SmartFridgeDataStream"].CreateValues();

                       

                      values.RSSI = RSSI; //NUMBER

                      values.Temperature = Temperature; //NUMBER

                      values.ExtTemperature = externalTemp; //NUMBER

                      values.SNR = SNR; //NUMBER

                      values.SpreadFactor = SpreadFactor; //NUMBER

                      values.FridgeHumidity = Humidity; //NUMBER

                      values.BatteryLifetime = BatteryLevel; //NUMBER

                      values.FridgeLocation = myLocation; //LOCATION

                       

                      var params = {

                          values: values /* INFOTABLE*/,

                          location: myLocation /* LOCATION */,

                          source: thingName /* STRING */,

                          timestamp: dateNow /* DATETIME */

                      };

                       

                      // no return

                      Things["SmartFridgeDataStream"].AddStreamEntry(params);

                      //-----------------------------------------------------------

                       

                      Normally I'm supposed to receive the parameters through HTTP POST request (that part is working fine I could test it by logging the received parameters.

                      But simply by clicking the "test" button of that update data service, I can clearly see that the script is successfully executed and that the properties of my things are updated. But nothing is added in the stream, and nothing appears in the logs.

                       

                      Another important detail to know, We have another Thingworx platform setup (default installation with H2 default DB) and everything is working fine on that one !
                      It is on our new production setup, with PostgreSQL that it is not working (everything in HA).

                        • Re: Stream do not store any data
                          ttielebein Collaborator

                          Can you tell me more about these properties? Are they remotely bound? Are they located on a thingshape? What version of ThingWorx?

                            • Re: Stream do not store any data
                              fpassiatore Explorer

                              Well they are simple properties, only numbers and strings. and no there is no binding, I just assign the values though the services.
                              The properties are all defines on 2 different shapes, and there is one template implementing the 2 shapes. The the things implement the template.

                               

                              But I just did a test and i'm now 99% sure that the issue is coming from the PostGreSQL setup (maybe HA is related?)

                               

                              I just exported my whole project and imported it on a default thingworx installation (with H2 DB), and without any modifications the streams are working fine !
                              So it's definitely not a thing/service issue. I'm nearly sure it's coming from a communication issue between PostGreSQL and Thingworx, or something is not configured properly on that side...

                                • Re: Stream do not store any data
                                  ttielebein Collaborator

                                  Aanjan Ravi what do you make of this last comment?

                                  • Re: Stream do not store any data
                                    aanjan Heavyweight Champ

                                    Francesco, how are the permissions setup on the server? Can you open/ access the 'ThingworxPersistenceProvider' present in the Persistence Providers section?

                                      • Re: Stream do not store any data
                                        fpassiatore Explorer

                                        Hello Aanjan, where exactly do you want me to check the permissions on the server?

                                         

                                        I can open the ThingworxPersistentProdiver but in readonly, I cannot edit it. ( It is tagged as a system object)

                                         

                                        Note that I can create things, templates, shapes, media, datashape, mashups, etc..

                                         

                                        Any other object than streams data and value streams DATA (that i tried at least, i've not tried with data table) are properly saved in the database. It's not working only for stream data/value stream data.

                                          • Re: Stream do not store any data
                                            aanjan Heavyweight Champ

                                            Ah, you should be able to edit it (atleast the configuration section). Streams, Value Streams, Data Tables, Blogs and Wiki data are stored/ maintained through the PersistenceProvider. Can you try giving your user Design and Runtime permissions on the PersitenceProvider?

                                              • Re: Stream do not store any data
                                                fpassiatore Explorer

                                                Well, my user is part of the default administrator group (and even if I use the default administrator user it's the same) and when I open the ThingworxPersistentProvider I have this beautiful message (which makes senses):

                                                Capture.PNG

                                                I don't know if it's normal but I'm not able to edit the configuration of the ThingworxPersistentProvider, even with design and runtimes rights.

                                                  • Re: Stream do not store any data
                                                    aanjan Heavyweight Champ

                                                    What version of ThingWorx is this? You should be able to access the Provider if you're an admin user (as that's where you would change the Stream and Value Stream processor settings). Is your connection going through a Connection Server by any chance?

                                                      • Re: Stream do not store any data
                                                        fpassiatore Explorer

                                                        Well, what you say make sense but it's not the case here

                                                         

                                                        ThingWorx 7.1.3-b14

                                                         

                                                        and for the rest:

                                                             PostgreSQL 9.4.8

                                                             pgpool-II version 3.5.2 (ekieboshi)

                                                         

                                                        Yes I am indeed passing by a connection server


                                                          • Re: Stream do not store any data
                                                            aanjan Heavyweight Champ

                                                            Instead of going through the Connection Server, can you try hitting your platform directly and check if you can access the Provider then?

                                                              • Re: Stream do not store any data
                                                                fpassiatore Explorer

                                                                I connected remotely on the server itself and opened thingworx through the localhost address and I still have read-only access even with the default admin user.

                                                                  • Re: Stream do not store any data
                                                                    fpassiatore Explorer

                                                                    I got some news today from the engineer who installed the platform.
                                                                    He have trace of the requests in the logs of PostGreSQL, but still, even if the request appears, nothing is stored and there is no errors.

                                                                     

                                                                    More than ever I think something is wrong on the PostGreSQL side.

                                                                    Here are the logs (test done with all values to 1):

                                                                    2016-08-30 16:51:29 CEST LOG:  execute pgpool21452/pgpool21452: SELECT current_setting('transaction_isolation')

                                                                    2016-08-30 16:51:29 CEST LOG:  execute <unnamed>: select * from upsert_stream_entry($1,$2,$3,$4,$5,$6,$7) as result

                                                                    2016-08-30 16:51:29 CEST DETAIL:  parameters: $1 = 'SmartFridgeDataStream', $2 = 'fakefridge', $3 = '2016-08-30 16:52:55.611', $4 = '{"SNR": 1.0, "RSSI": 1.0, "Temperature": 1.0, "SpreadFactor": 1.0, "ExtTemperature": 25.26, "FridgeHumidity": 1.0, "FridgeLocation": {"latitude": 1.0, "elevation": 1.0, "longitude": 1.0}, "BatteryLifetime": 1.0, "isGPSLocationFresh": true}', $5 = '1.0,1.0,1.0', $6 = 'Thing', $7 = '[]'


                                                                    I don't know if this helps but... we'll see