9 Replies Latest reply on Oct 18, 2017 6:13 AM by nagrawal-2 RSS
    mbereket Apprentice

    How to insert past-data to a Thing?

    Dear TW community members!

     

    So, this time we are working on "Local Data Storage" feature for our devices when the internet connection is not available.

     

    Here is a quick back-ground info:

    - We have smart sensors which updates thingworx every minute to post temperature,pressure, humidity etc.

    - We store all these information using "Value Streams" for each device.

    - and then we use "QueryNumberPropertyHistory" service to retrieve history of data to display meaningful graphs and other info

     

    When there is no internet connection, our devices skip sending information and whenever the connection is back, devices will start re-updating ThingWorx again.

     

    Now, we have managed to store information locally within devices even when there is no internet.

     

    However, I am curious, what would be the best way to update thingworx and insert all of those locally stored information in one single update? and I still want to be able to see meaningful (not collapsed) [temperature - timestamp] data pairs when I use "QueryNumberPropertyHistory" service on Things. (Generic Services)

     

    Thanks

      • Re: How to insert past-data to a Thing?
        carlesc Ninja

        If you use standard TW subscriptions it will preserve timestamps if you send it on property updates.

         

        If you use you own services to send data back to the platform, you can leverage "Add"+baseType+"ValueStreamEntry" service in order to add historical data ( but if you have subscriptions that depend on this past time DataChange event it won't be thrown ).

         

        Just my two cents, I don't know if it helps.

        • Re: How to insert past-data to a Thing?
          Ankit Gupta Ninja

          Hi mustafa bereket,

           

          As per my understanding you can create your own custom service which takes infotable as input parameter. Then iterate through all the rows and use UpdatePropertyValues service to update values in system. UpdatePropertyValues service takes timestamp as one of the input so we can use the same timestamp when the property was generated and make it look like it was inserted without any delay.

           

          I hope it helps.

          • Re: How to insert past-data to a Thing?
            mbereket Apprentice

            Hi Carles Coll and Ankit Gupta,

             

            Thanks for your help, I was curious to see if there is anyway of updating a property with a specific timestamp data.

             

            Like, I have a "Thing", that thing has "temp" attribute,

             

            When I only update [MyThing][temp] in real time, everything is perfect like following;

            operation 1: [MyThing][temp] = 25;    

            operation 2: [MyThing][temp] = 26;

            operation 3: [MyThing][temp] = 27;

            operation 4: [MyThing][temp] = 28;


            But then, I receive another update package stating it is for the past (offline- data collected during internet outage) , then I will need to stick following data, before operation 1


            operation -3: [MyThing][temp] = 19;

            operation -2: [MyThing][temp] = 11;

            operation -1: [MyThing][temp] = 20;   

            operation  0: [MyThing][temp] = 16;



            I am not sure if it is possible but if I somehow can manage to do that just by using ThingWorx generic services, that would be great.


            Because this way, I can just use "QueryNumberPropertyHistory" service, and keep generating my reports.


            PS: I can definitely provide timestamps for offline data stream.

              • Re: How to insert past-data to a Thing?
                Ankit Gupta Ninja

                mustafa bereket,

                 

                UpdatePropertyValues is the generic service which also takes timestamp as input.

                  • Re: How to insert past-data to a Thing?
                    mbereket Apprentice

                    Hi Ankit,

                     

                    I did not notice that, I am trying it out now.

                     

                    Thanks,

                    • Re: How to insert past-data to a Thing?
                      mbereket Apprentice

                      Thank you  Ankit Gupta

                       

                      This service worked perfectly and solved my problem.

                      • Re: How to insert past-data to a Thing?
                        nagrawal-2 Explorer

                        Hello Ankit,

                         

                        what format should be the timestamp to update the values in ValueStream ? I am reading data from CSV file and storing it in Valuestrem using custom timestamp but seems like my data got uploaded with wrong dates. Please see my code below :

                        var params = {
                        path: filename /* STRING */,
                        columnMappings: undefined /* STRING */,
                        hasHeader: undefined /* BOOLEAN */,
                        longitudeField: undefined /* NUMBER */,
                        dateFormat: "YYYY-MM-DD" /* STRING */,
                        fileRepository: repository /* THINGNAME */,
                        latitudeField: undefined /* NUMBER */,
                        fieldDelimiter: undefined /* STRING */,
                        stringDelimiter: undefined /* STRING */,
                        dataShape: "Building_DataShape" /* DATASHAPENAME */
                        };
                        
                        // result: INFOTABLE
                        var csvData = Resources["CSVParserFunctions"].ReadCSVFile(params);
                        var length = csvData.rows.length;
                        
                        
                        var params2 = {
                        infoTableName : "InfoTable",
                        dataShapeName : "Infotable_DataShape"
                        };
                        
                        // CreateInfoTableFromDataShape(infoTableName:STRING("InfoTable"), dataShapeName:STRING):INFOTABLE(TestCSV_DataShape)
                        var properties = Resources["InfoTableFunctions"].CreateInfoTableFromDataShape(params2);
                        
                        for each (var row in csvData.rows) {
                            // Thing1 entry object  
                        var newEntry = new Object();  
                        newEntry.time = row.date; // DATETIME - isPrimaryKey = true  
                        newEntry.quality = undefined; // STRING 
                            
                        newEntry.name = "Consumption"; // STRING  
                        newEntry.value = row.consumption; // STRING  
                        properties.AddRow(newEntry);
                            
                        newEntry.name = "Demand"; // STRING  
                        newEntry.value = row.demand; // STRING  
                        properties.AddRow(newEntry);
                            
                        // no return
                        Things["Building"].UpdatePropertyValues({ values: properties });
                        }
                        var result = properties;
                        

                         

                        The format of data in my CSV file is listed below:

                        2016-01-01,1227.75,123.58

                        2016-01-02,153,34.82

                        2016-01-03,167.21,28.05

                        2016-01-04,181.05,24.55