5 Replies Latest reply on Nov 2, 2015 1:45 PM by carlesc RSS
    carlesc Ninja

    Asyncronous Streams and Delayed Calculations Execution?

    Hi All,

     

    I'm importing on batch old data ( in blocks ), this data it's automatically stored on a Stream ( either Value Stream or Stream ), and I need to do a calculation based on current value and previous value (accumulated consumption counter). I don't have previous value in memory, hence I have to go to the Stream and get previous value, and here the async stream problem arises: if I do too fast the previous value ( which I'm importing in batch "super fast" ) won't be stored on the Stream it's on the write queue.

     

    As I control Data Change event on the property ( accumulated consumption counter ), I would like to execute a delayed Data Change event, there's an easy way to do it? All the ways I think of are too much or I don't like it:

    • Build a Delayed Events queue, which will execute events after a while. I think this is the best way, but It's a lot of work to build it correctly, also I will need a timer to trigger events once in a while and this will add load to the server,...
    • Do the calculations once a day . I don't like this solution, as not always I will load data on batch, for the same property on another Thing may I have "real time" data and I want on that case that the calculations are also in "real time"
    • Delay a service call ( something like setTimeout() ) --> I think this is not possible on TW? maybe a combination of Async Service call plus a pause(), but I don't know the performance impact of such a solution on the TW Threading model, something like this
      • Async Service:  void SetTimeout(thing,service,params,milliseconds) {
           pause(milliseconds);
           Things[thing][service](params);
        }
        

     

    Best Regards,

    Carles.

      • Re: Asyncronous Streams and Delayed Calculations Execution?
        paic Collaborator

        Thingworx supports the pause service

        basically

        pause(1000) would be pause for 1000 miliseconds.

        • Re: Asyncronous Streams and Delayed Calculations Execution?
          carlesc Ninja

          Hi Pai,

           

          Yes, already it's on my proposals, but the question about using pause(n) it's how will this affect on the threading model... I mean, let's say I import 100k registers in batch, which in turn does 100k SetTimout ( function as shown above ) what will happen to TW?

           

          Best Regards,

          Carles.

          • Re: Asyncronous Streams and Delayed Calculations Execution?
            adamr Creator

            Hi Carles,

            Is there a reason that you are not keeping the previous value you need in memory?  Since I do not know the full scope of the transactions I was not sure if the data value was large or some further reason.  If you can store the previous value it should eliminate the need to pause?

            Thanks,

            Adam

            • Re: Asyncronous Streams and Delayed Calculations Execution?
              carlesc Ninja

              Hi All,

               

              This post got lost on the community ( it went to another place instead of Developer Community ).

               

              Just for the record, and if someone has the same needs, or maybe anyone can think on other options...

               

              • Build a Delayed Events queue, which will execute events after a while. I think this is the best way, but It's a lot of work to build it correctly, also I will need a timer to trigger events once in a while and this will add load to the server,...
                • I followed this approach:
                  • Created a DataTable where Events are pushed with adding a Timeout, and with a timer I trigger delayed events when Timeout it's reached.
              • Do the calculations once a day . I don't like this solution, as not always I will load data on batch, for the same property on another Thing may I have "real time" data and I want on that case that the calculations are also in "real time"
              • Delay a service call ( something like setTimeout() ) --> I think this is not possible on TW? maybe a combination of Async Service call plus a pause(), but I don't know the performance impact of such a solution on the TW Threading model, something like this
                • I Tested this approach, but it ends ups consuming lots of RAM, as it creates a new Thread for each async SetTimeout call. If someone want's it, here it's, it can be useful but not for lot's of calls ( 10k calls = 1Gb RAM ),
              async function SetTimeout(thing,service,params,milliseconds) {
              if (milliseconds>=60000) {
                  var nTimes = parseInt(milliseconds/60000,10);
                  for(var i=0;i<nTimes;i++) pause(59999);
                  // -- As the following instruction doesn't works and it should: case: 12808137
                  // -- we will just trigger half a minute more 
                  //pause(parseInt(milliseconds%60000,10));
                  pause(30000);
              } else {
                  pause(milliseconds);
              }
              if (thing) {
                  Things[thing][service]((params?JSON.parse(params):null));
              } else {
                  me[service]((params?JSON.parse(params):null));
              }
              }
              

               

              Best Regards,

              Carles.