7 Replies Latest reply on Nov 27, 2017 4:17 PM by pchung RSS
    pselvaraj-2 Explorer

    Subscription - 20,000 property writes per 30 second

    Hi,

     

    I have a scenario, we have 1000 devices at multiple regions connected via edge sdk. Every 30 secs data received from devices and updated thing properties (20). On data change event I have enabled a subscription to write these entries on a stream. System becomes unresponsive, entries didn't as expected and hung. Pl someone assist me how to approach this scenario? Thanks in advance.

      • Re: Subscription - 20,000 property writes per 30 second
        wposner-2 Creator

        Why are you using a datachange event to write the properties to a stream?  Why not just enable logging on the properties and assign a value stream to  your thing template.  That will save you from the huge subscription performance hit.

        • Re: Subscription - 20,000 property writes per 30 second
          pchung Collaborator

          There are some settings you can increase for the Event/Stream queue, but if you are overrunning the server, you probably need a more powerful server.

          The valuestream suggestion is a good one as well.

          • Re: Subscription - 20,000 property writes per 30 second
            pselvaraj-2 Explorer

            Thanks for your reply! But valuestream entries are not consistent and some time we found gap in the entries.  Besides, we had to some manipulation at regular intervals and moved those calculations to another stream in order to increase the performance instead of querying the raw stream after entries written to raw stream. for eg. we are showing trend chart in hour wise. Any other suggestions would be helpful!

              • Re: Subscription - 20,000 property writes per 30 second
                pchung Collaborator

                Sounds like the issue you are experiencing is pretty complex.

                I'd like to recommend you engage with our Customer Success group to set up a workshop to fully understand your use case and give you expert recommendations.

                But my initial guess is that you are not passing a specific timestamp to your property values (VTQ) and it is writing values to the stream as they arrive, since the writes are asynch it might mean they go out of order.

                The missing values most likely is because you are writing within a period of 3ms apart which means that entries are overwritten.

                 

                You will need to have that verified but also have your architecture checked to make sure you aren't overrunning your JVM/Processing power as the writes come in and events are fired. Based on that you'll have to either increase your processing power, but most likely you will have to adjust your design.

                  • Re: Subscription - 20,000 property writes per 30 second
                    pselvaraj-2 Explorer

                    Thank you!! Sounds like need to check the design again.

                     

                    One other scenario.. Writing entries to stream every 15 mins interval and almost 96000 records per month. One of the use case is to diplay trend charts with hour/day/week/mont/year filters. So, we created a timer which fires every 15 or 30 mins to implement busniess logic and moved the aggregated data to new streams like hourstream and dailystream. Now data fetched from these new streams, hour from hourstream and for other filers from dailystream instead of raw stream which has for  100,000 plus records. What would be the other best approach for this scenario? Pl assist.