15 Replies Latest reply on Jul 27, 2016 11:32 AM by aanjan RSS
    preddy-2 Apprentice

    How to check neo4j size

    Hello all,

    I created a value stream and associated it to a Thing property. And that, thing property is updated on timer event at every 5 sec.

    value stream is created with default persistence provider, so I understand that data get stored in neo.

    I am interested to notice the size of neo and what rate its size grows.

    Can anyone please let me know, where can I check it?

    With B/R,

    Praveen.B

      • Re: How to check neo4j size
        varathanran Apprentice

        Check /ThingworxStorage/database/data folder. You will see bunch of database files like neostore.*.db.*

        • Re: How to check neo4j size
          meghan Communicator

          To expand upon Varathan's answer, a neo database records all transactional data in logical logs in the /ThingworxStorage/database/data directory. This is done as a way to create a backup of the database in the event that ThingWorx does not shut down cleanly. ThingWorx has a default retention period of 7 days before the logs are wiped, but this can be modified in a platform-settings.json file that is described in this article, which links to the Help Center for further details.

           

          Meghan

            • Re: How to check neo4j size
              varathanran Apprentice

              We did reduce the retention to one day last week to save space. The problem we are facing now is neostore.relationshipstore.db grown exponentially after purging the streams and value streams data. In that process it created tons of  nioneo_logical.log.v# files and we cleaned them up. The neo.*.db and its one day backup takes up all the memory in the system. We are not sure why the file size grows when we purge the data and not cleaning up after the restart of tomcat or backups. How to compact the size of those files.

               

              I would also appreciate if you could provide some sizing guidelines.

                • Re: How to check neo4j size
                  aanjan Heavyweight Champ

                  Varathan, if you're on a test server, can you try restarting Tomcat once and see how the behavior is? Does it start filling up as soon as the server is up (and the streams are getting written) or is it a gradual process?

                    • Re: How to check neo4j size
                      varathanran Apprentice

                      This is the demo instance at CVC. Currently we are 100% on the machine and we are not able to do much. I cleaned up all unwanted file to complete the backup yesterday. I was hoping backup will compact the database. I did restart the tomcat several times. I also turned off Streaming into the Thingworx now. We noticed the file size is growing and taking up the space yesterday morning. We thought it is because of value streams constant data flow. I purged the Stream and Value Stream data yesterday. I validated to see the purging is complete and I don't see the older values anymore.

                       

                      During the above process neostore.*.db file started growing exponentially and understandably created large number of nineo_logical.log.v# file (over 1200 files). I removed nineo_logical.log.v# on regular basis to free up the memory. But neostore.*.db is 70% more than what it was yesterday by this time. After taking backup yesterday night, the /ThingworxStorageBackup also increased by similar amount.

                       

                      After completing the purge, I turned the streaming for a while, the growth is not that significant. I am positive purging caused this increased and not sure how to compact the files.

                        • Re: How to check neo4j size
                          aanjan Heavyweight Champ

                          Couple follow up questions then - what are your Java options? Is the system memory where you have ThingWorx running atleast 8 gigs? Do you have a scheduled backup that runs everyday? And what is at a 100%? Is it the disk write or your processes?

                            • Re: How to check neo4j size
                              varathanran Apprentice

                              Here are the options used to start tomcat

                              /usr/lib/jvm/jre/bin/java -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dserver -Dd64 -Xms512m -Xmx2g -XX:+UseNUMA -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

                               

                              Yes. Backup runs every day. We have retention policy of 1 day.

                               

                              100% indicates the disk is full. we have about 25GB for Thingworx. ThnigworxStorage and ThingworxStorageBackup took it all.There is nothing left to clean up either. Our Hard disk usage increased around 70% just yesterday. The only extra thing we did from normal operation was purging of Value Stream data. During Purging we saw the files growing significantly. That is the problem we are trying to solve.

                                • Re: How to check neo4j size
                                  meghan Communicator

                                  Varathan,

                                   

                                  The recommend server sizing is 100GB dedicated hard disk space, a 16GB system, and a 4 core processor. This is of course geared towards a production server, and a demo server can certainly have less dedicated hard disk space and ram. When you say your "Hard disk usage increased around 70%", what was the initial value?

                                   

                                  Do you have any idea as to how many records were in the ValueStream that were being purged?

                                   

                                  Meghan

                                  • Re: How to check neo4j size
                                    aanjan Heavyweight Champ

                                    Adding to Meghan's response, 2 gigs set for the Xmx setting is pretty low. What is your available system RAM? What is in your BackupStorage? Is it just backups of your current ThingWorx entities? How many are there, and how big are they?

                                      • Re: How to check neo4j size
                                        varathanran Apprentice

                                        This is the virtual Machine running on Edge Line server. We may able to bump up the RAM. Currently it is 4GB I believe. This is just demo system and we did not experience any issues in terms of user experience. There is nothing else running on the machine. I see another 2GB is not used at all as per free command. However I am not sure how it is related db file size growth.

                                         

                                        Here is the disk usage

                                        11370180        ./ThingworxStorage

                                        0       ./ThingworxPlatform

                                        11277868        ./ThingworxBackupStorage

                                         

                                        It is just backup of data folder. Nothing else is backed up. That too one day retention.

                                          • Re: How to check neo4j size
                                            aanjan Heavyweight Champ

                                            This would directly correlate to your specs; as threads gets filled up and the memory gets taxed, neo will not be able to keep up and start writing more and more logs for it to complete at a future time. This will eventually lead to Tomcat crashing or even your system becoming unresponsive. As Meghan noted, 16 gigs is necessary, and with the current 4 gigs you have, this is not enough.

                                              • Re: How to check neo4j size
                                                varathanran Apprentice

                                                I did realize the limitation of Neo4j. I am submitted a request to build new VM with Postgres DB, 100GB HDD and 16GB RAM. However I need to figure out something to get by until we receive it. We have customer demos scheduled on daily basis. That is where I am trying to figure out short term solution.

                                                  • Re: How to check neo4j size
                                                    aanjan Heavyweight Champ

                                                    One thing to start off with is your backups - since you have backups taking space as well, you can temporarily disable them (or reduce the frequency) or change the backup type to incremental backup.

                                                      • Re: How to check neo4j size
                                                        varathanran Apprentice

                                                        How do I disable the backup. Disable the timer? I am also copying the backup folder to my local machine now. I am planning to delete the content of backup folder after that. We are not updating any entities. We don't care about data coming in at this point. I am thinking if something goes wrong, I should able to restore from my local machine backup. Do you see any problem?

                                                         

                                                        Once we get the new VM, we can move to Postgres.

                                                         

                                                        Thanks.

                                                          • Re: How to check neo4j size
                                                            aanjan Heavyweight Champ

                                                            That should work. Disabling the Scheduler works as well; I've linked 2 articles in my previous post with which you can extend the backup frequency if needed. The 2nd one would change the backup type to incremental, which means it wouldn't do a full backup every time; it will check for changes and save accordingly.