8 Replies Latest reply on Jan 20, 2016 2:05 PM by efreure RSS
    efreure Newbie

    ThingWorx Edge .NET SDK Crashing

    Hi,

    We are using a ThingWorx Edge .NET microserver based on a modified version of the included steam sensor example.

    We are noticing that it seems to sometimes crash randomly after running for a while, but have been having trouble determining the cause.

    Below are the errors are in the Windows event viewer after it crashes. Are there any known issues with the .NET Edge server crashing? Let me know if there's any other info we can send that would be helpful.

     

    Log Name:      Application

    Source:        .NET Runtime

    Date:          1/8/2016 3:23:48 AM

    Event ID:      1026

    Task Category: None

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      hqqatest50

    Description:

    Application: TestMonitor.exe

    Framework Version: v4.0.30319

    Description: The process was terminated due to an unhandled exception.

    Exception Info: exception code c0000005, exception address 000007FEF8FFF73C

    Stack:

     

     

    Event Xml:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

      <System>

        <Provider Name=".NET Runtime" />

        <EventID Qualifiers="0">1026</EventID>

        <Level>2</Level>

        <Task>0</Task>

        <Keywords>0x80000000000000</Keywords>

        <TimeCreated SystemTime="2016-01-08T08:23:48.000Z" />

        <EventRecordID>1046621</EventRecordID>

        <Channel>Application</Channel>

        <Computer>hqqatest50</Computer>

        <Security />

      </System>

      <EventData>

        <Data>Application: TestMonitor.exe

    Framework Version: v4.0.30319

    Description: The process was terminated due to an unhandled exception.

    Exception Info: exception code c0000005, exception address 000007FEF8FFF73C

    Stack:

    </Data>

      </EventData>

    </Event>

     

     

     

     

    Log Name:      Application

    Source:        Application Error

    Date:          1/8/2016 3:23:32 AM

    Event ID:      1000

    Task Category: (100)

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      hqqatest50

    Description:

    Faulting application TestMonitor.exe, version 1.0.0.0, time stamp 0x568d8e13, faulting module twApi.dll, version 0.0.0.0, time stamp 0x551a50f6, exception code 0xc0000005, fault offset 0x000000000000f73c, process id 0x%9, application start time 0x%10.

    Event Xml:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

      <System>

        <Provider Name="Application Error" />

        <EventID Qualifiers="0">1000</EventID>

        <Level>2</Level>

        <Task>100</Task>

        <Keywords>0x80000000000000</Keywords>

        <TimeCreated SystemTime="2016-01-08T08:23:32.000Z" />

        <EventRecordID>1046620</EventRecordID>

        <Channel>Application</Channel>

        <Computer>hqqatest50</Computer>

        <Security />

      </System>

      <EventData>

        <Data>TestMonitor.exe</Data>

        <Data>1.0.0.0</Data>

        <Data>568d8e13</Data>

        <Data>twApi.dll</Data>

        <Data>0.0.0.0</Data>

        <Data>551a50f6</Data>

        <Data>c0000005</Data>

        <Data>000000000000f73c</Data>

      </EventData>

    </Event>

     

    Message was edited by: Eric Freure Attached crash dump

      • Re: ThingWorx Edge .NET SDK Crashing
        stevegr Explorer

        From this link http://stackoverflow.com/questions/17168982/exception-error-c0000005-in-vc which you probably when to its a memory access issue.

        Assuming its time based, are you running out of memory on the device?

        What kind of device is the code running on?

        What is its basic activity?

        Can you share code snippet to provide insight to the changes you made.

          • Re: ThingWorx Edge .NET SDK Crashing
            efreure Newbie

            It's running on a PC running Windows 2008. I don't think it's running out of memory, although the machines we're running it on don't have much, only 2 gigs of RAM. I don't see any other programs that are running on it having memory errors.

            I don't know if I would say it is time based. It seems pretty random. We have it run on 14 different machines that are restarted each evening and it seems like there will usually be one or two instances of the Edge server that crash during the day.

            Its basic activity is to communicate with a Python program that we run tests with, then update a Thingworx mashup with the test's status.

            Here's a snippet with most of the changes that were made to the steam sensor example. I don't think it does anything too different from the steam sensor example:

             

            TestMonitorThing.cs:

             

            using com.thingworx.common;

            using com.thingworx.common.exceptions;

            using com.thingworx.communications.client;

            using com.thingworx.communications.client.things;

            using com.thingworx.metadata;

            using com.thingworx.metadata.annotations;

            using com.thingworx.metadata.collections;

            using com.thingworx.types;

            using com.thingworx.types.collections;

            using com.thingworx.types.constants;

            using com.thingworx.types.primitives;

            using System;

            using System.Collections.Specialized;

            using System.Linq;

            using System.Text;

            using System.Threading;

            using System.Globalization;

            using Microsoft.Win32.TaskScheduler;

            using System.Collections;

            using log4net;

             

             

            namespace TestMonitor

            {

                // Property Definitions

                [ThingworxPropertyDefinition(name = "TestBaselineTask", description = "Baseline Task", baseType = "STRING", category = "Status", aspects = new string[] { "isReadOnly:true" })]

                [ThingworxPropertyDefinition(name = "TestTask", description = "Test Task", baseType = "STRING", category = "Status", aspects = new string[] { "isReadOnly:true" })]

                class TestMonitorThing : VirtualThing

                {

                    private static readonly ILog logger = LogManager.GetLogger(typeof(TestMonitorThing));

                    private static String TEST_NAME = "TestName";

                    private static String TEST_STATUS = "TestStatus";

             

             

                    // Lock and bool field used to keep the shutdown logic from occuring multiple times before the actual shutdown

                    private object _shutdownLock = new object();

                    private bool _shuttingDown = false;

                    private TestMonitor.TestCommunicator _testCommunicator;

             

             

                    public TestMonitorThing(string name, string description, string identifier, ConnectedThingClient client, TestMonitor.TestCommunicator testCommunicator)

                        : base(name, description, identifier, client)

                    {

                        //Data shape definition that is used by the GetSteamSensorReadings service

                        FieldDefinitionCollection readingfields = new FieldDefinitionCollection();

                        readingfields.addFieldDefinition(new FieldDefinition(TEST_NAME, BaseTypes.STRING));

                        readingfields.addFieldDefinition(new FieldDefinition(TEST_STATUS, BaseTypes.STRING));

                        defineDataShapeDefinition("SteamSensorReadings", readingfields);

             

             

                        // Populate the thing shape with the properties, services, and events that are annotated in this code

                        base.initializeFromAnnotations();

                        _testCommunicator = testCommunicator;

                    }

             

             

                    // From the VirtualThing class

                    // This method will get called when a connect or reconnect happens

                    // Need to send the values when this happens

                    // This is more important for a solution that does not send its properties on a regular basis

                    public override void synchronizeState()

                    {

                        // Be sure to call the base class

                        base.synchronizeState();

                        // Send the property values to Thingworx when a synchronization is required

                        base.syncProperties();

                    }

             

             

                    // The processScanRequest is called by the SteamSensorClient every scan cycle

                    public override void processScanRequest()

                    {

                        // Be sure to call the base classes scan request

                        base.processScanRequest();

                        // Execute the code for this simulation every scan

                        this.scanDevice();

                    }

             

             

                    private Task getTestTask()

                    {

                        Task kickoffTask;

                        using (TaskService ts = new TaskService())

                        {

                            kickoffTask = ts.GetTask("Kickoff");

                        }

                        return kickoffTask;

                    }

             

             

                    private Task getTestBaselineTask()

                    {

                        Task kickoffBaselineTask;

                        using (TaskService ts = new TaskService())

                        {

                            kickoffBaselineTask = ts.GetTask("Kickoffbaseline");

                        }

                        return kickoffBaselineTask;

                    }

             

             

                    // Performs the logic for the steam sensor, occurs every scan cycle

                    public void scanDevice()

                    {

                        Random random = new Random();

             

             

                        Task testTask = getTestTask();

                        Task baselineTask = getTestBaselineTask();

                        string testTaskStatus = testTask.Name + " - " + testTask.State.ToString();

                        string testBaselineTaskStatus = baselineTask.Name + " - " + baselineTask.State.ToString();

             

             

                        // Set the property values

                        base.setProperty("TestTask", testTaskStatus);

                        base.setProperty("TestBaselineTask", testBaselineTaskStatus);

             

             

                        try

                        {

                            // Update the subscribed properties and events to send any updates to Thingworx

                            // Without calling these methods, the property and event updates will not be sent

                            // The numbers are timeouts in milliseconds.

                            base.updateSubscribedProperties(15000);

                            base.updateSubscribedEvents(60000);

                        }

                        catch (Exception ex)

                        {

                            // handle exception as appropriate

                            logger.Error("Error in scanDevice!");

                        }

                    }

             

             

             

             

                    [method: ThingworxServiceDefinition(name = "LaunchTest", description = "Launch test.")]

                    [return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "STRING")]

                    public string LaunchTest()

                    {

                        getTestTask().Run();

                        return "Success";

                    }

             

             

                    [method: ThingworxServiceDefinition(name = "StopTest", description = "Stop test.")]

                    [return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "STRING")]

                    public string StopTest()

                    {

                        getTestTask().Stop();

                        return "Success";

                    }

             

             

                    [method: ThingworxServiceDefinition(name = "LaunchBaselineTest", description = "Launch baseline test.")]

                    [return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "STRING")]

                    public string LaunchBaselineTest()

                    {

                        getTestBaselineTask().Run();

                        return "Success";

                    }

             

             

                    [method: ThingworxServiceDefinition(name = "GetTestState", description = "Get the state of tests that run on this machine.")]

                    [return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "INFOTABLE", aspects = new string[] { "dataShape:SteamSensorReadings" })]

                    public InfoTable GetTestState()

                    {

                        var table = new InfoTable(getDataShapeDefinition("SteamSensorReadings"));

                        try

                        {

                            OrderedDictionary testStatuses = _testCommunicator.GetTestStatuses();

                            foreach (DictionaryEntry de in testStatuses)

                            {

                                string currentTestName = (string)de.Key;

                                string currentTestStatus = (string)de.Value;

                                var entry = new ValueCollection();

                                entry.SetStringValue(TEST_NAME, currentTestName);

                                entry.SetStringValue(TEST_STATUS, currentTestStatus);

                                table.addRow(entry);

                            }

                        }

                        catch (Exception e)

                        {

                            // handle exception as appropriate

                            logger.Error("Error in GetTestState!");

                        }

             

             

                        return table;

                    }

             

             

                    [method: ThingworxServiceDefinition(name = "Shutdown", description = "Shutdown the client")]

                    [return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "", baseType = "NOTHING")]

                    public void Shutdown()

                    {

                        // Highly unlikely that this service could be called more than once, but guard against it anyway

                        lock (this._shutdownLock)

                        {

                            if (!this._shuttingDown)

                            {

                                // Start a thread to begin the shutdown or the shutdown could happen before the service returns

                                ThreadPool.QueueUserWorkItem(new WaitCallback(this.shutdownThread));

                            }

                            this._shuttingDown = true;

                        }

                    }

             

             

                    private void shutdownThread(object state)

                    {

                        try

                        {

                            // Delay for a period to verify that the Shutdown service will return

                            Thread.Sleep(1000);

                            // Shutdown the client

                            this.getClient().shutdown();

                        }

                        catch

                        {

                            logger.Error("Error in shutdownThread!");

                            // Not much can be done if there is an exception here

                            // In the case of production code should at least log the error

                        }

                    }

                }

            }

              • Re: ThingWorx Edge .NET SDK Crashing
                stevegr Explorer

                Eric,

                 

                Could the Python be causing an issue? I have run Python on my PI's and sometimes they stop. How are you communicating with the Python ?

                  • Re: ThingWorx Edge .NET SDK Crashing
                    efreure Newbie

                    I was leaning away from the Python communication causing the crashing since the exception specifically mentions "faulting module twApi.dll", but I guess I it's still possible that the communication with Python could be causing the problem.

                     

                    The Python process communicates with the .NET Edge server using zeromq. Python uses the pyzmq library and the .NET Edge server uses NETMQ. A thread on the .NET Edge server runs that listens for updates from the Python process through ZeroMQ. When an update comes in, a single string variable is updated. Another thread that runs the .NET Edge server sensor updates calls a method which parses the string and turns it in to an InfoTable that gets sent to the Thingworx mashup.

                      • Re: ThingWorx Edge .NET SDK Crashing
                        efreure Newbie

                        It's looking like it may have been a thread safety issue. I put in some locks where a shared variable was being used between threads in code that communicated with the Python program and we haven't seen the crash again yet.

                          • Re: ThingWorx Edge .NET SDK Crashing
                            efreure Newbie

                            Oops, spoke too soon, it crashed again on one of our machines last night with similar errors in the event log (see below). So, still not sure what could be causing this crash.

                             

                             

                            Log Name:      Application

                            Source:        Application Error

                            Date:          1/20/2016 1:45:52 AM

                            Event ID:      1000

                            Task Category: (100)

                            Level:         Error

                            Keywords:      Classic

                            User:          N/A

                            Computer:      hqqatest9

                            Description:

                            Faulting application TestMonitor.exe, version 1.0.0.0, time stamp 0x5696c6b9, faulting module twApi.dll, version 0.0.0.0, time stamp 0x551a50f6, exception code 0xc0000005, fault offset 0x000000000000f73c, process id 0x%9, application start time 0x%10.

                            Event Xml:

                            <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

                              <System>

                                <Provider Name="Application Error" />

                                <EventID Qualifiers="0">1000</EventID>

                                <Level>2</Level>

                                <Task>100</Task>

                                <Keywords>0x80000000000000</Keywords>

                                <TimeCreated SystemTime="2016-01-20T06:45:52.000Z" />

                                <EventRecordID>501767</EventRecordID>

                                <Channel>Application</Channel>

                                <Computer>hqqatest9</Computer>

                                <Security />

                              </System>

                              <EventData>

                                <Data>TestMonitor.exe</Data>

                                <Data>1.0.0.0</Data>

                                <Data>5696c6b9</Data>

                                <Data>twApi.dll</Data>

                                <Data>0.0.0.0</Data>

                                <Data>551a50f6</Data>

                                <Data>c0000005</Data>

                                <Data>000000000000f73c</Data>

                              </EventData>

                            </Event>

                             

                             

                             

                             

                             

                            Log Name:      Application

                            Source:        .NET Runtime

                            Date:          1/20/2016 1:46:08 AM

                            Event ID:      1026

                            Task Category: None

                            Level:         Error

                            Keywords:      Classic

                            User:          N/A

                            Computer:      hqqatest9

                            Description:

                            Application: TestMonitor.exe

                            Framework Version: v4.0.30319

                            Description: The process was terminated due to an unhandled exception.

                            Exception Info: exception code c0000005, exception address 000007FEF8B6F73C

                            Stack:

                             

                             

                            Event Xml:

                            <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

                              <System>

                                <Provider Name=".NET Runtime" />

                                <EventID Qualifiers="0">1026</EventID>

                                <Level>2</Level>

                                <Task>0</Task>

                                <Keywords>0x80000000000000</Keywords>

                                <TimeCreated SystemTime="2016-01-20T06:46:08.000Z" />

                                <EventRecordID>501768</EventRecordID>

                                <Channel>Application</Channel>

                                <Computer>hqqatest9</Computer>

                                <Security />

                              </System>

                              <EventData>

                                <Data>Application: TestMonitor.exe

                            Framework Version: v4.0.30319

                            Description: The process was terminated due to an unhandled exception.

                            Exception Info: exception code c0000005, exception address 000007FEF8B6F73C

                            Stack:

                            </Data>

                              </EventData>

                            </Event>

                  • Re: ThingWorx Edge .NET SDK Crashing
                    efreure Newbie

                    I found that a crash dump was generated. I've attached it to the first post in case anyone is interested in looking at it.

                      • Re: ThingWorx Edge .NET SDK Crashing
                        efreure Newbie

                        I've managed to open up the crash dumps in Visual Studio, and I'm seeing that it seems to have crashed on line 77 of the C Edge SDK file src\messaging\twMessaging.c:

                        if (msgHandlerSingleton && msgHandlerSingleton->incomingRequestCallbacks) {

                         

                        I'm not sure, but it looks like the msgHandlerSingleton variable may be invalid when it is referenced on that line.