cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Did you get called away in the middle of writing a post? Don't worry you can find your unfinished post later in the Drafts section of your profile page. X

Value Streams Vs Data Tables

esirohi
1-Newbie

Value Streams Vs Data Tables

As we know, Value Streams are for continuous data streams and Data tables are moreover designed for static data.

What would be an ideal use-case when one should use value streams and Data tables?

1 ACCEPTED SOLUTION

Accepted Solutions
paic
1-Newbie
(To:esirohi)

Not to confuse the picture.

but there are Streams, ValueStreams and DataTables

From the Digital Guide:

A ThingWorx stream is a list of activities from things or data associated with things. Data associated with things includes time series data and event-driven data.

ValueStreams are also like streams, but they will store the data 'flat', because it does it in an automated fashion, while you have to script how to store the information into a regular stream.

Streams also work in an asynchronous way and are more suited for large number of entries.

A ThingWorx data table is similar to a Relational Database (RDBMS) Table. A ThingWorx data shape defines the columns or fields of the data table. To use a data shape to define a data table, at least one of the fields in the data shape must be set as a primary key. This enables the update and delete functionality, as well as efficient querying of data.

DataTables are synchronous and support primary keys and indexing and are more suited for reference/lookup.

So generally anytime you are storing information over time, historical information, think Stream/ValueStream.

If you need a more static reference table, think DataTable.

View solution in original post

9 REPLIES 9
paic
1-Newbie
(To:esirohi)

Not to confuse the picture.

but there are Streams, ValueStreams and DataTables

From the Digital Guide:

A ThingWorx stream is a list of activities from things or data associated with things. Data associated with things includes time series data and event-driven data.

ValueStreams are also like streams, but they will store the data 'flat', because it does it in an automated fashion, while you have to script how to store the information into a regular stream.

Streams also work in an asynchronous way and are more suited for large number of entries.

A ThingWorx data table is similar to a Relational Database (RDBMS) Table. A ThingWorx data shape defines the columns or fields of the data table. To use a data shape to define a data table, at least one of the fields in the data shape must be set as a primary key. This enables the update and delete functionality, as well as efficient querying of data.

DataTables are synchronous and support primary keys and indexing and are more suited for reference/lookup.

So generally anytime you are storing information over time, historical information, think Stream/ValueStream.

If you need a more static reference table, think DataTable.

Is it possible to use Stream and save the data in Hadoop for MAP-R processing?

I believe, yes, this can be done by creating extensions.

qn
1-Newbie
1-Newbie
(To:esirohi)

Hi, everybody

And what would you recommend to use in following case?

I have to store SUM value from the number of Things according to 5 mins intervals.

I've created a simple service which gets the value history from the ValueStream for the desired period and adds the value and Updates the property finally with new SUM value for the desired period. But due to asynchronous ValueStreams if I call 5 times my service fast I'll get only 2 or 3 values added to the SUM value because data is written to disk in 2 seconds (according to default persistent provider) and the next Query will read the previous SUM value.

It should update SUM correctly also the periods in history when one of the RemoteThing  get connected again and uploads historical data.

Would DataTable be more suitable? Or is there other best practice?

Here is the code:

------------------------------------

// params - value = value to be logged, time = DATETIME of property in real time

var period = 300000; //5 min in ms

var periodStart = time.getTime();

periodStart /= period;

periodStart = Math.floor(periodStart) * period;

var periodEnd = periodStart + period - 1;

periodStart = new Date(periodStart);

periodEnd = new Date(periodEnd);

var curTotal = 0;

var params = {

  oldestFirst: undefined /* BOOLEAN */,

  maxItems: 1 /* NUMBER */,

  endDate: periodEnd /* DATETIME */,

  query: undefined /* QUERY */,

  startDate: periodStart /* DATETIME */

};

var res = me.QueryPropertyHistory(params);

if (res && res.rows && res.rows.length > 0){

    curTotal = res.rows[0].totalConsumption;

}

curTotal += value;

//now update value

var itDef = {

   infoTableName : "InfoTable",

   dataShapeName : "NamedVTQ"

};

var itNvtq = Resources["InfoTableFunctions"].CreateInfoTableFromDataShape(itDef);

// NamedVTQ entry object

var newEntry = new Object();

newEntry.name = "SUM"; // STRING

newEntry.time = periodStart; // DATETIME

newEntry.value = curTotal; // VARIANT

newEntry.quality = "GOOD"; // STRING

itNvtq.AddRow(newEntry);

me.UpdatePropertyValues({values: itNvtq /* INFOTABLE */}); //use update property to set current value for thing property correctly

---------------------------------------------------------------------------------------

While tests in design time I'd executed 5 times immediately with value=1 and current time and got as the result only 2 or 3. But expected to see 5.

I would recommend storing that in a Stream.

Data/Information over Time you should think Stream (and ValueStream for the logging)

If it is for reference or lookup think DataTable

But Stream also works asynchronously. Than means that physical data write will occur after my service is finished and another call of service can read invalid (not stored yet) value.

In this case I should have to be sure that write was done and block the execution with some mutex. But I have not found any tools for that. Are there some?

Of course I can create special persistent provider with 1 thread, 1 connection and 0 time of waiting before flushing, but I'm not sure that it is the best idea.

DataTable.

You don't have a "flush" operations or "Wait until writes are done" for Streams or ValueStreams.

If you don't want to use a DataTable, then you will have to call pause(XXX), but XXX it's limited to 1 minute... you can't wait for more.

Carles.

Sorry, I focused on SUM every 5 minutes. For which I think a Stream is perfectly fine.

I don't understand the reason for reading a valuestream several times very quickly if you need the SUM only every 5 minutes.

Top Tags