Page 1 of 3

json, CAN

Posted: Sat Apr 27, 2013 4:10 am
by andg
Published notes are attached to this post. Below is the notes that eventually became the presentation.

so the idea is to talk about mongo db in an implementation prepared to log data that doesn't necessarily have a schema. So json is a standard that defines data at a state in time. A sample user of a forum example.

Hi my name is Nathan Sparks and I have 42 posts on the diy_efi board.

Code: Select all

user
{ _id: BSON_id //a number like 0xb2d91ae6f8416660
{first_name: "Nathan",
last_name: "Sparks",
post_count: 42,
memberships:
{board_name: "diy_efi"}
}
Honestly I see the memberships embedded document being deeper but this is for a sample, and I'm trying to keep this simple. In use in engine management this could be used in combination with the network of sensors standard known as CAN. It provides banks of sensors so it would look something like this:

Code: Select all

{
p_0: 15.0
p_1: 0
p_5: duh is this getting boring?
}
that could be dynamically appended a t value that defines where in the sequence of the logging and that value would be indexed.

Re: json, CAN

Posted: Sat Apr 27, 2013 4:28 am
by andg
perfect place to put un implemented features so if at a later date we have data that is expected to be 0. we can filter that out easily later. so _max and _avg can be filtered until the register stars responding like an O2 sensor.

So the engine is at the mercy of the ecu until the feedback from the o2 sensor can keep the engine running. Before that time, the ecu needs to deal with throttle position, air bypass injection and spark.

voltage of the sensor ranges from 100 to 900 mV. and the current ECU will throw a trouble code if the range remains below 300 or above 800.

Code: Select all

{_id: BSON,
o2: 0.134,
t_value: 1},
]{_id: BSON,
o2: 0.1356,
t_value: 2},
]{_id: BSON,
o2: 0.256,
t_value: 3},
if I were in the console I could query the data

Code: Select all

db.sensors.find({t_value: {$lt:  4}})

Re: json, CAN

Posted: Sat Apr 27, 2013 4:56 am
by andg
so it takes event handlers like
t_value = now - 150 _avg
if we need to evaluate if a sensor has been offline for 150 cycles we need listeners
small mongodb instances that are self garbage collecting using capped collections
optimized data evaluation

This same capped collection can be used to determine a sensor is out of range.

Since we don't know many documents per second that we are going to receive we set the capped collection to be tuned on the least number that can be metriced when saving only the single sensor to the database. Since documents performance are based on size of the document, the best performant implementation can be set as a value based on the test.

You can then gauge your performance when you do a query on the t value for the time range the test was for. If our isolated test revealed maximized performance tooks 3000 documents and in implementation there were 2000 documents returned from

Code: Select all

db.sensor_log.find({created_at: { $lte: new Timestamp() - 0x10000 } } )
MongoDB support for timestamps docs.mongodb.org
mongo reference timestamps wrote:Timestamp values are a 64 bit value where:

the first 32 bits are a time_t value (seconds since the Unix epoch)
the second 32 bits are an incrementing ordinal for operations within a given second.

Re: json, CAN

Posted: Sat Apr 27, 2013 5:00 am
by andg
use it as an extaction layor for an existing SQL instance by issuing sql commands and interpreting the results into a JSON log output and output to targit {name to be determined, I know it starts with the letter x and released in a later version of their bi tool; the whole reason I'm implementing it in a data center with the upgraded release version so all the new features unlock as part of the migration] which is designed to show multi dimensional views of predefined unputs.

In troubleshooting large sql tables I often start by query with a

Code: Select all

SELECT TOP 1 * from PBAInstanceTable
which returns column names of the table that is used to define a product model which has configuration options for a product like Kent which can be seen in the configuration engine.

Options like vaulted, or finish, length, shade are records in sql using referencial number. I would want another SQL statement to return the collection of these configurations. By the way it will return pre and post configuration engine variables so you will actually need to filter on that as well if you want to return a single configuration detail set.

Code: Select all

SELECT * from PBATABLEINSTANCE WHERE INVENTTRANSID = {#instance_variable}
I would want to have JSON logging like this

Code: Select all

{ _id: BSON ,
query: "SELECT TOP 1 * from PBAInstanceTable",
headers: "INVENTTRANSID, variable, value"
rows: "1, pba_model_number, 1"
},
{_id: BSON,
query: "SELECT * from PBATABLEINSTANCE WHERE INVENTTRANSID = 3245",
rows: {
3245, pba_model_number, 25
3245, finish, "old_brass",
3245, shade, "Mission Satin Etched Flared Shade$35.00"
3245, upshade, "Mission Satin Etched Flared Gas-Style Shade$50.00"
3245, lights, 8
3245, lengths, 36
3245, socket, :turnkey
3245, vaulted, false
}
Asynchronous process at a higher level language can parse this document and run an update statement that appends to the model embedded document like

Code: Select all

db.pba_instance.find({_id:  bson}).shade
Can be queried to return the string

Code: Select all

db.pba_instance.update({_id:  bson} , $set : {shade : "Mission Satin Etched Flared Shade$35.00", state: "processed"}})
The state is the addition of statemachine which allows the document processing state to follow the document. It allows built in error handling that happens within the document itself. MongoDB is a great thing to add features like statemachine for some documents, and a capped collection so those that were old fall off and those that were in error which have likely been fixed by the time the error drops off makes for efficient troubleshooting with built in analytics.

For example your state machine might be inspecting the model for the product and validating that all of the required config variables exist and if it detects an anomoly against the model it can set the state to

Code: Select all

state: "expecting value for pba_model 25 variables [priority, gu24_bulb, gu24_socket_count], variables out of range []"

GridFS for large documents

Posted: Sat Apr 27, 2013 5:48 am
by andg
now lets say we want to tie in a shutter from an external web cam.
we need to tie in an embedded document within the logger with the image captured from the camera so it gets tied to the sensors

BSON id begins with the timestamp so just by logging data we get the timestamp and we can log any type of data, since the nature of binary storage is to save raw data. GridFS is also available if you intend to save very large dumps. GridFS would be applicable if you wanted to locally save images of computers under version control. Replication would come in handy but these machines should have a network control task to limit their bandwith, monitored, reported.

binary data storage makes storing any types of documents. UTF-8 seems to work well for me and file storage, and sensor data are all well optimized

The problem comes in with the overhead of a large database engine on top of small memory footprints. Transactional data being tied directly to point in time better suited application. User registrations, orders, inventory with relationships are far better suited. Mongo facilitates this through doc ref

Code: Select all

user
{_id,
name: {first: "Andrew",last: "Gauger"}}
order
{_id,
user: { "$ref" : user, "$id" : BSON}

Code: Select all

db.order.find( { ... }).user.name,first

Precision

Posted: Sat Apr 27, 2013 5:54 am
by andg
.sensor from laser readings nightly to apollo 1 Gw laser measuring the distance to a trillinth.

Code: Select all

{_id,
timestamp: 64bit_int,
distance: 356,700.83482019383433
}
allows very precise point in time to dynamic events

Further precision can be created by continually updating thought the millisecond. These inserts can then be averaged.

Installation distributed computing and timing

Posted: Sat Apr 27, 2013 5:56 am
by andg
oscilllation and synch
the per second signal is ignored, it is just log data. It may come in faster or slower depending on CPU utilization

basing distributed storage on universal cycle count of an engine rather than per second or per transaction.

In distributed environments dedicated database quickly becomes off loaded. development works best with a local instance. Mongo is easy to set up and works in Linux and mac very easily installation by environment

Re: json, CAN

Posted: Sat Apr 27, 2013 5:58 am
by andg
allows abstraction of bi tools to analyze pulse of signal that can be tied to signals that include voltages of every sensor providing feedback

Analyzing sensors with direct access to query the signals at different intervals within the stroke will allow tuning control to tie back into the forecasted tables. It is the only way to reliably test.

Re: json, CAN

Posted: Sat Apr 27, 2013 6:07 am
by andg
controllable multi core
assign processes to muliple cores based on number of detected cores as indicated by hearbeat of co processors

Ever played a ps3? coda

google search
also an example of point in time data.

monitoring of an online app is the single greatest ROI example

So here is where things get really interesting. We no longer need an engine mangement system. The car is autonomous. The injectors sense that the key is turned since they received their wake up 1x cam signal. They know to fire and they will begin to do so.

Spark is already calculated at the GM DIS interface this is really the basis of the project. This is where a 6 and 1 7x signal is translated to 3x which will inform the system of the 3x signal and the injectors will self align since they have their own computer chip in line with the harness that receives the CAN signal.This cpu is responsible for sending the MAP and CHT back to the system.

Re: json, CAN

Posted: Sat Apr 27, 2013 6:08 am
by andg
If multiple cores of co processors and I need to re google incase it was cuda processors.
We timestamp whenever the processor is able to process in the most thread safe manner

Also designing the system to handle failures by passing through the original voltage back through the harness. an inline fail back controller exists inline with the existing ecu as backup