Basic Datalog - What To Put In It?? Your thoughts!

[Fred]

One idea :
We can define the format to be variable length. Older devices can ignore newer fields. That way if we are a bit wrong we can tag stuff on the end later.

Pulling stuff out won't be OK though, so we should think carefully about this anyway.

Extending this idea a little...

What if we put the key things at the front of the packet and more stuff towards the end with less priority and have a length config on the device. We could specify the protocol to say all of the ones we know of at any given release and anything receiving one could just read in up until either its knowledge runs out, or the packet runs out and not populate other fields there after.

I'm going to do a list in the first post now based on what you guys have written and the structs.h file and will bump again when I have.

Fred.

[gearhead]

don't forget the bit registers, too. we want to know which mode it is in. Crank, Warmup, Run, Idle control, etc...

KeithG

[Fred]

don't forget the bit registers, too. we want to know which mode it is in. Crank, Warmup, Run, Idle control, etc...

You are SO MS :-p accel mode, warmup mode, these will be represented by values only. IE, if the value is zero, its not "in that mode" ;-) cranking and running are one and the same thing.

Bit registers were suggested somewhere above and will be listed in the first post in less than an hour :-)

Thanks for the reminder :-)

Fred.

[jharvey]

You are SO MS :-p accel mode, warmup mode, these will be represented by values only.

So what about MS's ass mode. I'm hoping you make that a value of 0 here....

[Fred]

Fred, I think you'd save yourself a lot of time and effort if you'd explain better what you're after from the start. It seems like you're interested in what would be useful information to the driver of a car. Or is it the tuner of an engine?

I guess it is two fold :
1) Display stuff for the driver OR tuner
2) A generic potentially long term log that you can look back on when something funky happens

1) requires very little, but 2) requires quite a bit more

I've got a massive list forming here. It will be useful for knowing what to add and how to structure the data inside the code too. This thread has been very valuable! :-)

I'll tag the post on the end here as well as in the first post.

Fred.

[Fred]

Logging methods planned :

1. Basic fixed - Request (with length)/Response
2. Basic fixed - Request (length from flash)/Response
3. Basic fixed - Async continunous (length from flash)
4. Configurable - Request (with definition)/Response
5. Configurable - Request (definition from flash)/Response
6. Configurable - Async continuous (definition from flash)

If we go with the variable length idea (and I can't see any reason not to) then we'll need that 6th option too :-)

The restrictions then would be :

Order is fixed
Can't add in or remove things from the middle/at the front

However the flexibility of being able to cut it down or extend it on the trailing edge for uber fast readings or more info by restricting or adding to the length would be very handy IMO.

Fred.

[Fred]

Here ya go :

1. ------------ Current Core Vars ------------
2. 2 IAT
3. 2 CHT
4. 2 TPS
5. 2 EGO (lambda actual)
6. 2 MAP
7. 2 AAP
8. 2 BRV
9. 2 MAT
10. 2 EGO2 (lambda actual)
11. 2 IAP
12. 2 MAF
13. 2 DMAP
14. 2 DTPS
15. 2 RPM
16. 2 DRPM
17. 2 DDRPM
18. ------------ Current Derived Vars ------------
19. 2 LoadMain
20. 2 VEMain
21. 2 Lambda (target)
22. 2 AirFlow
23. 2 densityAndFuel
24. 2 BasePW
25. 2 IDT
26. 2 ETE
27. 2 TFCTotal
28. 2 FinalPW
29. 2 RefPW
30. ------------ Suggested Definites --------------
31. Ignition advance base
32. Dwell
33. Time - See below
34. Datalog sequence number incremented per log sent - 8 bits is enough, only looking for discontinuity anyway
35. Spare variable fields to be somehow user configurable without code change 4 x 16 bits enough?)
36. Flag vars = bit fields = on/off or yes/no or 0/1 for specific states of various things
37. ----------- Specific Flags ------------
38. Fully closed throttle on/off
39. Fully open throttle on/off
40. Fuel pump on/off
41. AC input on/off
42. AC output on/off
43. Brake on/off
44. Clutch on/off
45. Idle bypass on/off
46. Box in neutral on/off
47. VVT - fixed - on/off
48. VICS - fixed - on/off
49. Power steer on/off
50. Fans (1 or 2?) on/off
51. CEL (how many) on/off
52. Primary sync on/off
53. Secondary sync on/off
54. What else? Go nuts... I can only say no :-)
55. Bank left/right of a V engine for the entire log
56. ------------ Suggested To Be Discussed ------------
57. Injection advance figures - v twin, siamese, standard, all to be considered, may require two fields.
58. TFC - per algorithm ?
59. ------------ Non core, but desirable ---------------
60. Idle valve Duty - should be generic enough to cover stepper and pwm, ie, not sure if duty is the right term
61. Knock count
62. Knock level
63. Vehicle speed sensor
64. Boost control duty
65. Cooling fan duty (1 or 2)
66. VVT angle value or duty value or both
67. EGT sensor
68. Fuel pressure
69. Fuel temperature
70. All fuel pulsewidths
71. All ign advances
72. Staged pulsewidth
73. Timing error
74. O2 Correction level
75. Trigger input angle logging - primary and secondary
76. What else? Go nuts... I can only say no :-)

Bit fields are cheap, so there should be more than enough of those to cater into the future. Word fields are expensive, so we should think carefully about what to include and what not to include.

I think the structures used need some rework. I'd like to migrate the rpms and delta map out of core vars and replace them with ADC figures that correspond to the raw ADC array. I think some of the stuff in derived vars doesn't belong there too... We will also need some sort of accessory output struct or similar. Finally, I think it would be nice to have them all 16 fields long such that it is easy to mask them off on an individual basis for the configurable log.

Some sort of internal running time stamp - perhaps add another clock that incs when some other rolls over? or perhaps just use the 32 bit ECT for this? Or maybe we should use a real time value from the clocks section so we have a time frame for everything.

Additionally, the following fields are also available :

RAW ADCs (useful to see noise pre averaging) :

1. 2 IAT
2. 2 CHT
3. 2 TPS
4. 2 EGO
5. 2 BRV
6. 2 MAP
7. 2 AAP
8. 2 MAT
9. 2 EGO2
10. 2 IAP
11. 2 MAF
12. 2 SpareADC3
13. 2 SpareADC4
14. 2 SpareADC5
15. 2 SpareADC6
16. 2 SpareADC7

Clocks :

1. 2 realTimeClockMain
2. 2 realTimeClockMillis
3. 2 realTimeClockTenths
4. 2 realTimeClockSeconds
5. 2 realTimeClockMinutes
6. 2 millisToTenths
7. 2 tenthsToSeconds
8. 2 secondsToMinutes
9. 2 timeoutADCreadingClock

ISR Latencies :

1. 2 primaryInputLatency
2. 2 secondaryInputLatency
3. 2 Injector1Latency
4. 2 Injector2Latency
5. 2 Injector3Latency
6. 2 Injector4Latency
7. 2 Injector5Latency
8. 2 Injector6Latency
9. 2 DwellLatency
10. 2 IgniteLatency
11. 2 mathLatency ??
12. 2 mathSampleTimeStamp [2] ??

(Can't remember what the story with those last math ones is)

Runtimes :

1. 2 primaryInputLeadingRuntime
2. 2 primaryInputTrailingRuntime
3. 2 secondaryInputLeadingRuntime
4. 2 secondaryInputTrailingRuntime
5. 2 calcsRuntime
6. 2 genCoreVarsRuntime
7. 2 genDerivedVarsRuntime
8. 2 mathTotalRuntime
9. 2 mathSumRuntime
10. 2 RTCRuntime
11. 2 mainLoopRuntime
12. 2 logSendingRuntime
13. 2 serialISRRuntime

Counters :

1. 2 UISRCounter
2. 2 lowVoltISRCounter
3. 2 lostCrankSyncCounter
4. 2 lostCamSyncCounter
5. 2 lostRPMValidityCounter
6. 2 primaryTeethDroppedFromLackOfSync
7. 2 primaryTeethCounter
8. 2 secondaryTeethCounter
9. 2 syncedADCreadingCounter
10. 2 timeoutADCreadingCounter
11. 2 calcsPerformedCounter
12. 2 logsSentCounter
13. 2 serialPacketOverLengthCounter
14. 2 serialStartInsidePacketCounter
15. 2 serialEscapePairMismatchCounter
16. 2 serialChecksumMismatchCounter
17. 2 serialNoiseErrorCounter
18. 2 serialOverrunErrorCounter
19. 2 serialFramingErrorCounter
20. 2 serialParityErrorCounter
21. 2 serialDebugUnsentCounter

Many thanks to Jared's doxygen efforts which made creating this post a lot easier!!

Many thanks to everyone that put ideas in too, keep them coming :-)

Fred.

[y09223]

A common approach that is used in satellite telemetry is to divide data into multiple levels. Level 0 data is raw, unprocessed data, generally binary states and ADC/DAC counts. Level 1 is minimally processed data, for example ADC/DAC counts having been converted to engineering/scientific units. Level 2 and above are derived, combined, and reduced. The more processing that's done, the more parameters are available for logging - in other words: data bloat. Capturing all data and parameters requires copious processor and communciation bandwidth. The only way to reliably log EMS operation is to log Level 0 or 1 data. Since the EMS algorithms are known, total EMS operation can be re-created from the Level 0 or 1 data with appropriate post-processing.

Only my 2 cents worth.

John

[y09223]

One additional thought. To understand what the EMS is doing via the logs, log entries need to either be generated at fixed time intervals, or each entry needs to be timestamped. Timestamping is an absolute necessity if entries are generated asynchronously.

[Fred]

Agreed on the recreate thing, except that a large part of the logs purpose early on is to catch bugs in my code and that doing only level zero will get you double the work load (for the tuning author). Also, for example, if the ADCs are filtered then you need that filtered value at the very least.

I 100% agree on the time stamping being a good idea, but disagree that it is 100% required, remember that we are mostly trying to tune an engine, ie, we want to know what its current advance, dwell, AFR, load and RPM are so we can trim them to be correct.

I like the idea of sending only the raw stuff, but I don't think it's a practical one, especially given that we have all the variables ready to send as actually used by the code and sending a good sized set of them can produce about 67/s rate which is more than adequate and 3x - 6x what our competitor is producing :-)

Bandwidth is an issue, as is change over time, hence this thread, but CPU time isn't really. Most people will configure logging off when just using the car and thus the CPU will have max time to do good stuff. I calculated (on the high side) that we'll burn about 25% of CPU sending a constant maxed out stream.

Well considered 2c such as yours are always welcome!

Fred.