FreeEMS-Tuner Development Diary - Comments

[sry_not4sale]

I captured all that text into a file and catted it out, similar CPU usage Still, it is a LOT of text : 1 meg

I'll zip and send. You can clearly find good packets in amongst it though which means your error handling is dodgy. It should never be possible to get massive blocks like that. Each AA start byte should mean a new block. Not sure what you are up to, but hopefully the text dump helps you out!

Fred.

EDIT BZIP2 FTW : 1meg >> 5k LOL must be a bit of parse-able repetition in there ey

Hey Fred,

I went through this log file, and apart from spewing out far too much debug information (it includes the rest of the buffer in debug messages - which I have now fixed) it does everything right.

I think your code is off! Checksums wrong in some packets.

Tell me this is right?!?:

Code: Select all

0xAA,0x08,0x01,0x91,0x00,0x60,0x63,0x25,0x64,0x5E,0x00,0x00,0x62,0x80,0xF3,0x1C,0x43,0x7C,0xE3,
0x51,0x60,0x63,0x25,0x64,0x5E,0x00,0x00,0x62,0x80,0xF3,0x1C,0x43,0x7C,0xE3,0x51,0x71,0xCA,0x27,
0x10,0x80,0x00,0x00,0x03,0x00,0x00,0x00,0x00,0xFD,0xD4,0x00,0x00,0x00,0x00,0xF3,0x1C,0xFD,0xD3,
0x5F,0xC8,0x34,0x89,0x34,0x27,0x3F,0x3B,0x00,0x00,0x00,0x00,0x00,0x00,0x3F,0x3B,0x3F,0x3B,0x00,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x03,0x86,0x03,0x6F,0x02,0x32,0x02,0x13,0x03,0x63,
0x03,0x98,0x03,0xD5,0x02,0x07,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
0x00,0x00,0x00,0xA0,0xCC

Checksum is incorrect! Provided: 160, generated: 46

[Fred]

I would imagine it would be easy to add another serial adapter style plugin? If so, make one for file reading so we can read in binary streams from disk. This will enable us to have 100% certainty of what is and isn't coming in and how it should or shouldn't behave. I can easily generate binary files with data streams in them from cutecom. You have one in your mail :-)

If you do, perhaps you should split up the stream with a timer into chunks such that packets come through in pieces like they would in real life?

Fred.

[sry_not4sale]

Will do that next, and also logging to file.

[Fred]

I'm 2/3 of the way through writing a C CLI util to read and parse those binary files and output some stats on them. I'll run that file I sent you through it as soon as I complete it and let you know if I suck or you suck :-)

So far it runs almost instantly. You may be able to use it as a python plugin or something if you can't make python fast?

Code: Select all

fred@rwdlsd:~$ time freeemsParser binary.stream.freeems
The length of file binary.stream.freeems is 469712

Attempting to parse file...
There were 464933 chars dropped and 4779 packets of which 0 had bad checksums.

real    0m0.031s
user    0m0.028s
sys     0m0.000s

Obviously that isn't actually checking checksums yet.. the packet count should be true or close to it though.

Fred.

[Fred]

Code: Select all

fred@rwdlsd:~$ time freeemsParser binary.stream.freeems
The length of file binary.stream.freeems is 469712

Attempting to parse file...
There were 81 chars dropped and 21 packets of which 4209 had bad checksums.

real    0m0.034s
user    0m0.032s
sys     0m0.000s

Needs work LOL, but getting there :-) Note, processing escapes etc didn't slow it down pretty much at all.

Fred.

[Fred]

Code: Select all

Data stream statistics :
4230 packets were found of which 2661 had bad checksums.
81 leading characters were dropped
548 false starts occurred
1182 escape bytes were found
20 escaped stop bytes were found
26 escaped start bytes were found
1136 escaped escape bytes were found
0 escape pairs were mismatched

real    0m1.455s
user    0m0.032s
sys     0m0.032s

OK, either my code sucks, or my serial adapter sucks. Given that the adapter is made in china and purchased from dick smith, I'm crossing my fingers that my code is good.

I'll have to get Sean and others to help me verify this I think.

The extra time it took was purely dumping to the console across a lan and ssh. the actual code is fast.

Fred.

[Fred]

I've picked up a new USB serial adapter and it exhibits the same behaviour. It would be nice to know what was going wrong with the data. Perhaps this time I'll leave the settings on the device all the same through the run. Perhaps it's some obvious corruption due to the speed I'm transmitting at and the frequency discrepancy.

Fred.

[Fred]

Code: Select all

Data stream statistics :
13342 packets were found of which 1244 had bad checksums.
38 leading characters were dropped
1252 false starts occurred
9364 escape bytes were found
98 escaped stop bytes were found
9012 escaped start bytes were found
254 escaped escape bytes were found
0 escape pairs were mismatched

OK, the 1244 in this sequence which I kept parsing while it was still recording were ALL generated while the RPM was non-zero. IE, there is a corruption and/or concurrency issue with respect interrupts and serial transmission. I'll file a mantis against myself over this and will now attempt to run the tuner against it with the JimStim off.

Thank christ I created this utility, it's fantastic :-)

Fred.

[Fred]

http://freeems.aaronb.info/tracker/view.php?id=99

Read all about it there :-)

Additionally some serial adapter advice is here :

http://www.diyefi.org/forum/viewtopic.php?f=43&t=585

I'm now chasing another issue as displayed when I ran Aaron's code against mine and confirmed by my nifty little tool. Read about that here :

http://freeems.aaronb.info/tracker/view.php?id=81

Fred.

[Fred]

Weird :

Code: Select all

fred@rwdlsd:~$ time freeemsParser binary.stream.freeems11
The length of file binary.stream.freeems11 is 549656

Attempting to parse file...

Stray data or unfinished packet found following packet number 0
Packet number 1 ending of length 10 at char number 136 failed checksum! Received 1 Calculated 35
Packet number 2 ending of length 79 at char number 217 failed checksum! Received 1 Calculated 128
Stray data or unfinished packet found following packet number 2
Packet number 3 ending of length 166 at char number 396 failed checksum! Received 1 Calculated 113
Packet number 4 ending of length 81 at char number 479 failed checksum! Received 1 Calculated 97
Stray data or unfinished packet found following packet number 476
Double start byte occurred following packet number 476
Stray data or unfinished packet found following packet number 523
Double start byte occurred following packet number 523
Stray data or unfinished packet found following packet number 1078
Double start byte occurred following packet number 1078
Stray data or unfinished packet found following packet number 1083
Double start byte occurred following packet number 1083
Stray data or unfinished packet found following packet number 1106
Double start byte occurred following packet number 1106
Stray data or unfinished packet found following packet number 1109
Double start byte occurred following packet number 1109
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2005
Double start byte occurred following packet number 2005
Stray data or unfinished packet found following packet number 2073
Double start byte occurred following packet number 2073
Stray data or unfinished packet found following packet number 2082
Double start byte occurred following packet number 2082
Stray data or unfinished packet found following packet number 2248
Double start byte occurred following packet number 2248
Stray data or unfinished packet found following packet number 2265
Double start byte occurred following packet number 2265
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2409
Double start byte occurred following packet number 2409
Stray data or unfinished packet found following packet number 2515
Double start byte occurred following packet number 2515
Stray data or unfinished packet found following packet number 2551
Double start byte occurred following packet number 2551
Stray data or unfinished packet found following packet number 2557
Double start byte occurred following packet number 2557
Stray data or unfinished packet found following packet number 2771
Double start byte occurred following packet number 2771
Stray data or unfinished packet found following packet number 2789
Double start byte occurred following packet number 2789
Stray data or unfinished packet found following packet number 2990
Double start byte occurred following packet number 2990
Stray data or unfinished packet found following packet number 3008
Double start byte occurred following packet number 3008
Stray data or unfinished packet found following packet number 3035
Double start byte occurred following packet number 3035
Stray data or unfinished packet found following packet number 3037
Double start byte occurred following packet number 3037
Stray data or unfinished packet found following packet number 3049
Double start byte occurred following packet number 3049
Stray data or unfinished packet found following packet number 3066
Double start byte occurred following packet number 3066
Stray data or unfinished packet found following packet number 3174
Double start byte occurred following packet number 3174
Stray data or unfinished packet found following packet number 3239
Double start byte occurred following packet number 3239
Stray data or unfinished packet found following packet number 3274
Double start byte occurred following packet number 3274
Stray data or unfinished packet found following packet number 3333
Double start byte occurred following packet number 3333
Stray data or unfinished packet found following packet number 3341
Double start byte occurred following packet number 3341
Stray data or unfinished packet found following packet number 3410
Double start byte occurred following packet number 3410
Stray data or unfinished packet found following packet number 3410
Double start byte occurred following packet number 3410
Stray data or unfinished packet found following packet number 3452
Double start byte occurred following packet number 3452
Stray data or unfinished packet found following packet number 3522
Double start byte occurred following packet number 3522
Stray data or unfinished packet found following packet number 3597
Double start byte occurred following packet number 3597
Stray data or unfinished packet found following packet number 3597
Double start byte occurred following packet number 3597
Stray data or unfinished packet found following packet number 3661
Double start byte occurred following packet number 3661
Stray data or unfinished packet found following packet number 3689
Double start byte occurred following packet number 3689
Stray data or unfinished packet found following packet number 3874
Double start byte occurred following packet number 3874
Stray data or unfinished packet found following packet number 3879
Double start byte occurred following packet number 3879
Stray data or unfinished packet found following packet number 3945
Double start byte occurred following packet number 3945
Stray data or unfinished packet found following packet number 3951
Double start byte occurred following packet number 3951
Stray data or unfinished packet found following packet number 3957
Double start byte occurred following packet number 3957
Stray data or unfinished packet found following packet number 3963
Double start byte occurred following packet number 3963
Stray data or unfinished packet found following packet number 4112
Double start byte occurred following packet number 4112
Stray data or unfinished packet found following packet number 4139
Double start byte occurred following packet number 4139
Stray data or unfinished packet found following packet number 4139
Double start byte occurred following packet number 4139
Stray data or unfinished packet found following packet number 4160
Double start byte occurred following packet number 4160
Stray data or unfinished packet found following packet number 4958
Double start byte occurred following packet number 4958
Stray data or unfinished packet found following packet number 4959
Double start byte occurred following packet number 4959
Stray data or unfinished packet found following packet number 4960
Double start byte occurred following packet number 4960
Stray data or unfinished packet found following packet number 4961
Double start byte occurred following packet number 4961
Stray data or unfinished packet found following packet number 4962
Double start byte occurred following packet number 4962
Stray data or unfinished packet found following packet number 4963
Double start byte occurred following packet number 4963
Stray data or unfinished packet found following packet number 4966
Double start byte occurred following packet number 4966
Stray data or unfinished packet found following packet number 4967
Double start byte occurred following packet number 4967
Stray data or unfinished packet found following packet number 4968
Double start byte occurred following packet number 4968
Stray data or unfinished packet found following packet number 4969
Double start byte occurred following packet number 4969
Stray data or unfinished packet found following packet number 4970
Double start byte occurred following packet number 4970
Stray data or unfinished packet found following packet number 4971
Double start byte occurred following packet number 4971
Stray data or unfinished packet found following packet number 4972
Double start byte occurred following packet number 4972
Stray data or unfinished packet found following packet number 4973
Double start byte occurred following packet number 4973
Stray data or unfinished packet found following packet number 4974
Double start byte occurred following packet number 4974
Stray data or unfinished packet found following packet number 4975
Double start byte occurred following packet number 4975
Stray data or unfinished packet found following packet number 4976
Double start byte occurred following packet number 4976
Stray data or unfinished packet found following packet number 4977
Double start byte occurred following packet number 4977
Stray data or unfinished packet found following packet number 4978
Double start byte occurred following packet number 4978
Stray data or unfinished packet found following packet number 4979
Double start byte occurred following packet number 4979
Stray data or unfinished packet found following packet number 4980
Double start byte occurred following packet number 4980
Stray data or unfinished packet found following packet number 4981
Double start byte occurred following packet number 4981
Stray data or unfinished packet found following packet number 4982
Double start byte occurred following packet number 4982

Data stream statistics :

Packets and checksums :
5154 packets were found
5150 had good checksums
4 had incorrect checksums

General issues :
19 leading characters were dropped
232 false starts occurred
115 double start bytes occurred
117 stray part packets occurred
11844 chars lost from false starts 

Escaped byte profile :
1611 escape bytes were found
0 escaped stop bytes were found
1611 escaped start bytes were found
0 escaped escape bytes were found
0 escape pairs were mismatched

real    0m0.055s
user    0m0.036s
sys     0m0.012s

I thought it was suspicious that there were exactly and almost exactly the same number of double starts and lost ends... so I added the two log messages and low and behold they come in pairs. I need to understand that better at this stage, but I've put in a hack that I hope helps. If it does, sweet, if not, then there is a more serious issue to deal with, possible frequency of connection related?

Fred.