FreeEMS-Tuner Development Diary - Don't Post!

[sry_not4sale]

I have been talking to some guys in the local python user group, and it seems the use of python lists is what is slowing the app down so much.

Apparently switch to a native binary data type (the structs module) would give us a "thousand fold" increase

So thats the current job

[sry_not4sale]

My Toshiba laptop (only home computer at the moment (EEEpc died recently as well )) has died

Will let you guys know how things go. Hard drive is ok thank god, but looks like either mobo, graphics card or ram is fuxxord

[sry_not4sale]

Have a new (old) laptop Acer Aspire 5000 widescreen, running Ubuntu 8.10.

Everything worked out of the box except wifi which only involved apt-getting a package

So, back into code!

Just pushed a commit (probably shouldn't have to master tho, but I did) that updates the serial protocol to match Fred's proposed changes

[sry_not4sale]

So.... got heaps done on the train this morning.

I am moving away from using python lists in the Tuner, and trying to stay with more binary suitable data types. It was suggested to me to use the Structs module, however it doesn't handle variable length packets

Sooo... I decided to convert all the code to use 8 bit strings and avoid looping as much as possible. Python 2.6/3.0 have a binary string data type, but I want to keep it 2.5 compatible.

Spent an hour on it this morning and pretty much have it almost converted. So far I have taken out twice as much code as I have added/modified, and removed almost every loop I have come across. The receive buffer processing code has gone from two monolithic functions, to two tiny ones

I have updated

Code: Select all

devtrunk

with my progress (although it hasn't been tested/prob wont run). Should have a working version tomorrow!! Tempted to work out in now at work, but it'll wait until tonight.

[sry_not4sale]

So pretty much finished the port.

Builds and sends packets fine. Even fixed a bug I didn't realise existed until I compared the output lol. Displays them in the gui grid fine too.

However, there is some bugs in the receive buffer parsing code. Its throwing a checksum error, but worse than that is it is not removing parsed bytes from the buffer. So going to look into this now. Won't push until I have fixed it.

Also, guy from the local python user group suggested that although the struct module doesn't handle variable length packets. I could just parse the header with it, discover the length of the payload and then parse out that. Problem with that is I would have to look up the rules for that packet to see if it was fixed length, etc, etc. So I think I will just stick with the string implementation I have, I think it is going to be a hell of a lot faster than it was anyways.

I have used the struct module in place of some perhaps dodgy int to 8bit conversion code. So now instead of from8bit() and to8bit(), we have shortFrom8bit(), intFrom8bit(), longFrom8bit(), shortTo8bit(), etc, etc.

[sry_not4sale]

Ok done a whole heap more testing, fixing.

Turns out the processed bytes weren't being remove from the buffer because I was trying to pass it by reference to the _processBuffer() method. Strings are immutable in python, so when they are modified you are creating a new string. Hope you can see where I am going with that...

Anyway, that is fixed now. Receives buffer beautifully, and processes and displays received packets no problem. Until a byte needs to be escaped that is. There is a bug causing an infinite loop in there I need to fix.

Found a great article regarding efficent string concatentation in python. If my new code doesn't produce the massive speed increase we require, I want to create a CLI for the comms/protocol code and do some tests using some of the alternative methods in that article, and some other ideas I have

[sry_not4sale]

Fixed the escaping bug, turns out it was an unescaping bug lol

Created a test packet with a payload of the bytes 0x00 - 0xFF, but that found a bug in the escaping code where the escape byte wasn't being escaped...

[sry_not4sale]

Fixed bug! Pushed to devtrunk.

Your turn Fred testy testy!

I think there might be another bug tho, I have changed the request test packet to send the 0x00 - 0xFF bytes instead of "test". Sends find, Test response packet doesn't through any errors. However, I don't think its escaping correctly. Or maybe its escaping stuff twice.
Maybe someone could test the new code and have a look? My brain has turned to mush

[sry_not4sale]

No bugs fixed last night, but I did create a very simple CLI test framework and tested a couple of bugs.

So now I have a test framework to work against as I fix bugs, to make sure there aren't any regressions e.g. like it not handling 0xAA,0xAA properly anymore

I'll give the test framework a small tidyup and make it read from a json file so Fred can add his own test packets to it

[sry_not4sale]

Done heaps on the test frame work now

Now tests:

Code: Select all

    test_good,
    test_threeGood,
    test_doubleStart,
    test_oneBadByte,
    test_twoBadBytes,
    test_badByteBeforePacket,
    test_badBytesBetweenPackets

And fixed the receive code to actually pass them all

It wasn't handling erroneous start bytes very well, so that is fixed. It also was doing a lot of copying of the buffer, which I have taken steps to reduce.

Changes pushed to devtrunk.

The testing interface is a bit messy, but it works. If anyone wants to make some improvements you are more than welcome!!!!