FreeEMS-Tuner Development Diary - Don't Post!
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
I have been talking to some guys in the local python user group, and it seems the use of python lists is what is slowing the app down so much.
Apparently switch to a native binary data type (the structs module) would give us a "thousand fold" increase
So thats the current job
Apparently switch to a native binary data type (the structs module) would give us a "thousand fold" increase
So thats the current job
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
My Toshiba laptop (only home computer at the moment (EEEpc died recently as well )) has died
Will let you guys know how things go. Hard drive is ok thank god, but looks like either mobo, graphics card or ram is fuxxord
Will let you guys know how things go. Hard drive is ok thank god, but looks like either mobo, graphics card or ram is fuxxord
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
Have a new (old) laptop Acer Aspire 5000 widescreen, running Ubuntu 8.10.
Everything worked out of the box except wifi which only involved apt-getting a package
So, back into code!
Just pushed a commit (probably shouldn't have to master tho, but I did) that updates the serial protocol to match Fred's proposed changes
Everything worked out of the box except wifi which only involved apt-getting a package
So, back into code!
Just pushed a commit (probably shouldn't have to master tho, but I did) that updates the serial protocol to match Fred's proposed changes
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
So.... got heaps done on the train this morning.
I am moving away from using python lists in the Tuner, and trying to stay with more binary suitable data types. It was suggested to me to use the Structs module, however it doesn't handle variable length packets
Sooo... I decided to convert all the code to use 8 bit strings and avoid looping as much as possible. Python 2.6/3.0 have a binary string data type, but I want to keep it 2.5 compatible.
Spent an hour on it this morning and pretty much have it almost converted. So far I have taken out twice as much code as I have added/modified, and removed almost every loop I have come across. The receive buffer processing code has gone from two monolithic functions, to two tiny ones
I have updated with my progress (although it hasn't been tested/prob wont run). Should have a working version tomorrow!! Tempted to work out in now at work, but it'll wait until tonight.
I am moving away from using python lists in the Tuner, and trying to stay with more binary suitable data types. It was suggested to me to use the Structs module, however it doesn't handle variable length packets
Sooo... I decided to convert all the code to use 8 bit strings and avoid looping as much as possible. Python 2.6/3.0 have a binary string data type, but I want to keep it 2.5 compatible.
Spent an hour on it this morning and pretty much have it almost converted. So far I have taken out twice as much code as I have added/modified, and removed almost every loop I have come across. The receive buffer processing code has gone from two monolithic functions, to two tiny ones
I have updated
Code: Select all
devtrunk
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
So pretty much finished the port.
Builds and sends packets fine. Even fixed a bug I didn't realise existed until I compared the output lol. Displays them in the gui grid fine too.
However, there is some bugs in the receive buffer parsing code. Its throwing a checksum error, but worse than that is it is not removing parsed bytes from the buffer. So going to look into this now. Won't push until I have fixed it.
Also, guy from the local python user group suggested that although the struct module doesn't handle variable length packets. I could just parse the header with it, discover the length of the payload and then parse out that. Problem with that is I would have to look up the rules for that packet to see if it was fixed length, etc, etc. So I think I will just stick with the string implementation I have, I think it is going to be a hell of a lot faster than it was anyways.
I have used the struct module in place of some perhaps dodgy int to 8bit conversion code. So now instead of from8bit() and to8bit(), we have shortFrom8bit(), intFrom8bit(), longFrom8bit(), shortTo8bit(), etc, etc.
Builds and sends packets fine. Even fixed a bug I didn't realise existed until I compared the output lol. Displays them in the gui grid fine too.
However, there is some bugs in the receive buffer parsing code. Its throwing a checksum error, but worse than that is it is not removing parsed bytes from the buffer. So going to look into this now. Won't push until I have fixed it.
Also, guy from the local python user group suggested that although the struct module doesn't handle variable length packets. I could just parse the header with it, discover the length of the payload and then parse out that. Problem with that is I would have to look up the rules for that packet to see if it was fixed length, etc, etc. So I think I will just stick with the string implementation I have, I think it is going to be a hell of a lot faster than it was anyways.
I have used the struct module in place of some perhaps dodgy int to 8bit conversion code. So now instead of from8bit() and to8bit(), we have shortFrom8bit(), intFrom8bit(), longFrom8bit(), shortTo8bit(), etc, etc.
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
Ok done a whole heap more testing, fixing.
Turns out the processed bytes weren't being remove from the buffer because I was trying to pass it by reference to the _processBuffer() method. Strings are immutable in python, so when they are modified you are creating a new string. Hope you can see where I am going with that...
Anyway, that is fixed now. Receives buffer beautifully, and processes and displays received packets no problem. Until a byte needs to be escaped that is. There is a bug causing an infinite loop in there I need to fix.
Found a great article regarding efficent string concatentation in python. If my new code doesn't produce the massive speed increase we require, I want to create a CLI for the comms/protocol code and do some tests using some of the alternative methods in that article, and some other ideas I have
Turns out the processed bytes weren't being remove from the buffer because I was trying to pass it by reference to the _processBuffer() method. Strings are immutable in python, so when they are modified you are creating a new string. Hope you can see where I am going with that...
Anyway, that is fixed now. Receives buffer beautifully, and processes and displays received packets no problem. Until a byte needs to be escaped that is. There is a bug causing an infinite loop in there I need to fix.
Found a great article regarding efficent string concatentation in python. If my new code doesn't produce the massive speed increase we require, I want to create a CLI for the comms/protocol code and do some tests using some of the alternative methods in that article, and some other ideas I have
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
Fixed the escaping bug, turns out it was an unescaping bug lol
Created a test packet with a payload of the bytes 0x00 - 0xFF, but that found a bug in the escaping code where the escape byte wasn't being escaped...
Created a test packet with a payload of the bytes 0x00 - 0xFF, but that found a bug in the escaping code where the escape byte wasn't being escaped...
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
Fixed bug! Pushed to devtrunk.
Your turn Fred testy testy!
I think there might be another bug tho, I have changed the request test packet to send the 0x00 - 0xFF bytes instead of "test". Sends find, Test response packet doesn't through any errors. However, I don't think its escaping correctly. Or maybe its escaping stuff twice.
Maybe someone could test the new code and have a look? My brain has turned to mush
Your turn Fred testy testy!
I think there might be another bug tho, I have changed the request test packet to send the 0x00 - 0xFF bytes instead of "test". Sends find, Test response packet doesn't through any errors. However, I don't think its escaping correctly. Or maybe its escaping stuff twice.
Maybe someone could test the new code and have a look? My brain has turned to mush
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
No bugs fixed last night, but I did create a very simple CLI test framework and tested a couple of bugs.
So now I have a test framework to work against as I fix bugs, to make sure there aren't any regressions e.g. like it not handling 0xAA,0xAA properly anymore
I'll give the test framework a small tidyup and make it read from a json file so Fred can add his own test packets to it
So now I have a test framework to work against as I fix bugs, to make sure there aren't any regressions e.g. like it not handling 0xAA,0xAA properly anymore
I'll give the test framework a small tidyup and make it read from a json file so Fred can add his own test packets to it
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82
- sry_not4sale
- LQFP144 - On Top Of The Game
- Posts: 568
- Joined: Mon Mar 31, 2008 12:47 am
- Location: New Zealand, land of the long white burnout
- Contact:
Re: FreeEMS-Tuner Development Diary - Don't Post!
Done heaps on the test frame work now
Now tests:
And fixed the receive code to actually pass them all
It wasn't handling erroneous start bytes very well, so that is fixed. It also was doing a lot of copying of the buffer, which I have taken steps to reduce.
Changes pushed to devtrunk.
The testing interface is a bit messy, but it works. If anyone wants to make some improvements you are more than welcome!!!!
Now tests:
Code: Select all
test_good,
test_threeGood,
test_doubleStart,
test_oneBadByte,
test_twoBadBytes,
test_badByteBeforePacket,
test_badBytesBetweenPackets
It wasn't handling erroneous start bytes very well, so that is fixed. It also was doing a lot of copying of the buffer, which I have taken steps to reduce.
Changes pushed to devtrunk.
The testing interface is a bit messy, but it works. If anyone wants to make some improvements you are more than welcome!!!!
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
165hp @ 6psi standard - fastest production car in japan Oct 82