FreeEMS-Loader

FreeEMS topics that aren't specific to hardware development or firmware development.
User avatar
SleepyKeys
LQFP144 - On Top Of The Game
Posts: 549
Joined: Mon Feb 11, 2008 10:52 pm
Location: Arizona
Contact:

FreeEMS-Loader

Post by SleepyKeys »

Although my firmware duties come first, I figured I would at least get started on this. I have decided on using QT/c++.

I see two different modes/stages at this point. An upgrade mode that will require certain firmware functions to operate. In this mode the firmware's integrity can be checked after/during a burn(among other things) which would be nice. A force burn mode that will require use of the BDM to initially get the loader firmware(or possibility put these functions into freeEMS firmware releases) on the mcu, after which it can be rebooted and upgraded. Right now hcs12mem can be used for the "force burn" so I think I will give that a lower priority.

Right now I need some feature requirements and some input on my GUI.

-sean

Image
Rip it up
You snooze, you lose!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS-Loader

Post by Fred »

Beware of bell/whistle ideas :

By checksum, do you mean something like an md5 that I may in future distribute with the s19 files? or what? The s19 files have checksums, but it's per line. or is that first record an overall one too? I'm not sure.

I'd like to see it have a detection feature where you tell it to figure out what it's talking to and it tells you what it's talking to and what actions you need to take from that state to get loading. I don't need this, but it idiot proofs it a bit which could cut the support load on us.

As per the comment i just put on the mantis : http://freeems.aaronb.info/tracker/view.php?id=21 I'd like to see it with the ability to just update the settings/code and just update the tables. I know gearhead would really like this and so would I actually. This could be done just with address/page ranges/sets that are supplied by me to you in some config file with the s19. most likely it won't change much/at all for a while so you could hard code the two sets for now data and the rest.

Obviously you want to be able to verify a load, but less obviously it would be nice to verify just the code at a later date.

So, to reiterate those regions :

tables of long term format
settings of dynamic nature
code of dynamic nature

tables you want to update from a stored s19 of your own or some other settings file.
settings you want to get from the firmware you are loading and should be treated with the code... except for verification. Obviously all the settings AND tables can change so ONLY the code should be verified when doing a verify at a later date. When loading a whole file the whole thing should be verified. when loading just the data just the data should be verified.

I'd like to see a transfer rate indicator, maybe two slow/fast or an adjustable averaging slider? :-) I told you bells and whistles ;-) Some sort of real time speed indicator is important though.

When loading with verify you could have a choice of two options :
fail fast
verify last

the latter is good to see what went wrong and where, the former good to not waste time and flash cycles.

speaking of which, there are no time or space constraints, so use an md5 or sha1 hash of each sector in the packet payload (if using the same packet format) then its a 100% guarantee that things are right. You could have multiple modes or combination of modes for returning data to ensure integrity :

return the actual data
return a strong hash
return both? - probably best to always do both.

what about adjustable load speed. You'd need to do a scan of your firmware to know what speed its at before telling it to change speed. If you are having checksum fails on transmission slowing it down could probably help get it burned correctly. I plan to support adjustable speed in the vanilla firmware too.

when loading, what are you thinking? something like :
  • extract data from s19 and check each line with its checksum while building an image in memory
  • generate a strong hash
  • build a packet with a sector, address, page, and hash
  • send it
  • firmware receives it
  • checks the hash against the contents
  • burns it down
  • checks the contents against the contents
  • checks the contents hash against the received hash
  • for fast fail, returns data from flash with address, page and hash
  • for verify last, returns some small ack only to say next block please
  • for verify last, send back all the blocks (keep a record of what you receive) with hashes and address/page sequentially in a row.
Sound good? :-)

Thoughts?

I don't give a toss how it looks, but others might, and good GUI feedback will make the user experience truly golden :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
gearhead
LQFP112 - Up with the play
Posts: 120
Joined: Sun Feb 03, 2008 9:30 pm
Location: Chicago, USA

Re: FreeEMS-Loader

Post by gearhead »

Wooo Hooo!. This is a fantastic idea. I never really thought about it, but then again I am not a true developer, anyway.

Yes, Yes, Yes. We absolutely need to have 3 different reserved areas that we write to when we flash something. 1) Code, 2) calibration (inc files) 3) Tuning data. Wouldn't it be awesome if we could flash a code and not have to deal with reflashing config and calibration?

One possible sticky part would be to force a wipe of the config data (or cal data if necessary, though this should be pretty constant) if there is a major change in the structure or offset of the config data.

Can this be done?

Gearhead
User avatar
SleepyKeys
LQFP144 - On Top Of The Game
Posts: 549
Joined: Mon Feb 11, 2008 10:52 pm
Location: Arizona
Contact:

Re: FreeEMS-Loader

Post by SleepyKeys »

Fred wrote:Beware of bell/whistle ideas :

By checksum, do you mean something like an md5 that I may in future distribute with the s19 files? or what? The s19 files have checksums, but it's per line. or is that first record an overall one too? I'm not sure.
Yes I think it would be a good idea to md5/sha the compiled files.
Fred wrote: I'd like to see it have a detection feature where you tell it to figure out what it's talking to and it tells you what it's talking to and what actions you need to take from that state to get loading. I don't need this, but it idiot proofs it a bit which could cut the support load on us.
With the coms protocol that shouldn't be too hard, but I was thinking of saving it for a bit later, maybe not though.
Fred wrote: As per the comment i just put on the mantis : http://freeems.aaronb.info/tracker/view.php?id=21 I'd like to see it with the ability to just update the settings/code and just update the tables. I know gearhead would really like this and so would I actually. This could be done just with address/page ranges/sets that are supplied by me to you in some config file with the s19. most likely it won't change much/at all for a while so you could hard code the two sets for now data and the rest.
Back in the beginning I was going to suggest this, never would have guessed I would be the one to implement :) Should be easy to do as long as you include a "regions" file with releases.
Fred wrote: Obviously you want to be able to verify a load, but less obviously it would be nice to verify just the code at a later date.
Should be somewhat remedial once the verify function works at the sector level. Maybe we can also add this to Tuner so it's checked every time you connect ?
Fred wrote: tables you want to update from a stored s19 of your own or some other settings file.
Shouldn't importing/exploring maps be part of tuner? Although pulling/exporting maps or from/to s19s would be sweet.
Fred wrote: I'd like to see a transfer rate indicator, maybe two slow/fast or an adjustable averaging slider? :-) I told you bells and whistles ;-) Some sort of real time speed indicator is important though.
That would be slick too, I'll keep that in mind. Would be cool to see how much overhead we have.
Fred wrote: speaking of which, there are no time or space constraints, so use an md5 or sha1 hash of each sector in the packet payload (if using the same packet format) then its a 100% guarantee that things are right. You could have multiple modes or combination of modes for returning data to ensure integrity
Thats what I had in mind. In firmware md5 engine.................. j/k Hey you said we have all the time in the world.

I'll keep the adjustable speeds in mind.

List looks good

Thoughts? I had better get started :)

I'll do my best to keep it purdy, easy enough to rearrange though in case its upsetting :)
You snooze, you lose!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS-Loader

Post by Fred »

seank wrote:Yes I think it would be a good idea to md5/sha the compiled files.
Added a note to the Makefile just now to put this in.
Fred wrote:I'd like to see it have a detection feature where you tell it to figure out what it's talking to and it tells you what it's talking to and what actions you need to take from that state to get loading. I don't need this, but it idiot proofs it a bit which could cut the support load on us.
With the coms protocol that shouldn't be too hard, but I was thinking of saving it for a bit later, maybe not though.
Definitely save it for later :-)
Back in the beginning I was going to suggest this, never would have guessed I would be the one to implement :) Should be easy to do as long as you include a "regions" file with releases.
I've thought up a few issues, but the implementation should be easy, just thinking it through is a challenge. I have a big post half typed, I'll knock it out tomorrow some time.
Maybe we can also add this to Tuner so it's checked every time you connect ?
Good idea! :-) That's a sooner for you and a much later for him though.
Fred wrote:tables you want to update from a stored s19 of your own or some other settings file.
Shouldn't importing/exploring maps be part of tuner? Although pulling/exporting maps or from/to s19s would be sweet.
Well, yes, but probably not in a bulk fashion. I'm not suggesting you know what it is that you are copying as much as just bang in a big block of stuff that the user wants from last time for whatever reason.
Fred wrote:I'd like to see a transfer rate indicator, maybe two slow/fast or an adjustable averaging slider? :-) I told you bells and whistles ;-) Some sort of real time speed indicator is important though.
That would be slick too, I'll keep that in mind. Would be cool to see how much overhead we have.
Now that I think about it, it's kinda pointless if you don't have some retry behaviour. If you did have a dirty connection and retry behaviour for dropped packets (bad checksum from corruption etc) then it would give an indication of connection quality. You could also have two : raw speed (all bytes sent, start, stop, headers the lot) and data speed, the amount of data you got burned per time. The latter will be about half or less of the former with the verification and overheads and burn/erase pauses etc. Definitely a later thing though.
Fred wrote:speaking of which, there are no time or space constraints, so use an md5 or sha1 hash of each sector in the packet payload (if using the same packet format) then its a 100% guarantee that things are right. You could have multiple modes or combination of modes for returning data to ensure integrity
Thats what I had in mind. In firmware md5 engine.................. j/k Hey you said we have all the time in the world.
Why joke? You can probably pick up some existing md5 gpl code and just port it / integrate it without too much trouble. A stronger hash than an 8 bit checksum is a good idea though. What you use is your call :-) That is also a later thing. The first step would be getting data in and out in normal checksumed packets and verifying it with return from flash itself after the burn. Maybe it never gets extended past that.
I'll keep the adjustable speeds in mind.
That's as easy as a config option. The other part is sending something to the firmware to adjust the speed. I'll implement that at some point and you can lift that code straight out of the main line.

As for implementation and staying open to easy addition of these ideas...

I assume C++ has some concept of interfaces? If so, you can create functions that say something like :

Code: Select all

boolean burnAndVerifySector(address, page) /* sends a sector to be burned and ensures it was burned as we intended. */
and implement it simply for now you can easily add to it later with extra md5 goodness etc without other aspects of the code changing. If you follow interfaces and keep the methods and classes isolated and with singular concern then making it grow should be straight forward.

I have a huge post half written that I'll put up as soon as I finish it. It's all about firmware interface versioning and meaning thereof.

Good luck! :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
SleepyKeys
LQFP144 - On Top Of The Game
Posts: 549
Joined: Mon Feb 11, 2008 10:52 pm
Location: Arizona
Contact:

Re: FreeEMS-Loader

Post by SleepyKeys »

I haven't created all of the "visual todo's" from your posts yet. I did add a few though, and kicked around the idea of a "queue list" instead of a button for each action. What do you guys think?

Image







Thanks,
Sean
You snooze, you lose!
User avatar
jharvey
1N4001 - Signed up
Posts: 1607
Joined: Tue Jun 10, 2008 5:17 pm

Re: FreeEMS-Loader

Post by jharvey »

I like the overall layout, seems intuitive to me. Looks like it could fit in 640 x 480 so it would work on older crap tops, ect. As a pinch of flash, perhaps we can include a icon like the below somewhere.

Image
User avatar
SleepyKeys
LQFP144 - On Top Of The Game
Posts: 549
Joined: Mon Feb 11, 2008 10:52 pm
Location: Arizona
Contact:

Re: FreeEMS-Loader

Post by SleepyKeys »

Front-Loaders RULE, I like it. Intuitive is the idea, Fred touched on that as well(detection ect.). I thought maybe taking the queue route would provide a simple review before actually doing anything. I have always been a function over form guy, so I'll have a few retarded UI posts until I get down to writing code for the functions. Just about anylap top that can run 640 can run 800 no ?

Thanks,
Sean
You snooze, you lose!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS-Loader

Post by Fred »

eee 701 is 800 480, planning smaller than that is screwing more of your audience than you need to :-)

I like the idea of a list of stuff it is doing and maybe highlighting in green for running and blue for done and yellow for going to be done (or whatever else you prefer). I still think you should be able to tell it what to do with buttons or menu items though. upload, download, this region range etc.

The main thing up front is to load a standard s19 to it through the sm and be able to use the whole 512k. No flash gui and stuff required for that, just a progress bar or something and a go button and way to choose the file.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS-Loader

Post by Fred »

A further idea :

An optional feature to burn only data that is different to what is already on there.

IE, for any of the things you want to burn, it first reads the block, compares with the one required and then burns only if different.

There are multiple advantages to this :

* Faster
* Less erase/write cycles on the flash.

Should be yet another piss easy to implement thing too :-)

It has to be optional though such that you can force the loading of all data. If it is verified to be good we can make it the default action to speed things up and save cycles.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply