FreeEMS-Loader

[jbelanger]

Saving cycles is almost a non-issue. With a rating of 100000 cycles, you'd need to burn the flash every 5 minutes 24 hours a day for a year before reaching this number of cycles.

And while it would be slightly faster, since you need to transfer the content of the blocks over serial, which takes longer than reading from or writing to flash, you'd effectively only save the block erase time which is very minimal (a few seconds).

So even though it's not a bad thing to have, I think someone's time is better spent somewhere else.

Jean

[Fred]

Saving cycles is almost a non-issue. With a rating of 100000 cycles, you'd need to burn the flash every 5 minutes 24 hours a day for a year before reaching this number of cycles.

I must be getting damn close on mine then! ;-)

It would be faster than you think, I think. Reason being, you don't have to transfer the block either way, just a strong hash of it (40 bytes) and the address. This would definitely speed things up a bunch because it takes ages to reflash (15seconds currently up to 2 minutes later) and large chunks of the code are the same each time (at least during dev). I guess it depends on how fast the hash runs on the chip :-)

Anyway, you are right about that, his decoder should definitely come first :-)

Fred.

[jharvey]

Would be nice if we see it working. Right now, I've got 0 burns, so at this rate it will be quite some time before I hit the 10,000 limit. I'm also sure if you exceed the 10,000, we can find a way to get you another one. I don't see the burn limit as a real issue, it's more of a shed color.

[SleepyKeys]

Fred I dubbed your idea "Smart Burn" with a check box

[Fred]

I was just cleaning up, deleting files, etc etc and I came across one called "loader-post" dated 12 Jan 2009 and opened it up... It looked good, but I didn't remember writing it. So I started searching the forum and I couldn't find it. Then I found this thread, and sure enough, the post is not in here.... but this is :

I have a huge post half written that I'll put up as soon as I finish it. It's all about firmware interface versioning and meaning thereof.

Woops.

In any case, I'm not going to finish it, I'm just going to post it up as is and we can continue discussing it from now on. I'll bang it in a big quote below so you can't confuse the contents with something more recent than 1.5 years ago LOL

Wooo Hooo!. This is a fantastic idea.

Thought you might like this :-)

We absolutely need to have 3 different reserved areas that we write to when we flash something. 1) Code, 2) calibration (inc files) 3) Tuning data.

Actually, what we need are three different types of data with an arbitrary count of each type.
1) Code
2) Things that you want to keep between this revision and the last or something to that effect
3) Things that definitely need a reburn with the code

Perhaps it would be best to spec out four types. Lookups, main tables, code, config.

You are relying on several things when you do this. You need to be sure that the tables are in the same place. For example the IAT and CHT could swap and then your setup would be wrong to keep. I won't just do that for no reason, but it could happen I guess. I may also add further lookup tables once this app is done as there will be plenty of space available then.

Wouldn't it be awesome if we could flash a code and not have to deal with reflashing config and calibration?

This much is the easiest to do. We have a versioning scheme that takes care of this, just the addition of a "where is the code" piece of info and you are away laughing, implementation should be pretty easy.

One possible sticky part would be to force a wipe of the config data (or cal data if necessary, though this should be pretty constant) if there is a major change in the structure or offset of the config data.

Can this be done?

Let us consider the versioning scheme in a little more detail...

Code versioning :

Code: Select all

/* - Major Version                        Significant jumps in functionality                      */
/* - Minor Version                      Minor features or significant fixes                     */
/* - Revision Number                    Initial dev and minor fixes only                        */

Interface versioning :

Until now I hadn't considered a specific meaning for the interface version parts, just that there were enough of them. Thinking about this in more detail with a view to moving it to the serial spec change thread, what do we actually need to know about the interface and from whose point of view?

Firstly we assume we know what we are plugging in and that it uses the basic protocol packet format, so that doesn't need to be covered except with an english language statement.

• We need to know if the available payload ids/types and behaviours are the same or not
• We need to know if the way the code behaves the same way with respect to the settings that are available
• We need to know if the meaning, format, location and size of the fine grained config changes at all
• We need to know if the meaning, format, location and size of the main tables changes at all
• We need to know if the meaning, location and size of the lookups changes at all
• We need to know if the code size or location range changes at all

The code behaviour is covered by the code versioning scheme and thus not relevant here. Currently the config and tables are covered, but in a detailed way, there is no overview.
The payload id/types and behaviours are covered with an english language document, but also have no id. The code size, location ranges are not covered at all.




Which tells me that we should extend our 3 part number + an identifier to a 4 part number plus an identifier where the meaning is as follows :

• 1) Payload ID options and behaviours
• 2) Meaning, location and size of lookups
• 3) Meaning, format, location and size of main tables
• 4) Meaning, format, location and size of the configuration
• UID) Same string across all versions of a given fork or branch

With that level of detail we can know which things to burn and not burn with Sean's tool explicitly. The only extra required is another interface descriptor document in JSON or XML that maps the memory sectors to the four types such that if two firmware revisions use the same interface version or part there of it is straight forward to know that it is OK to leave bits of data in place and burn only others.

I can now see why I called it unfinished... but still it's good to get it out there. I'll keep thinking abou this and maybe post up more in the future. This is the sort of stuff that has to be right from the start...

Fred.

[EssEss]

assuming we have good parseable config structure .. whats to keep us from simply sucking out all contents before an update .. then the loader would re-arrange and re-place and/or re-link configs and maps etc, upon loading the new version. if it ever gets hosed, you still have this nice config file which you can use to hop around to other versions . . I also realize that this works more cleanly moving forward than backwards ..

maybe my idea is too simplified.

[Fred]

Possibly, but what would be best would just be to pull the s19 out as is, so it could be put back as is again, tune included - this is simple and easy to achieve. so first stage of loading is retrieve old data and stash it, just in case. i think if we define all the locations of stuff in the binary structure too its duplicate info. the serial interface does not expose memory representation to the tuner app etc, it has ids and descriptions/sepcs and this is the definition we'll be providing with the firmware for tuning apps to use, not memory structure.

[Fred]

Bump for new exciting activity! :-)

I was thinking about drawing an mind map for this. IE, layout the different components and data flows and logic etc and present it as a sort of rough design. But I should focus on my CV for a while... I'll keep up with the forum for sure, and do any testing you want, Sean! Just let me know :-)

Fred.

[Fred]

Just having a look at the source at commit # 3692e0448989ccc7e85c6e297ff59f1e921c8232

1) You're missing parity odd/even/none selection in your advanced tab
2) You *were* missing some .gitignore stuff and some directories (build errors), all fixed now, please pull/merge.
3) I updated the PNG to suit the scaling and fixed the borders on the xcf, please pull/merge.
4) I personally don't like underscores, or upper case binaries, but that's your decision. However, please make them consistent :-)
5) No time to actually get stuck into the code, I'll leave that for you. I hope the above helps a little! :-)

https://github.com/fredcooke/freeems-loader

Fred.

[Fred]

Just got off the phone after speaking to SeanK and said that I'd post up my game plan for the app here. So, my idea of how the app should work, critique accepted willingly, unlike the gitolite author, who is a grade A wanker, but that is another story...

• App starts up without arguments into gui mode by default and does nothing
• User selects file to burn, unzips it if necessary, verifies integrity of file, both internally, and against any matching md5 or sha1 hashes present in the same directory. - check box to ignore errors, do checks anyway, as they wont take long
• User tells the app to burn the file down
• App probes for the SM, and if found, initiates comms
• If SM not found, App pops up asking the user to set the SM switch to load, then offers three options: software reset old firmware, retry for SM after manual reset, cancel. If software reset pressed sends a reset packet to the current firmware and rechecks for SM, if SM still not found, same dialog reappears
• App now extracts the old firmware, and saves it to an appropriate dated file - check box to skip this
• App now compares old and new data and figures out which parts require burning - check box to skip this (previous checkbox auto checks this box, of course)
• App now burns down whatever was required from the previous steps (all if either check box was checked)
• App now verifies that load was successful by pulling the data back out and comparing it. - check box to disable this
• As a final touch, optionally, app can ask user to place the switch in run, and send an SM reset packet, then send the firmware version request packet and either display to the user or compare with the s19 file name. another check box...

It might be nice to offer the same "dump full image" or "dump only data that exists" options. It'd also be nice to have a checkbox for grabbing the SM or not, off by default. Another nice thing might be to extract the SM code into its own s19.

When doing the erase, it shall be done page by page, which is what affords us the opportunity to only burn the differing parts. Each block that is erased should be logged, and stats for how many succeeded (or not) should be presented after the erase portion is complete. Likewise the load and retrieve segments.

It would be nice to have a .freeems-loader-rc file that allowed the user to specifiy which checkboxes and options were and weren't switched on.

It should be possible to grab an s19, ignore all errors (there usually wont be any anyway), load it in completely (saves time reading the old one out first, as opposed to saving flash write cycles), and do nothing else by selecting appropriate boxes. This willl keep dev times down for you and I ;-)

It could be nice instead of having check boxes and a non stop load after hitting go, to have three options per option: no, yes, ask when ready - ask would pop up a box, or present a text mode question offering yes/no/abort.

All the same stuff could be done without the gui present with sufficient command line options, including "--nogui". The configuration could come from a file firstly, overridden by command line options, then finally if neither of those are present, in code defaults.

if i could setup the options in my rc file and then run it with --nogui device file and have it not ask me anything and just report what happened, that would be cool. Thinking about that, an option to not spew out granular real time load data messages and just have a progress bar like hcs12mem does and summary stats afterwards, would be good.

I hope that aligns with your ideas! :-)

Fred.