The Ultimate Loader

Reference implementations of various tools for use with FreeEMS. Written in Java for my personal use, they're 100% cross platform, 100% compatible and 100% reliable with thorough test coverage.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

The Ultimate Loader

Post by Fred »

One small boy's wet dream about his idea of the ultimate firmware loading utility application. This is all stuff that I've gradually realised while using other's loaders over the years, starting with HCS12MEM then Sean's Loader and MTXLoader, thinking about the way the SM works, and how software should be built, etc.

I will be gradually implementing this in private, however I'm sharing this design specification such that others can excel in their applications with overlapping functionality in future. In particular, I'm not planning to make my version a CLI app, though it wouldn't be difficult to do if required, so I expect this calibre of work in a CLI capable app at some point.

General Principals

So far, only one, convention over configuration! IE, defaults in code, app only saves what you change such that config is small and tidy, config saved or chosen at runtime (and immediately written out to the config file) overrides code config.



Overall Module Or Application

Global Display Items
  • Connect/Disconnect button - state change based on connection
  • Reset on disconnect check box - enabled by default
  • Reset & Reconnect button - greyed when not connected
  • Abort Operation button - greyed when nothing is in progress
  • Show/Hide Console Window button - always active, changes state when pushed or the window is closed.
  • Device information - green LED when connected and detailed information, red LED when disconnected and blank info
  • User goals tab set
  • Progress bar for overall progress per goal - part of interface to goals
  • Each user goal tab has two tabs - config and detailed progress
  • Button at the bottom of the configuration "Start <Goal>" - greyed when all prerequisites aren't met
  • Starting a goal switches to the detailed progress tab and locks out other actions until done
  • Detailed progress is greyed and unreachable before starting action
  • Detailed progress displays stages with progress bar for each stage
General Options
  • Connection device
  • Console logging on/off
  • File logging overwrite/rotate
Advanced Options
  • Use broken erase-all function workaround - on by default, two options, decide which for default later
  • Serial data format - 8N1 by default
  • Device page and address ranges - FreeEMS values by default
  • Number of low level tries to connect before fail


General Behaviours

File Selection & Loading Semantics
  • Text field with file name
  • Text field with file path
  • Choose file button
  • Select or type in file name/path
  • Verify button - invokes actions:
  • For input files check ability to read, check checksum, check contents
  • For output files check ability to write, create empty file
  • Text area with file stats: results from verify button actions
Retry Behaviour Semantics
  • Connect/Disconnect - connect uses config, disconnect just closes device, if resetting it sends bytes, pauses, then closes device, and shouldn't care about the response, only that it waited long enough.
  • Erase device - to be thought through
  • Rip device to file - initial tries, then pop up asking to try more or fail with some stats on how far it got.
  • Load file to device - similar to rip semantics, independent config, uses rip and erase semantics where enabled
  • Verify against file - uses rip semantics + fail on IO errors
  • Compare two s19 files: just output messages or pop up for IO errors on chosen files and fail.


Foundation Functionality

Lower SM API layer provides raw calls that these use to do their individual tasks. Implementation below that API can change around without affecting the calls and results. Lower level implementation depends upon an abstract serial device implementation that can change in any way without affecting the calls and results.

SM Connection
  • Requires serial device selection, similar display to file choice
  • Requires attached SM capable device
  • Requires user to activate
  • Sends opening sequence and listens for either "fresh from reset" or "was already running", fails if one not found, succeeds otherwise.
Reset Device
  • Requires "SM Connection"
Erase Device
  • Requires "SM Connection"
  • Erase only specific ranges - off by default - can be activated by callers, including user
  • Standard mass erase
  • Custom mass erase
  • Page by page
Rip Device
  • Requires "SM Connection"
  • Rip only specific ranges - off by default - can be activated by callers, including user
Load Data
  • Requires "SM Connection"
  • Load only specific ranges - off by default - can be activated by callers, including user
Selecting A File
  • Need to choose file - remember last dir and last file, possibly remember history
  • Check file MD5 sum OR sha1 sum, file structure including line sums md5/sha1 are optional and automatic if matching file present.
  • Display information about the parsed file, lead string, record type stats, total binary data to load, address ranges, checksum stats/statuses


User Goals

These have a GUI tab each, options and sequencing, the gui contains config each goal, plus the config inherited from each option with dependencies, on a selected basis.
  • Erase device
  • Rip device to file
  • Load file to device
  • Verify against file
  • Compare two s19 files
Load Image To Device

Sequence: Configure >> Start Button >> Optional Rip To File >> Optional Erase >> Load Data >> Optional Full Post Verify >> Finish

(Note, configuring involves loading and verifying the data files in advance, this should be near instant)
  • Requires "Selecting a file" - to load from
  • Requires "Load data"
  • Requires "Selecting a file" - to rip to, if rip first enabled make this transitive!
  • Requires "Erase device" first - on by default - rarely turned off
  • Requires "Rip device" first - on by default - turned off by devs
  • Load only specific ranges - off by default - slow to configure or requires some sort of config drive
  • Load only different parts - off by default - slow due to pre-rip
  • Transitive: Requires "SM Connection"
  • Records memory extents of last load for next time such that quick rip of last loaded file can be done.
Verify Against File

Sequence: Configure >> Start Button >> Rip Relevant Data >> Optionally Save To File >> Compare Relevant Data >> Finish
  • Requires "Selecting a file" - to compare to
  • Requires "Rip device"
  • Check only specific ranges - off by default - slow to configure or requires some sort of config drive
  • Full rip first - off by default - slow and not required.
  • Transitive: Requires "SM Connection"
Rip Device To File

Sequence: Configure >> Start Button >> Rip Relevant Data >> Save To File >> Finish
  • Requires "Rip device"
  • Requires "Selecting a file" - to rip to, if rip first enabled make this transitive!
  • Rip only specific ranges - off by default - slow to configure or requires some sort of config drive
  • Transitive: Requires "SM Connection"
  • Option to rip only what was last loaded
Erase Device

Sequence: Configure >> Start Button >> Optionally Rip Relevant Data To File >> Erase Relevant Data >> Finish
  • Requires "Erase device"
  • Requires "SM Connection"
  • Requires "Rip device" first - on by default - turned off by devs
  • Erase only specific ranges - off by default - can be activated by callers, including user
Compare Two S19 Files

Sequence: Configure >> Start Button >> Compare Data >> Optionally Write Report To File >> Finish

(Note, configuring involves loading and verifying the data files in advance, this should be near instant)
  • Requires "Selecting a file" - file one
  • Requires "Selecting a file" - file two
  • Match patterns and sections and align lines based on start/end points such that visual diff apps work and git repos store well
  • Leverage S19 tools package code?
  • Launch external diff application?
  • Provide this functionality LAST

If you can't make a fucking awesome loader from this description, something is wrong. Possibly with the description :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

The above code will not know how to talk to a serial device, that will be provided to it from the application container. The above design will reside in the blue and purple boxes in the following diagram, thereby making it independent of serial driver implementations on the underlying architecture.

Image
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

There is some missing information, and some inconsistency in the interactions between the goals and the fundamental providers, but it's largely complete. Enjoy! :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

The above diagram was wrong, here is a new but slightly simplified diagram that is correct:

Image

What's missing? The InputStream has a buffer inside it too. That's about all.

Code: http://pastebin.com/QX8DB9u7

You could easily implement this in basic C as well.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

BTW, when I drew the old diagram, I already had working code in the style of the new diagram talking to FreeEMS, interrogating it and streaming packets from it. I was just so wound up by someone saying it was a bad approach that I forgot! I got it close in the heat of the moment, though. Moment being an hour or three while I wrote the above.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

The above call back mechanism is ideal for this application, however for constant bulk data from the firmware, it's too slow with largish overheads. Still usable, but not desirable. I got 14% of one core burn with a normal datalog stream from the firmware. With a sleep, read mechanism I get 0.2% of a core burn under the same conditions. Because the SM is quiet, and loading is a short term process where per-byte timing is important, the call back method seems ideal.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

Just for others reference:

Time taken to get a response back in your hands = the sum of these:
  • send latency (including inter-byte stuff, if there is any)
  • sent byte count * time per byte
  • processing time required on device (negligible except for erases)
  • receive byte count * time per byte
  • receive latency (buffering and driver lag and os lag and so on in the app)
The first and last can be clumped together and given a setting in the UI with a reasonable default for the style of comms in use. Processing time needs to be determined empirically with a logic analyser. Byte counts can be calculated trivially at runtime.

So you end up with this:

Code: Select all

time = latency + calculated + measured.
This is applicable whether you're waiting for data or blocking on it. Waiting = go lower as higher impacts performance (depending on design). Blocking = use this as a minimum to block for a given command.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

Goal for simple initial CLI-only version since no functional CLI version exists right now:

Functions
  • tufl load file device (loads verbatim, warns and then fails during load of sm content) DONE!
  • tufl loadfast file device (no verify = ~2x faster) DONE!
  • tufl loadnosm file device (excludes SM code from file)
  • tufl verify file device (checks what you give it, not anything else, ie, works for sm, or fw)
  • tufl rip targetfile device (default ignores SM region)
  • tufl ripwithsm targetfile device (rips everything) DONE!
  • tufl ripsm targetfile device (rips only SM)
Advanced

Later I'll add these:
  • tufl verifysm device (checks for hash of data from sm against internal DB of known SMs)
  • tufl eraseall (clear device out) DONE!
  • tufl loadnoerase file device (same as load, assumes erased)
  • tufl copy srcDev destDev (reads data from one device and writes it to another)
  • tufl diff command file file (prepares two files for diffing with an external tool)
User Feedback

All , except diff, should output timings and stats, per section, and totals.

S19 File I/O

Any discrepancies in any S19 file, at all = fail fast. (no interactivity yet)

Comms Issues
  • error during erase = N-1 retries, fail fast, dump stats/info.
  • error during write = N-1 retries, fail fast, dump stats/info.
  • error during read = N*10 retries, fail slow, dump stats/info.
  • mismatch during verify = N rechecks, fail fast, dump stats/info.
Default N = 3
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

All of the plumbing is in place!

Code: Select all

Total good lines: 3613
Found and ignored non-S2 record! Type: S0
Found and ignored non-S2 record! Type: S8
Total contiguous blocks: 15
PPAGE: 0xE0 Start address: 0x8800 Size: 484 Writes: 2 of which 1 are full size.
PPAGE: 0xE1 Start address: 0x8000 Size: 144 Writes: 1 of which 0 are full size.
PPAGE: 0xF8 Start address: 0x8000 Size: 258 Writes: 2 of which 1 are full size.
PPAGE: 0xF9 Start address: 0x8000 Size: 767 Writes: 3 of which 2 are full size.
PPAGE: 0xF9 Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
PPAGE: 0xFA Start address: 0x8000 Size: 275 Writes: 2 of which 1 are full size.
PPAGE: 0xFA Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
PPAGE: 0xFB Start address: 0x8000 Size: 303 Writes: 2 of which 1 are full size.
PPAGE: 0xFB Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
PPAGE: 0xFC Start address: 0x8000 Size: 275 Writes: 2 of which 1 are full size.
PPAGE: 0xFC Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
PPAGE: 0xFD Start address: 0x8000 Size: 4479 Writes: 18 of which 17 are full size.
PPAGE: 0xFD Start address: 0xB800 Size: 2048 Writes: 8 of which 8 are full size.
PPAGE: 0xFE Start address: 0x8000 Size: 5302 Writes: 21 of which 20 are full size.
PPAGE: 0xFF Start address: 0x8000 Size: 10346 Writes: 41 of which 40 are full size.
Total writes: 230
Full blocks: 220
Total bytes: 57449
Tomorrow morning I'll connect the dots.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Ultimate Loader

Post by Fred »

Dots connected! Couldn't sleep, so got a bit more done: This includes verify and is twice as fast as Sean's loader. :-/

Code: Select all

Total good lines: 3622
Found and ignored non-S2 record! Type: S0
Found and ignored non-S2 record! Type: S8
Total contiguous blocks: 15
Opening serial device: /dev/ttyUSB0
Opened serial monitor comms!
Successful dinfo: 6 {0xDC, 0xC4, 0x10, 0xE0, 0x00, 0x3E, }
PPAGE: 0xE0 Start address: 0x8800 Size: 484 Writes: 2 of which 1 are full size.
.. DONE!
PPAGE: 0xE1 Start address: 0x8000 Size: 144 Writes: 1 of which 0 are full size.
. DONE!
PPAGE: 0xF8 Start address: 0x8000 Size: 258 Writes: 2 of which 1 are full size.
.. DONE!
PPAGE: 0xF9 Start address: 0x8000 Size: 767 Writes: 3 of which 2 are full size.
... DONE!
PPAGE: 0xF9 Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
................................ DONE!
PPAGE: 0xFA Start address: 0x8000 Size: 275 Writes: 2 of which 1 are full size.
.. DONE!
PPAGE: 0xFA Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
................................ DONE!
PPAGE: 0xFB Start address: 0x8000 Size: 303 Writes: 2 of which 1 are full size.
.. DONE!
PPAGE: 0xFB Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
................................ DONE!
PPAGE: 0xFC Start address: 0x8000 Size: 275 Writes: 2 of which 1 are full size.
.. DONE!
PPAGE: 0xFC Start address: 0xA000 Size: 8192 Writes: 32 of which 32 are full size.
................................ DONE!
PPAGE: 0xFD Start address: 0x8000 Size: 4479 Writes: 18 of which 17 are full size.
.................. DONE!
PPAGE: 0xFD Start address: 0xB800 Size: 2048 Writes: 8 of which 8 are full size.
........ DONE!
PPAGE: 0xFE Start address: 0x8000 Size: 5441 Writes: 22 of which 21 are full size.
...................... DONE!
PPAGE: 0xFF Start address: 0x8000 Size: 10344 Writes: 41 of which 40 are full size.
......................................... DONE!
Total writes: 231
Full blocks: 221
Total bytes: 57586

real	0m22.485s
user	0m3.108s
sys	0m0.540s
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Locked