Serial protocol pre-design discussion - give us your input!

[Fred]

Firstly, I want to point out (again) that this protocol should NOT be symmetrical. There is likely to be no room for state to be stored and no room for state handling code to be stored either, nor time for it to run.

It should be the case that if exterior device A asks core device (FreeEMS) for XYZ, FreeEMS should handle that request and only that request till it is done. If FreeEMS never gets the request it will never send it. FreeEMS will never want to send something, it will always have to be asked for, or configured to be sent. The only case where autonomous sending should occur is logging data. If A needs something it should ask for each thing in sequence as time allows. In the case of datalogging, it should be either packet by packet in the same manner (possibly slow) and/or a continuous stream of data as fast as can be sent out.

I vote a combination between 1 and 3 for reliable transmission. 3 for the mechanics behind it, but when FreeEMS recieves a request to do something, it should disable comms with flow control (or just by ignoring the world for a while) before doing it and re-enable afterwards.

String + version sounds good to me. For simplicity sake, it should be one char array with the first 3 chars used as the version number and the remainder as the descriptor, this keeps it simple and compact (and allows for a fully alphanumeric string version later with no change or longer numeric part etc.)

"FreeEMS Vanilla" is 15 chars long, version is 3 chars = 18+ , how about 32 total for that field? so if we throw a string representation of the v number in we would get a max of :

255,255,255,FreeEMS Vanilla v255.255.255X

And currently :

000,000,012,FreeEMS Vanilla v0.0.12XXXXXX

Where X represents free space.

I've just implemented that.

I just checked those wiki pages ... All three as they were :-p (not that I mind, I didn't see a need for them)

Cheers,

Admin.

[jbelanger]

Firstly, I want to point out (again) that this protocol should NOT be symmetrical. There is likely to be no room for state to be stored and no room for state handling code to be stored either, nor time for it to run.

I totally agree. The goal should be to have the maximum throughput from the ECU and the minimum overhead in bandwidth AND processing (on the ECU side). The lower layer should take care of the reliable transmission as much as possible. So for RS-232, the use of hardware flow control should be used. As for CAN, as far as I know, the reliability is taken care of by the CAN hardware itself.

Also, the flow control doesn't have to be symmetrical either: if the buffer is big enough on the PC side, then the ECU can send whenever it's ready and no data should be lost. In any case, if logging data is lost (which is what should consume most of the bandwidth in that direction), nothing catastrophic will happen.

It should be the case that if exterior device A asks core device (FreeEMS) for XYZ, FreeEMS should handle that request and only that request till it is done. If FreeEMS never gets the request it will never send it. FreeEMS will never want to send something, it will always have to be asked for, or configured to be sent. The only case where autonomous sending should occur is logging data. If A needs something it should ask for each thing in sequence as time allows. In the case of datalogging, it should be either packet by packet in the same manner (possibly slow) and/or a continuous stream of data as fast as can be sent out.

Agreed except for the continuous stream of logging data. I understand the logic behind this but I'm not sure that it will actually result in better throughput because it may require more overhead to make sure that everything stays in sync and can recover from some error. With constant polling, at worse, you lose one packet of log data and lose some time due to time out but then you should recover.

I vote a combination between 1 and 3 for reliable transmission. 3 for the mechanics behind it, but when FreeEMS recieves a request to do something, it should disable comms with flow control (or just by ignoring the world for a while) before doing it and re-enable afterwards.

String + version sounds good to me. For simplicity sake, it should be one char array with the first 3 chars used as the version number and the remainder as the descriptor, this keeps it simple and compact (and allows for a fully alphanumeric string version later with no change or longer numeric part etc.)

"FreeEMS Vanilla" is 15 chars long, version is 3 chars = 18+ , how about 32 total for that field? so if we throw a string representation of the v number in we would get a max of :

255,255,255,FreeEMS Vanilla v255.255.255X

And currently :

000,000,012,FreeEMS Vanilla v0.0.12XXXXXX

Where X represents free space.

I've just implemented that.

I just checked those wiki pages ... All three as they were :-p (not that I mind, I didn't see a need for them)

Cheers,

Admin.

I think the 2 fields should be separate: one 3-byte field for the version and one 32-byte (or whatever) field for the version label. This way, if you only need to check the version number you get the 3-byte field (what any program that communicates with the ECU will require) and if someone needs to see the label then you get the 32-byte field (what someone will want to see from a terminal connection or displayed in a tuning program).

Also to come back on some of the previous discussions, this should be the only thing sent in a human-readable format: anything else should be sent in the most bandwidth-efficient manner (which will also be the least computing-intensive manner). And using anything other than a custom protocol would be a mistake in my mind: first of all, there is no need to be compatible with other hardware/software (unless I'm mistaken) and, second, most "official" protocols are compromises between different parties of the sanctioning body and are usually not the preferred solution for anyone. At least, that's the experience I have had in a completely different situation (ATM Forum and ITU). However, using ideas and concepts from existing protocols and adapting them is certainly a good idea.

By the way, I'm glad the discussion is going on here rather than in a wiki. Although I haven't contributed anything up to now on this subject, I hate wikis for anything but final documentation (and even then I don't think that's the best solution). But that's just my opinion.

Jean

[Fred]

Agreed except for the continuous stream of logging data.

What I meant was to have a configurable mode in the EMS where either it always streams logging constantly as fast as possible, or it doesn't. If it doesn't you have to poll it for data. In my mind this allows the EMS to work faster and more accurately when continuous logging is switched off. The obvious draw back is that certain problems may only manifest themselves in one mode or the other...

I agree with the strings = only human things. I think having two strings could be an idea though. Someone suggested that some time ago on another message board. As a note to remind ones self of what they last did/the state it was left in. I'm not too worried about version + name vs. version and name. They can exist in separate vars inside, but I think what Stu meant was that I might have vanilla called 2.3.1 and Joe Blogs might re write the whole thing in ASM but want to use Stu's tool to interface and call his firmware JoeBlogsEMS 2.3.1 with totally different interfaces etc (except the versioning scheme). If it is acceptable to base comms on just the number then branches and forks etc will have to use random high numbers like 100.3.1 or similar to avoid overlap with my original ones. If you don't do that then you would have to request both numbers and name separately and check both. What do you think about that Jean?

I agree with all of your "wiki" and "other protocol" comments too :-)

Admin.

[ababkin]

I agree with all of your "wiki" and "other protocol" comments too

Admin.

ok, so there are two of you who don't think that information conciseness, structure and comprehensiveness for people not following the discussion is important - ok, you win, let's do it the spaghetti-way

And, as i've mentioned, i've dug up the "other protocol" specs and docs to fish for proven ideas as opposed to reinventing a bike. I am well aware that the platform doesn't have spare resources to implement compromised generic protocols.

<constructive part>
Let me propose the following for datalogging output stream of data:
- PC gives some composite command to EMS, telling it to start streaming in particular format (for example out of five types of data a,b,c,d,e, it only wants 3: a,c,d)
- EMS starts to stream immediately.
EMS streams the data as 'un-intrusively' as possible: It is important (or at least a good practice) to ensure the main-loop period to be as consistent as possible (to maintain the real-time-ness of the system). I imagine that due to various hw interrupts, dynamic branching, etc, main-loop period can deviate drastically from one iteration of the loop to another. We can use the latency introduced by the 'datalogging update cycle' to even out the main-loop period to an extent.
I understand that the events that will make this period 'un-even' will all have their own frequency of sorts, and once combined, this cumulative frequency may have some undesirable peaks or troughs. So the idea is to employ the inevitable delays caused by sending the datalog data to level out this cumulative frequency. (and do it using some low-priority polling scheme as opposed to using uart interrupts that may actually contribute to main-loop period deviations)

simplest and very primitive algo that gets the idea across:

int old_time;
while(1){
old_time = current_time();
...<important code>

if (current_time() - old_time >= too_long){
continue;
}
send_datalog()

}//while

// constant 'too_long' will need to be tuned manually or algorithmically of course
// current_time is based on hw timer

Alex

[Fred]

You are free to use those wiki pages, just type bump and hit post when you make an edit :-)

We can use the latency introduced by the 'datalogging update cycle' to even out the main-loop period to an extent.

Good idea! :-)

As for the stream and command thing. I would prefer that the list of a,c,e was stored in flash/ram like all other tunable values such that if saved it became the default for next time. Ditto the send/not send as you want the ability to have it automatically start flooding out when you next hit the key if you have a dumb "record everything" datalogger.

Thanks for the constructive criticism :-)

[FlappySocks]

I think what Stu meant was that I might have vanilla called 2.3.1 and Joe Blogs might re write the whole thing in ASM but want to use Stu's tool to interface and call his firmware JoeBlogsEMS 2.3.1 with totally different interfaces etc (except the versioning scheme).

Yes, thats the idea. A bit like MS vanilla and MS Extra. I dont see any problem in using a String here. Its not like your going to request it that often. It also has the advantage of allowing the "Megatune" type software to present that information to the user, if it decides it doesn't know what it is. "Joe Blogs Firmware not recognized - [Retry] [Cancel]"

I was going to put my two docs on the wiki, but I have been busy building MS stuff
As details get worked out here, they can be put straight into the wiki.

Dekiwiki is my favorite. Yet to try Sourceforge

[Fred]

Yes, thats the idea. A bit like MS vanilla and MS Extra.

That is exactly what they do. They have a string 20 chars long that identifies the firmware. I was just thinking if we have 3 chars used as numbers we will effectively get higher data density in there. 20 chars is enough for me at the moment, but thinking about it right now, what does it matter what the length is. The length of the string returned is yet another unique identifier to use. Just request the version/name and get a string of 4+ chars long, the first 3 being the version number, and the last 1+ being the name. Why restrict something that doesn't need to be restricted! This is the sort of thing discussions provoke. I wouldn't have thought of leaving it open in terms of length without the Stu/Jean dialog :-)

[FlappySocks]

Just to add another idea to the pot. How about just using a string for the version, without encoding any numbers in there. The version numbers could become part of the string. "FreeEMS vanilla 1.2.3a 2008-05-21". Then the firmware writer can use any version numbering scheme he chooses.

[Fred]

The only thing that bothers me about that is the length of it I guess. However, if I am free to keep mine short I guess it doesn't matter. "FreeEMS Vanilla v255.255.255" is 28 chars and "FreeEMS Vanilla v0.0.13" is 23 chars. "nnnFreeEMS Vanilla" is 18. This would be a text book case of optimising too early though :-)

How about a rule for encoding in general where the string is knocked down into displayable chars and non displayable and the non displayable are considered so because they are prefixed with a back slash. In those cases the numeric value should be shown.

Seems a bit kludgey however something is needed such that the full range of chars can be used rather than just 0-9 + a-z + A-Z + common symbols. It is only needed for display though as the pattern matching side of it will work perfectly regardless of what you throw in.

Ideas?

[FlappySocks]

Just keep it simple. A string of any length, but put a recommended cap on it of say 40 characters. From the firmware writers perspective, all thay have to do is have a constant string variable in the main part of their code, that gets sent A) when powered up. B) when requested.

In your "Megatune" tune app, you have something like this.

Code: Select all

if (stringVersionIn.StartsWith("FreeEMS")
{
   if (stringVersionIn>"FreeEMS 1.2.3")
   {
     showMesaage("Warning, firmware more recent than this application... update me");
   }
   elseif ((stringVersionIn=="FreeEMS 1.1.1a")
   {
     showMesaage("Warning, this firmware has serious bugs - please update it");
   }
   elseif ((stringVersionIn>"FreeEMS 1.0.0")
   {
      //No problem, run app
   }
   else
   {
     showMesaage("Error: unknown firmware "+stringVersionIn);
     exit;
   }
else if (stringVersionIn.StartsWith("Bloggs MegaEMS")
{
.....
}
}