Serial protocol pre-design discussion - give us your input!

[FlappySocks]

Would it not be better if the header data was part of the checksum? After all, it contains important information too which has just as bad a consequences if corrupted.

Can do. Originally with the CRC being in the header itself, it would have been awkward.

You should spec that the payload length doesn't include the optional checksum too.

Will do.

The idea behind the acknowledgment number is that there may be multiple threads going on, each sending data. If two processes send data with an acknowledgment request simultaneously, you then know which acknowledgment is which.

Also, if your sending a bulk load of data, it could be possible to acknowledge multiple packets, with just one acknowlegment back. For example, one message that says, acknowledge message 10 to 55, or 200 to 16.

[Fred]

Hmm, I can't see a circumstance where the amount of data would be significantly sized.

Considering the foundation assumption of our discussion : only realtime/data logging needs to be FAST, would it not be prudent to KISS with the protocol and acknowledge each packet separately? This seems simpler to code and understand on both ends as it's a 1:1 relationship then rather than 1:1 OR 1:many.

I envisage sending the logging data in one packet as per the mask which both the EMS and PC/gauge/etc know about. That has the potential to be one of the biggest packets we will need. The only other one that could be sizable is sending a non-table data structure.

For sending back say ALL settings, I think it should be on a one by one basis with the complexity (to iterate through them all) left on the PC side and the EMS code left most simple with minimal functionality. ie. it can return/modify/save any given setting individually as would be needed for tuning, but it doesn't know how to return the whole lot at once. (perhaps it would be easy enough to implement anyway, but if there is any doubt...)

[FlappySocks]

I envisage sending the logging data in one packet as per the mask which both the EMS and PC/gauge/etc know about.

Agree.

That has the potential to be one of the biggest packets we will need. The only other one that could be sizable is sending a non-table data structure.

What about firmware updates? Or will you have to use a jumper each time?

[Fred]

Apparently I wrote too much at once :

I hope firmware doesn't need to be updated through this protocol as the boot loader is there for that, however it is a possibility if the boot loader doesn't work out the best for multi paged data etc. If that is the case and the fault lies with hcs12mem, it would be better to fix hcs12mem. If it lies in the bootloader we would have to live with it and code around it on the cpu in a similar style to kens c64 to c128 bootstrap loader does.

There is no jumper on the board, there is a switch, and the bootloader pin comes out to the main headers so we can include an external case switch in the design easily.

I did this on my MS. I think it's more fool proof than spending effort writing code to do something that is already done very well with the possibility of having our own bugs etc etc.

My ms front panel :

http://i50.photobucket.com/albums/f323/ ... 0_4016.jpg

It's a locking switch once used in a nuclear physics lab as part of a particle accelerator setup. It makes it very easy. To load through serial like that you have to copy your code up to memory and then run and then hope you don't get any errors/power cuts etc while doing it or you are out of luck and have to use the switch anyway.

Duplication = bad. The main reason MS2E went to that method was that the bootloader only supported 64k and by doing it manually they could support 128k. Ours already works and supports the whole chip AFAIK.

Admin.

[dandruczyk]

Apologies for not following every bit of this thread.

I don't like the funnky sized integers stuff. It adds a lot of weird overhead to both the ECU and the tuning software (and it's just plain tricky to wrap one's head around, i.e. why is it really necessary?, what does it gain you? a byte or two fo space?). Other bits could be adapted from the TCP protocol. i.e. when sending send append a byte with a "code" or "ID" to it, the receipt FOR THAT COMMAND should have the same code, it allows out of order processing (whcih may happen for CAN transactions where a remote device is slow in responding). This also allows the concept of multiple outstanding simultaneous comands and well as possibly reducing overhead, by not waiting for "acks" for each command sent. In this proposed idea, the response would have the same code as the sender (as well as the data) and would server as the command acknowledgement.

Things that would need to be done is to CLEARLY define the header section. I prefer using indivuidual bits to mean specific things which is more space efficient, but that's just an idea. The escaping also is a little kludgy, but I can't think of any other way to get around it without using something liek a "long sequence" (i.e. 8 byte start header in a very unique sequence which is statistically unlikely to be hit in the datastream), it just makes it harder for the ecu (added overhead to scan each byte before sending to assemble the escape sequences, as well as the tuning software side doing the same thing in pseudo reverse to decode the original message...

[FlappySocks]

The funnky sized integers thing is something I borrowed from a protocol I use for vehicle tracking kit. I have found it to be very useful, in two particular areas. 1) it saves you from having to commit yourself to a theoretical maximum value now, which may kick you in the teeth later. 2) If the two devices at either ends of the protocol are independently developed and/or there is the possibility of either one device is using an older version of the specification, then this method reduces the chance of things breaking.

For example, if I were to develop a simple digital display thing, that gets mounted into the dashboard, then I dont want to have to worry about customers sending it back because someone decided they needed an extra byte in the protocol spec.
Likewise, the developer of the EMS is not going to want to break the protocol without having to make some tough decisions first.

I'm not saying we should keep the idea... just worth looking at it from all angles. And it's not all that hard to implement... just masks, and bit-shifting (i can provide code). Once it's coded into a subroutine, you wont even have to think about how it's done.

[karsten]

I've had a look through the documents; what you describe is very similar to PPP with HDLC framing and your "strechy integers" are generally called "BER compressed integers".

I think instead of inventing a somewhat arbitrary protocol it might be a good idea to actually just use PPP. A basic implementation should be fairly straightforward and does not need to support any optional PPP options initially (even though protocol and address compression would be simple and useful). Have a look at http://en.wikipedia.org/wiki/Point-to-Point_Protocol for a summary and RFC 1661 and RFC 1662 for the details.

PPP also supports reliable transmission via LAPB (RFC 1663). Handling reliable transmission at this low level (layer 2) would remove this burden from higher levels so may be worthwhile.

[Fred]

Velcome to Ze mad German :-)

Thanks for posting Mr K, I'll give it a good read.

It may be that it is somewhat overkill in some areas and/or underkill in others, but I have no idea till I read it.

Good to know you GF let you near the PC :-)

[Fred]

BUMP

I have to say people, the turnout of comments, questions and discussion here has been fairly poor. We are all here too learn right? So, if you already know about some aspect, or have some opinion, even if it is from a user point of view, share it! If you don't understand something, ask questions about it (pm will do if you are shy, but seriously, we are all friends here...) We need to try to get this nailed down ASAP so that work can begin on it's implementation.

I've had a bit of a read on ptp, and so far it seems like massive over kill, but I didn't dig enough to know that for sure.

Minimal and effective.

Data logging needs to be lean and fast and configurable.
Burn/Store commands need to be robust and reliable with acknowledgment of transactions.
Burn/Store/etc do not need to be fast, BUT, the protocol does need to be efficient both size and code wise for lots of reasons, but number one, the understanding of those who will use it (including me).

If you so much as think "hmm, i'm not sure i agree with that" post it!

Please.

Admin.

[thebigmacd]

I work with BACnet on a daily basis and one neat idea we may be able to glean is the idea of "subscription". This is where for every value available, we can send a command to include it in the datalog or not. The source device then sends out a list of all the subscribed values at a predetermined interval.

I'm thinking the PC could sent out a command like:

Code: Select all

!s00t0

Where:
"!" is the start of a packet (could be replaced with a standard ANSI byte)
"s" is the subscribe command
"0" means ECU (1 means transmission controller etc - this would have to be defined)
"0" means bank 0 (default. on a multi-bank engine bank 0 would always be an average of all the banks)
"t" means the value is a temperature
"0" means coolant temp when preceded by "t" (1 would mean IAT, etc - again would need to be defined)

The complement would be !u00t0 (unsubscribe)

The ecu response would be the following as part of the datastream:

Code: Select all

!d00t0 sw 00FF

Where
"!" means start of a packet
"d" means this is a data packet
"0" means ECU
"0" means bank 0 (average of same value on all banks, or single value in a single-bank system)
"t" means temp
"0" means temp0, or IAT
"sw" means signed word
"00FF" means 255

Just throwing ideas out there. I got the "bank" idea from OBDII. This could be handy for V8s without a crossover pipe. Even if banks arent supported for a long time, we could just always use bank0 for everything until multiple banks are supported.