Serial protocol pre-design discussion - give us your input!

[Fred]

Hi,

This is something that needs to be discussed fairly early on as it won't be long and I'll need to watch parameters on a PC while developing.

I have a few ideas about how this should all work, but they are by no means a thorough well thought out set of ideas.

I'll try to present them here, and change and update them as your suggestions and critical discussions ensue.

• All "burn to flash" and "store in memory" commands and data should be fully checksummed such that no bad data ever gets in.
• An error should be sent back to the tuning software providing information if an corrupt packet is received or a framing error etc occurs.
• There should be some facility for a datalogging mask to be set such that small subsets can be datalogged with much higher frequency
• Data logged should have a weak checksum included in it for later processing (don't care at the time)
• Config for external hardware actuation should be supported by "chosen port" config sends

What are your thoughts? (yes, I know its incomplete, tell me what to add)

This is quite likely full of holes right now, be merciless, rip it apart :-) no offense will be taken!

Admin

[Fred]

Wow, the flood of comments is hurting my eyes ;-)

http://en.wikipedia.org/wiki/List_of_hash_functions

Someone suggested Adler-32, however the wikipedia states that it is not suited to short datasets which many of ours will be. If you have some knowledge in this area, please contribute your thoughts :-)

[SleepyKeys]

That looks like a really good plan of attack/feature set. I wish I knew more specifics to help.

[Fred]

BTW, having an action plan like that means we need a new tuning tool, or, we need to fix one of the two existing GPL ones.

I really think it's full of holes, but I don't know enough to say where yet. I'm working on it though.

[Fred]

I think a store/burn commands should have an ID of the data it is sending, a SIZE for what it thinks it's sending, and the firmware should be in control of where exactly this gets burned/stored. The address, expected size, format etc should be cross reffed with the received info and if there is a discrepancy, an error should be returned and nothing done. Only if transmitted size and endianess, etc etc match the stored version looked up from the ID should it be used. this in tandem with the checksums should make it impossible to burn/store bad data to the unit. It will introduce an overhead, but tuning stuff is typically quite small amounts of data anyway, so that should not pose a performance penalty.

Thoughts?

Admin.

[jbelanger]

If there is a message returned in case of an error, it could be a good idea to return an acknowledge when the data is accepted. Also, it would be a good idea to protect the header part of the message with it's own checksum (or another more robust method). This way you make sure you're not going to store data where you shouldn't (which could be much worse than bad data).

As for the overhead, I don't think the time taken for those checks is a big issue because it's not time critical data as it would be for logging information. The main issue is the needed RAM to store the data until it has been validated. Since it has to be validated, this can't be the same as the RAM used for live data. I think there is enough memory available but it's something to keep in mind.

I'm curious about how you plan on checking endianess of the data. Unless you check for a certain range I don't quite see how you'd do it.

Another question is would it be better to packetize the data so that we don't try to transfer 1k of data just to end up with corrupted data because of a single bit flip. This is more overhead but might be worth it.

Jean

[Fred]

The acknowledgment is a good idea for sure :-) (did I not have that up there already? not sure)

I may have been unclear, but I think you have misunderstood what I was trying to say.

In the tuning software a table that says :
ID = what, how big, etc
0 = ve1, 144 short, etc
1 = ve2, 144 short,
2 = advance1, 144 short,
3 = pwmBoost, 8 char,

and the same thing on the box.

You send the ID of the data, the description of what you think that ID means, and the data (and other stuff to be speced in this thread)

then the box compares the description as received with the one it has stored after looking it up from the ID.

The idea is to take memory access away from the outside world. Instead of saying put these 100 bytes at this address in memory, we'll say heres 100 bytes, it consists of 100 single byte big endian chunks, they are ve bytes. Then you can go "well, you've said ID 1 should be 100 bytes, but in fact, I think it should be 200 shorts, sod off." or similar. Does that make more sense?

Good point on the memory. We have 12k accessible through normal methods, but if we reduce that to 8k, then we have 24k of pageable 4k blocks to do as we please with using pointers etc. Perhaps that won't be required, but I could see it working, even if it is a bit "low n hacky".

Does that make more sense?

Admin.

[Fred]

What I'm trying to say is that I don't think internal memory layout should be exposed much or at all. The tuning software should communicate through a defined interface too.

Admin.

[jbelanger]

Makes sense. I guess I'm too much into a MS/MT mindset.

I was actually trying to think of a good reason for having direct memory access from outside but couldn't come up with anything. And it's easy to think of reasons not to do it.

As for single byte endianess, I don't think you need to worry about it at the firmware level. This will be taken care of at the lower hardware level (SCI, SPI, CAN) unless you bit-bang some serial data, which wouldn't make sense here. And for multi-byte data, each uP has it's own preferred endianess and the PC should just provide the data in this manner. If someone ports the code to a different processor than the current one, then it should be their responsibility to do what's needed either on the PC side or on the processor side.

As for the ID, do we want to have a version number to go with it so that if ve1 goes from 144 to 256 we keep the ID at 0 but change the version from 1 to 2 or do we change the ID from 0 to 73 (or whatever). I'm not sure that 256 IDs is enough for any future updates and using a version number plus ID might make it more manageable than simply using a 16-bit ID.

Jean

[Fred]

Makes sense. I guess I'm too much into a MS/MT mindset. :)

We'll shake it out of you yet :-)

As for single byte endianess, I don't think you need to worry about it at the firmware level. This will be taken care of at the lower hardware level (SCI, SPI, CAN) unless you bit-bang some serial data, which wouldn't make sense here. And for multi-byte data, each uP has it's own preferred endianess and the PC should just provide the data in this manner. If someone ports the code to a different processor than the current one, then it should be their responsibility to do what's needed either on the PC side or on the processor side.

That's true. I was mainly just throwing it in there as filler to bulk up the list of "description" that would get passed around. However, you are 100% correct.

As for the ID, do we want to have a version number to go with it so that if ve1 goes from 144 to 256 we keep the ID at 0 but change the version from 1 to 2 or do we change the ID from 0 to 73 (or whatever). I'm not sure that 256 IDs is enough for any future updates and using a version number plus ID might make it more manageable than simply using a 16-bit ID.

I'm not sure what you mean, the question is, will there be more than 256 tunable chunks? Not sure. Will we use an ID for each code revision, yes, 100%. A group of ID/desc/address sets should be applicable to one firmware revision for sure.

The way to do that seems to me to be ask the EMS what its revision code/number is, and then scan through a directory of possible configs and load the match. Maybe fail or ask you to choose if there are more than one?

There may be/probably are better ways to do the whole address-less access thing. I'm no expert, but like you I can see big reasons to NOT do that. Same for any future CAN implementations.

As I said, each tunable should probably have a machine address too.

In fact, if we do this right, we can have a single implementation for CAN and serial and XYZ and it should just work (tm).

Thanks for posting :-)