Serial protocol pre-design discussion - give us your input!

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
nitrousnrg
LQFP144 - On Top Of The Game
Posts: 468
Joined: Tue Jun 24, 2008 5:31 pm

Re: Serial protocol pre-design discussion - give us your input!

Post by nitrousnrg »

Thnks for the welcome!
shameem wrote:Do you have a link to the source code? I am looking to develop a visualization tool for the serial interface - i am kinda bogged down by something right now - but i should be able to get on it soon. Your gauges and graphs look interesting - if all that code is GPL then i may borrow some - just for gauges...

Well, I don't have any link (i should, but time is not my friend). It's all GPL, but this gauges and graphs uses Qt4 libraries, adding all that library just because a couple of gauges is a waste. Oh, I also must recommend qextserialport. It reaches my usb-serial adapter perfectly (working on a debian laptop. Worked with kubuntu too), and is multiplatform.
Really, i wrote 80% of that code in a couple of months. It should be multithreading, but i thought it was hard to do (it's not! maybe in some free time i'll do it). The code is a little messy.

Lets keep talking about serial comm. This is getting off topic ;)

The length of the string sent to the PC is not relevant. Unless you are transmitting 90+ channels, the communication speed will be fine. The procol can be as tricky as you want (but not trickier). For example, i was sending a VE table stored in the PC's RAM, to the ECU, and the ECU use it to do all the math.
This is the nice thing:
For on-the-fly table changes, the ECU used a sort of micro_table, sent by the PC on demand. So the data table is stored and changed in the PC, and it send all variables needed by ecu in real time. THE CODE FOR THIS WAS A MESS. It took me many months of trial and error. But, incredibly it worked. I was sending lots of data @ 119200baud, filling tables on real time (i think 4 milliseconds). So don't worry about string lenght :D
I did this because i have only 400 bytes of ram on the ECU. A few weeks later i changed the whole thing.

For developing, i think that a tool like minicom or cutecom (i love it) isn't enough. It surely is good enough for the first attempts but, in my case, i found my own software for debugging a priceless tool. Its just a thought, take it or leave it.

btw, i'm 20 years old, so feel free to refuse my ideas, hehe.

Byebye
Marcos
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

nitrousnrg wrote:Lets keep talking about serial comm. This is getting off topic ;)
True, thanks!
The length of the string sent to the PC is not relevant. Unless you are transmitting 90+ channels, the communication speed will be fine.
By channels, do you mean data fields/variables? My only concerns for speed come from my desire to track my cars and the time resolution for things like shock pots an G force which I would like to log using the ECU in some fashion if possible.
The procol can be as tricky as you want (but not trickier).
LOL, I love the way ESL people say some things, it really puts a smile on my face!
So don't worry about string lenght :D
I was only concerned from a "wasting good space in flash" point of view.
I did this because i have only 400 bytes of ram on the ECU.
Wow, your code must be super compact and efficient!
For developing, i think that a tool like minicom or cutecom (i love it) isn't enough. It surely is good enough for the first attempts but, in my case, i found my own software for debugging a priceless tool. Its just a thought, take it or leave it.
Absolutely, I think we all agree on that. I just wanted to design the serial spec and THEN implement it to save work and end up with a better implementation too. As soon as it is close to ready, myself and a willing tuning author can begin on implementing it. Speaking of which, if you want to do a write up of your serial interface specifications in a thread here perhaps I could implement it fairly quickly (being a more basic spec) and use it for the mean time?

cutecom : THANK YOU!! :-) I love it too!!! Nifty tool, hello cutecom, good BYE minicom.
btw, i'm 20 years old, so feel free to refuse my ideas, hehe.
I'm 29, and various members on here range from 18 - 50 odd and IMO it should not affect the way we interact very much/at all. I mean that in both ways. IE, I'm happy to learn from your knowledge and experience, and so should a 40 year old be happy to learn from mine. Experience is a good teacher, but some people go a long way and learn little. Thus each on their merits I say :-)

Thanks for sharing with us!

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
nitrousnrg
LQFP144 - On Top Of The Game
Posts: 468
Joined: Tue Jun 24, 2008 5:31 pm

Re: Serial protocol pre-design discussion - give us your input!

Post by nitrousnrg »

Fred wrote:By channels, do you mean data fields/variables? My only concerns for speed come from my desire to track my cars and the time resolution for things like shock pots an G force which I would like to log using the ECU in some fashion if possible.
Ohh. Yes I mean that, buy I wasn't thinking about sending shock position. For such a serial performance I think a "binary" protocol should be used, but we have time to do it, don't worry. Before that the basics must be working.
Fred wrote:I was only concerned from a "wasting good space in flash" point of view.
Did you mean space in flash for datalogging? If you mean space in flash for code, I think you are underestimating your micro. My ECU have only 8kb of flash, 2 of those used for tables and config storage, so I have 6k of flash for code.
512K of flash is amazing!! I can only dream with that. Nevertheless, if the source code fill more than 100-150k of flash, i would think something is wrong.
Fred wrote:Absolutely, I think we all agree on that. I just wanted to design the serial spec and THEN implement it to save work and end up with a better implementation too. As soon as it is close to ready, myself and a willing tuning author can begin on implementing it. Speaking of which, if you want to do a write up of your serial interface specifications in a thread here perhaps I could implement it fairly quickly (being a more basic spec) and use it for the mean time?

cutecom : THANK YOU!! :-) I love it too!!! Nifty tool, hello cutecom, good BYE minicom.
Yup, its a wise way to do it. It takes a little longer, but the results will be evident. My protocol was written "on the fly", and it was changed a couple of times, leaving all datalogs useless. It came out extremely fast (a few hours), but what you are doing is what i should have done from day one. Anyway, i learned a lot :)

Surely i'll have the time in a couple of weeks to write down something about my spec, but don't expect an amazing chat between the ecu and the PC. It just works. Consider i don't have many channels or variables to log (at now, 13, eventually, 20-30), so i log everything.

And yes. cutecom is just nice. You are using linux, right? ttyUSB0 is the device for an usb-serial adapter. if you are using a laptop, its a relevant information (took me 6 weeks to figure it out!!)
Fred wrote:I'm 29, and various members on here range from 18 - 50 odd and IMO it should not affect the way we interact very much/at all. I mean that in both ways. IE, I'm happy to learn from your knowledge and experience, and so should a 40 year old be happy to learn from mine. Experience is a good teacher, but some people go a long way and learn little. Thus each on their merits I say :-)
Cool. i agree. Its great to see my college teachers (eng., mg. ,doctors) being so interested in little projects like mine, or others. I'm learning cool things about linux from a boy :D
I have to go. See u
Marcos
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

nitrousnrg wrote:For such a serial performance I think a "binary" protocol should be used, but we have time to do it, don't worry. Before that the basics must be working.
Agreed :-) I'm not worried, just being prudent.
Did you mean space in flash for datalogging? If you mean space in flash for code, I think you are underestimating your micro.
I think you are underestimating what I want to do with it :-) Also, only 48kB are available for direct addressing with the current compiler. It would be nice to fit it all inside there. There is a similar situation with RAM : 12kB directly addressable, and 2kB of EEPROM directly addressable, the balance is 2k of register space. Currently I'm up around 13kB of code and variables :

Code: Select all

FLASH write: image size <13008 B> time <3.37 s> rate <3859 B/s>
Firmware load was successful!
And 5000 lines of code (6000 of data) :
fred@rwdlsd:~/workspaces/home/freeems/src$ wc *.* inc/* Makefile
<snip>
5918 34984 232657 total
And there is a LOT of code to write yet! You could be right though, I could be paranoid, but I'd rather not hit the paged flash situation if possible.
Yup, its a wise way to do it. It takes a little longer, but the results will be evident. My protocol was written "on the fly", and it was changed a couple of times, leaving all datalogs useless. It came out extremely fast (a few hours), but what you are doing is what i should have done from day one. Anyway, i learned a lot :)
Feel free to share your lessons learned in this thread (and others) and also a youtube vid of your running engine would be good! :-)
Surely i'll have the time in a couple of weeks to write down something about my spec, but don't expect an amazing chat between the ecu and the PC. It just works.
That would be awesome :-) Looking forward to it. If it's as simple as you are indicating, I could probably implement it pretty quickly and have myself a pretty gui to play with :-)
Consider i don't have many channels or variables to log (at now, 13, eventually, 20-30), so i log everything.
Ahhh, this explains your code size! I have : 259 vars, mostly 16 bit, some 8 bit, some 32 bit and there are more to come :-) Megasquirt has something like 600 configurables and many more vars which you can't log. Some of the 259 are settings and some vars, but more vars than settings so far. Looking at it that way though, I think you are right, 48k should be plenty of space really.
And yes. cutecom is just nice. You are using linux, right? ttyUSB0 is the device for an usb-serial adapter. if you are using a laptop, its a relevant information (took me 6 weeks to figure it out!!)
Linux noob then? I'm at about 7 years on only Linux. Others on this site are WAY more veteran than me though :-) Perhaps you just need a google lesson? ;-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
nitrousnrg
LQFP144 - On Top Of The Game
Posts: 468
Joined: Tue Jun 24, 2008 5:31 pm

Re: Serial protocol pre-design discussion - give us your input!

Post by nitrousnrg »

Fred wrote:I think you are underestimating what I want to do with it :-) Also, only 48kB are available for direct addressing with the current compiler. It would be nice to fit it all inside there. There is a similar situation with RAM : 12kB directly addressable, and 2kB of EEPROM directly addressable, the balance is 2k of register space. Currently I'm up around 13kB of code and variables :
I see. I'm not familiar with freescale micros, but those are logic numbers. I have your source code, it looks very good.
Fred wrote:And there is a LOT of code to write yet! You could be right though, I could be paranoid, but I'd rather not hit the paged flash situation if possible.
hehe My pages are 2k long, so my code is spread in 3 pages. Believe me, you don't want to be there. I'll take a look to the 6000 lines of data to see how it is implemented.
I wrote something like 2000 lines of code, in a single main.c :oops: A makefile would be a nice tool (I have an old C compiler). Anyway, it is done, i'm not going back.
Fred wrote:Quote:
Consider i don't have many channels or variables to log (at now, 13, eventually, 20-30), so i log everything.
Ahhh, this explains your code size! I have : 259 vars, mostly 16 bit, some 8 bit, some 32 bit and there are more to come :-) Megasquirt has something like 600 configurables and many more vars which you can't log. Some of the 259 are settings and some vars, but more vars than settings so far. Looking at it that way though, I think you are right, 48k should be plenty of space really.
259 variables?? wow, i think you want to send _every_ variable, its fine for an open source project. I only send variables like RPM, VE, MAP, temperatures, pressures, TP, Injection time, %duty, fuel advance, ign advance, corrections and a few digital inputs.
Any other variable used for calculation is hide in the microcontroller. Please understand the limitations of my micro, i cant spend half of the code space in communications :(
Fred wrote:Feel free to share your lessons learned in this thread (and others) and also a youtube vid of your running engine would be good! :-)
I don't have a running engine :( :(
Everything should be working fine, but we (my uncle and me) spent the last 6 months in the car's chassis. It was a big, dirty ball of rust. We have no money to buy fancy stuff, so we have to make or own parts. Its a custom manifold, used injectors, used fuel pump and sensors... I need a sponsor :P
I don't have a digital camera, maybe this weekend i'll borrow one and take some pictures to the car.
A video: http://es.youtube.com/watch?v=yOOCg2--JFs
that was on christmas, with my dad's laptop (which stop working after that week, motherboard problem).

The fuel injection project was stuck for the last 5 months. I have to build the engine, i hope it will be running for august.

Back to the topic:
How do you want to implement all datalogging?
Log in flash memory? external memory? PC?
I use a PC for data log...
Marcos
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

nitrousnrg wrote:I have your source code, it looks very good.
Thank you :-) (that always gives a warm fuzzy) I try my best, the latest one (0.0.14) looks the best so far, and what I will release next will be what I would call fairly decent. It's starting to come together, that's for sure!
hehe My pages are 2k long, so my code is spread in 3 pages. Believe me, you don't want to be there.
Oh, I don't mean like that, the compiler will handle it automatically, but it is slower you see. I'd still like to avoid it. If i have to get into paging ram etc I'll be unhappy. I think 12k is enough for that though. It should be more than enough actually. For now anyway.
I'll take a look to the 6000 lines of data to see how it is implemented.
I wouldn't bother yet, it's not implemented except as normal variables. Also, that was 6000 lines including makefiles and linker scripts etc. 5000 lines of source including the 260 fields of data many of which are in structs and arrays. We could get up around the tripple region for total variables and fields, but by double it should be fairly complete as a basic unit.
I wrote something like 2000 lines of code, in a single main.c :oops: A makefile would be a nice tool (I have an old C compiler). Anyway, it is done, i'm not going back.
Don't feel bad, MS2 code is 10000 lines in a single source file! 2000 is up there, but not unreasonable for a single C file, but it kinda isn't right for a whole project, but still, it doesn't matter really.
259 variables?? wow, i think you want to send _every_ variable, its fine for an open source project.
I want to be able to send every variable, or just some, in a configurable way. The default logging mask will just carry the normal stuff for most people. Devs and people with install problems to diagnose will be grateful to be able to access it all though.
Any other variable used for calculation is hide in the microcontroller. Please understand the limitations of my micro, i cant spend half of the code space in communications :(
I do understand, and I do believe our goals are quite different too :-) Your project sounds cool though, I look forward to seeing it run. Check out the youtube user markistuff, he has also built his own controller.
Everything should be working fine, but we (my uncle and me) spent the last 6 months in the car's chassis. It was a big, dirty ball of rust. We have no money to buy fancy stuff, so we have to make or own parts. Its a custom manifold, used injectors, used fuel pump and sensors... I need a sponsor :P
Don't we all ;-)
I don't have a digital camera, maybe this weekend i'll borrow one and take some pictures to the car.
That would be cool, what is the car?
How do you want to implement all datalogging?
Log in flash memory? external memory? PC?
I use a PC for data log...
Datalogging should be an external device IMO. As such the serial protocol will support it and then it is up to the user as to what they want to do with it :-) PC will be the natural choice for most people, but I would like to see usb or sd card on my version for my own personal track use requirements. Perhaps I could put my old MS2 to good use passing data through to a disk :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
nitrousnrg
LQFP144 - On Top Of The Game
Posts: 468
Joined: Tue Jun 24, 2008 5:31 pm

Re: Serial protocol pre-design discussion - give us your input!

Post by nitrousnrg »

Fred wrote:I want to be able to send every variable, or just some, in a configurable way. The default logging mask will just carry the normal stuff for most people. Devs and people with install problems to diagnose will be grateful to be able to access it all though.
Yeah, you need that. Specially for developers. It's a relevant part of the project, so make sure it works correctly. The ease of use and develop depends strongly in the communication with the device.
And datalogging is just one part. I use the ecu as an injector tester (with its own GUI), i'll try to make it able to test everyting that can be tested.
Fred wrote:and I do believe our goals are quite different too
yes, we do. I just wanted to do it with minimal resources. Your project is _serious_.
Fred wrote:That would be cool, what is the car?
Its a fiat 128, 1981, 1600cm³, 8v.

Your micro comes with an usb module, or something to access easily a usb or sd card?

You are doing all in the exact way it should be done. Congratulations!

I'm not used to write in english... its stressing :P
regards
mrk
Marcos
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

nitrousnrg wrote:I use the ecu as an injector tester (with its own GUI), i'll try to make it able to test everyting that can be tested.
That's a good idea, but I'd rather have such "test" code separate to the real code to keep the real code focussed and lean. It would be straight forward to whip up an injector test firmware and interface to JUST do that, and considering how often someone normal would do this, I think it's no big deal to load a new firmware and reload the real stuff afterwards. For those that are doing it often, they would likely use a permanent setup with JUST the injector test code on it. This is something to work on later anyway :-)
Your micro comes with an usb module, or something to access easily a usb or sd card?
No, nothing like that. It has SPI which can interface with the SD card through the MMC standard. For other things there are interface chips available that make it easy to do.
You are doing all in the exact way it should be done. Congratulations!
Thanks :-)
I'm not used to write in english... its stressing :P
Relax, you are doing a great job!! I have some dutch friends that speak english fairly well, and I have trouble communicating via a keyboard with them sometimes. You I can understand easily!

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

OK, getting back on topic solidly, here is where we stand with regards serial discussions :
  • CRC semantics, header included, separate, or header not checked, pre escaping, post escaping, 8 bits/16 (I vote 8)
    8 bits decided by Stu and I because no one else said anything!
  • Flow control - hardware, software, ignoring incoming requests when busy - requirements for each etc.
    Still need to determine how to do this in hardware.
  • Payload type bit flag for protocol/firmware use (eliminate the 20 limit at the expense of a header bit)
    Agreed by Stu and I again.
  • Stretchy ints, yes or no, if yes, how, if no, how big for numbers.
    Waiting example code.
  • Appendix A payload format details correct/incorrect etc.
    Strings will be sent with trailing \0 byte as per standard C practice.
  • Include extensive "basic" vars in part 1 spec for compatibility with simple devices that can't cope with flash protocols ;-) ?? (would require detailed thinking...)
    Undecided, but bit flag for protocol/firmware use of payload type field allows this.
  • Acknowledgment form. I say empty = positive, full = stretchy int with error code of some sort. Thoughts?
    Undecided, but can't see why not.
I'm going to add the things we have agreed on to the documents and do a release of them again. Please Stu, Shameem, Marcos, Jean, Abe, Keith, etc discuss these outstanding things such that we can start to get initial implementations of this protocol out and running asap.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

I've had another thought to throw into the mix too :

When receiving data the ECU has to buffer ALL the data before it can verify the packet and act upon the contents. Because the capabilities are small it may not be possible to buffer all that much. Certainly there is a finite limit.

I can see two solutions to this, though there may be others :
  1. Put a hard limit on packet size into the protocol spec (seems like a bad idea if we ever get a new cpu etc)
  2. Include a max packet length in the ping response with the interface version string or have a separate command to get that figure. That way the protocol isn't limited, but the ECU can say "I can only handle 1024 bytes of data before I overflow, so don't try to send more or it will be discarded".
Thoughts?

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply