SD Card and FAT FS for FreeEMS

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
Spudmn
LQFP112 - Up with the play
Posts: 232
Joined: Thu Feb 10, 2011 12:27 am
Location: Auckland, NZ

SD Card and FAT FS for FreeEMS

Post by Spudmn »

This thread is to track the progress of the SD card and FAT file system used for data logging on FreeEMS.

EDIT: Comments here only please: viewtopic.php?f=8&t=1080

Specifications

SD Card
FAT files system
Non blocking of the main loop.
Try to reduce the RAM usage.

Current limitations (so that a working solution can be done quickly)

FAT 16 only.
Can only open 1 file at a time.
Can only write files.

Possible Future Enhancements.

FAT 32
SDHC Cards
File read and download to a PC.

Try to avoid posts that change the specifications. We need to get something running first and then can add enhancements later.

Aaron
User avatar
Spudmn
LQFP112 - Up with the play
Posts: 232
Joined: Thu Feb 10, 2011 12:27 am
Location: Auckland, NZ

Re: SD Card and FAT FS for FreeEMS

Post by Spudmn »

A few years ago I built my own data logging system that logged GPS data. This used a SD card and the FAT16 file system. I used bits of code that I found on the internet and got the system working very well. I enhanced the code so that if power was lost to the logger only the last block would be lost.

At the time, it was only going to be used by me so I didn't mind if the code I used was GPL or not, as long as it was free for me to use.

Before starting this project I wanted to check that I could GPL what I have done. Unfortunately some of the code I have used is licensed Non Commercial. So I am starting from scratch.

I will base my code on ChaN's excellent work on his FATFS. http://elm-chan.org/fsw/ff/00index_e.html

His license is as follows

About FatFs License
This is a copy of the FatFs license document that included in the source codes.
/*----------------------------------------------------------------------------/
/ FatFs - FAT file system module R0.08b (C)ChaN, 2011
/-----------------------------------------------------------------------------/
/ FatFs module is a generic FAT file system module for small embedded systems.
/ This is a free software that opened for education, research and commercial
/ developments under license policy of following trems.
/
/ Copyright (C) 2011, ChaN, all right reserved.
/
/ * The FatFs module is a free software and there is NO WARRANTY.
/ * No restriction on use. You can use, modify and redistribute it for
/ personal, non-profit or commercial products UNDER YOUR RESPONSIBILITY.
/ * Redistributions of source code must retain the above copyright notice.
/
/-----------------------------------------------------------------------------/
Therefore FatFs license is one of the BSD-style license but there is a big difference. Because FatFs is for embedded projects, the conditions for redistributions in binary form, such as embedded code, hex file and binary library, are not specified to increase its usability. The documentation of the distributions need not include about FatFs and its license document, and it may also. Of course FatFs is compatible with the projects under GNU GPL. When redistribute it with any modification, the license can also be changed to GNU GPL or BSD-style license.

I plan to make lots of changes to it and turn it into a non-blocking File System. Referred to as NBFS from now on.

Whenever the code is waiting of IO access it will return to the main line. I will explain this in more detail another day.

I have got the FATFs code complied under GCC and written a simple HAL that will use a File as virtual disk. This file can be mounted as a loop back to check every thing is working.
Starting the development on a x86 platform will allow me to use gdb and other tools to get the high level done and later write a HAL for the FreeEMS.

Aaron
User avatar
Spudmn
LQFP112 - Up with the play
Posts: 232
Joined: Thu Feb 10, 2011 12:27 am
Location: Auckland, NZ

Re: SD Card and FAT FS for FreeEMS

Post by Spudmn »

Next we need to think about hardware for awhile.

The SD card is a 3.3V device so we will need to do some level shifting. I have a circuit that I am currently using so that shouldn't be a problem. I will design up a little daughter board that has a 3.3V regulator, level shifters and a card reader socket.

The SPI pins MISO, MOSI, SCK are available on connector P71 but I still need a Chip select. There is the CAN bus on P71 but I want to use CAN later as well so I can't use those pins.

I need to find CS from some where.

Do we want to worry about the Card detect switch and the write protect switch?

My thoughts are not to bother. We should be able to detect if the card is present through software and if you have a card plugged into your Puma then I think that you should expect data to be written to it.

It might be a good idea to be able to control the power to the SD card, so that may be another pin we will need.

Aaron
User avatar
Spudmn
LQFP112 - Up with the play
Posts: 232
Joined: Thu Feb 10, 2011 12:27 am
Location: Auckland, NZ

Re: SD Card and FAT FS for FreeEMS

Post by Spudmn »

OK. Its been a long time since I did anything on the SD Card and FAT FS. I have started work on this again.

I have changed my plans a bit so here are my new milestones

Milestones

* Take Chans code and get it to run on a PC with a virtual file system disk image.
* Write Unit tests for the code.
* Refactor the code.
* Write the SPI interface.
* Make enchantments as needed.

Enhancements

*These may include fine tuning the trade-offs between RAM usage and speed.
*Make sure the FAT is updated as the files are added so that a lose of power to the card will not destroy the whole file.
*Implement a non blocker state machine if needed.


Doing the coding this way will be quicker and with the unit testing in place any changes made can be quickly checked.

I will be releasing the virtual file system code on git hub soon.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: SD Card and FAT FS for FreeEMS

Post by Fred »

Spudmn, we need to have a chat, I have a number ending in 31 for you, I assume that it's still valid. I can't call you just yet, but as soon as I can, I will ask you for a date/time and give you a call.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Spudmn
LQFP112 - Up with the play
Posts: 232
Joined: Thu Feb 10, 2011 12:27 am
Location: Auckland, NZ

Re: SD Card and FAT FS for FreeEMS

Post by Spudmn »

Time to discuss what features we want on the SD Card code.
Some features will use more RAM and code so we need to play the trade off game.

FAT16 and/or FAT32

Only one file open at a time.

Only one partition.

8.3 file names or long file name.

Folder structure.

Let me know your thoughts.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: SD Card and FAT FS for FreeEMS

Post by Fred »

Spudmn wrote:Time to discuss what features we want on the SD Card code.
Some features will use more RAM and code so we need to play the trade off game.
Can you tell us which features will use more RAM and code so we can play the trade off game effectively. EDIT < epic copy paste work there. Note reuse of logical blocks of text? Keyboard had some wine intake and is drying...
Spudmn wrote:FAT16 and/or FAT32
What's the file size limit for each?
Only one file open at a time.
100% fine.
Only one partition.
Covers most cards, seems reasonable.
8.3 file names or long file name.
Isn't this handled at some meta level anyway? What's the extension called? How much does this cost us? Code is fine, we've got acres of flash, RAM is less fine, but there should be enough for this SD card stuff :-)
Folder structure.
Max files in one dir? Speed of access with diff contents? Deeper = slower? Wider = slower? I don't mind flat for this, but flat could have negative technical impacts too.

There you have it! :-)
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: SD Card and FAT FS for FreeEMS

Post by sry_not4sale »

Re folder structure - what about:

/YYYY/MM/YYYY-MM-DD_hh:mm.extension ?
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: SD Card and FAT FS for FreeEMS

Post by Fred »

Assuming that we end up supporting directory structure at all, then I agree with that, or maybe one deeper with DD also in the path. However, this is only applicable if we have an RTC attached, what do we do with the names if we don't? We can use EEPROM to hold a "last file name written" value and even a preferred name prefix and just go like this, based on last eeprom value and over write as we create the new file:

name.00000001.bin
name.00000002.bin
name.00000003.bin

etc. Or the same in 8.3 without the name. part.

What do you do for last modified time/date without date? What about date starts at 1979 at time of starting engine, gets updated while alive, and gets used when log starts, and used again when log ends? That removes the "what date is it" logic from the SD card stuff, and if we DO have RTC, then it's likely only asked at init time anyway and self updated. Though we could have a getSystemTimeAndDate() call that gets it directly from the RTC and returns or some null value in a struct as a flag instead. Lots of options there.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Spudmn
LQFP112 - Up with the play
Posts: 232
Joined: Thu Feb 10, 2011 12:27 am
Location: Auckland, NZ

Re: SD Card and FAT FS for FreeEMS

Post by Spudmn »

The source code can now be found here. http://github.com/Spudmn/FreeEMS_DataLogger

So far it doesn't do much. It runs on a PC and I have simulated a MMC card and the files get written to a disk image.

I will try to answer the above questions soon.
Post Reply