diff --git a/ChangeLog b/ChangeLog index 5860d25e14..35fe431284 100644 --- a/ChangeLog +++ b/ChangeLog @@ -6100,4 +6100,5 @@ functions (2013-11-20). * configs/olimex-lpc-h3131/src/Makefile: Add SDRAM support. Untested and probably needs some fine tuining (2013-11-21) - + * fs/smartfs/README.txt: Add README for SMARTFS file system. From + Ken Pettit (2013-11-23) diff --git a/Documentation/README.html b/Documentation/README.html index 240771459b..9cad7087f8 100644 --- a/Documentation/README.html +++ b/Documentation/README.html @@ -8,7 +8,7 @@

NuttX README Files

-

Last Updated: November 18, 2013

+

Last Updated: November 23, 2013

@@ -239,6 +239,8 @@ | | | `- README.txt | | |- nxffs/ | | | `- README.txt + | | |- smartfs/ + | | | `- README.txt | | `- procfs/ | | `- README.txt | |- graphics/ diff --git a/README.txt b/README.txt index 70ddb7e4ab..f060faa480 100644 --- a/README.txt +++ b/README.txt @@ -1167,6 +1167,8 @@ nuttx | | `- README.txt | |- nxffs/ | | `- README.txt + | |- smartfs/ + | | `- README.txt | `- procfs/ | `- README.txt |- graphics/ diff --git a/fs/smartfs/README.txt b/fs/smartfs/README.txt new file mode 100644 index 0000000000..8d4db1c930 --- /dev/null +++ b/fs/smartfs/README.txt @@ -0,0 +1,363 @@ +SMARTFS README +^^^^^^^^^^^^^^ + +This README file contains information about the implemenation of the NuttX +Sector Mapped Allocation for Really Tiny (SMART) FLASH file system, SMARTFS. + +Contents: + + Features + General operation + SMARTFS organization + Headers + Multiple mount points + SMARTFS Limitations + ioctls + Things to Do + +Features +======== + + This implementation is a full-feature file system from the perspective of + file and directory access (i.e. not considering low-level details like the + lack of wear-leveling, etc.). The SMART File System was designed specifically + for small SPI based FLASH parts (1-8 Mbyte for example), though this is not + a limitation. It can certainly be used for any size FLASH and can work with + any MTD device by binding it with the SMART MTD layer. The FS includes + support for: + - Multiple open files from different threads. + - Open for read/write access with seek capability. + - Appending to end of files in either write, append or read/write + open modes. + - Directory support. + - Support for multiple mount points on a single volume / partition (see + details below). + +General operation +================= + + The SMART File System divides the FLASH device or partition into equal + sized sectors which are allocated and "released" as needed to perform file + read/write and directory management operations. Sectors are then "chained" + together to build files and directories. The operations are split into two + layers: + + 1. The MTD block layer (nuttx/drivers/mtd/smart.c). This layer manages + all low-level FLASH access operations including sector allocations, + logical to physical sector mapping, erase operations, etc. + 2. The FS layer (nuttx/fs/smart/smartfs_smart.c). This layer manages + high-level file and directory creation, read/write, deletion, sector + chaining, etc. + + SMART MTD Block layer + ===================== + + The SMART MTD block layer divides the erase blocks of the FLASH device into + "sectors". Sectors have both physical and logical number assignments. + The physicl sector number represents the actual offset from the beginning + of the device, while the logical sector number is assigned as needed. + A physical sector can have any logical sector assignment, and as files + are created, modified and destroyed, the logical sector number assignment + for a given physical sector will change over time. The logical sector + number is saved in the physical sector header as the first 2 bytes, and + the MTD layer maintains an in-memory map of the logical to physical mapping. + Only physical sectors that are in use will have a logical assignment. + + Also contained in the sector header is a flags byte and a sequence number. + When a sector is allocated, the COMMITED flag will be "set" (changed from + erase state to non-erase state) to indicate the sector data is valid. When + a sector's data needs to be deleted, the RELEASED flag will be "set" to + indicate the sector is no longer in use. This is done because the erase + block continaing the sector cannot necessarly be erased until all sectors + in that block have been "released". This allows sectors in the erase + block to remain active while others are inactive until a "garbage collection" + operation is needed on the volume to reclaim released sectors. + + The sequence number is used when a logical sector's data needs to be + updated with new information. When this happens, a new physical sector + will be allocated which has a duplicate logical sector number but a + higher sequence number. This allows maintaining flash consistency in the + event of a power failure by writing new data prior to releasing the old. + In the event of a power failure causing duplicate logical sector numbers, + the sector with the higher sequence number will win, and the older logical + sector will be released. + + The SMART MTD block layer reserves some logical sector numbers for internal + use, including + + Sector 0: The Format Sector. Has a format signture, format version, etc. + Sector 1: The 1st (or only) Root Directory entry + Sector 2-8: Additional root directories when Multi-Mount points are supported. + Sector 9-11: Reserved (maybe for sector wear-leveling, etc.) + + To perform allocations, the SMART MTD block layer searches each erase block + on the device to identify the one with the most free sectors. Free sectors + are those that have all bytes in the "erased state", meaning they have not + been previously allocated/released since the last block erase. Not all + sectors on the device can be allocated ... the SMART MTD block driver must + reserve at least one erase-block worth of unused sectors to perform + garbage collection, which will be performed automatically when no free + sectors are available. + + Garbage collection is performed by identifying the erase block with the most + "released" sectors (those that were previously allocated but no longer being + used) and moving all still-active sectors to a different erase block. Then + the now "vacant" erase block is erased, thus changing a group of released + sectors into free sectors. This may occur several times depending on the + number of released sectors on the volume such that better "wear leveling" + is achieved. + + Standard MTD block layer functions are provided for block read, block write, + etc. so that system utilities such as the "dd" command can be used, + however, all SMART operations are performed using SMART specific ioctl + codes to perform sector allocate, sector release, sector write, etc. + + A couple of config items that the SMART MTD layer can take advantage of + in the underlying MTD drivers is SUBSECTOR_ERASE and BYTE_WRITE. Most + flash devices have a 32K to 128K Erase block size, but some of them + have a smaller erase size available also. Vendors have different names + for the smaller erase size; In the NuttX MTD layer it is called + SUBSECTOR_ERASE. For FLASH devices that support the smaller erase size, + this configuration item can be added to the underlying MTD driver, and + SMART will use it. As of the writing of this README, only the + drivers/mtd/m25px.c driver had support for SUBSECTOR_ERASE. + + The BYTE_WRITE config option enables use of the underlying MTD driver's + ability to write data a byte or a few bytes at a time vs. a full page + at at time (which is typically 256 bytes). For FLASH devices that support + byte write mode, support for this config item can be added to the MTD + driver. Enabling and supporting this feature reduces the traffic on the + SPI bus considerably because SMARTFS performs many operations that affect + only a few bytes on the device. Without BYTE_WRITE, the code must + perform a full page read-modify-write operation on a 256 or even 512 + byte page. + + SMART FS Layer + ============== + + This layer interfaces with the SMART MTD block layer to allocate / release + logical sectors, create and destroy sector chains, and perform directory and + file I/O operations. Each directory and file on the volume is represented + as a chain or "linked list" of logical sectors. Thus the actual physical + sectors that a give file or directory uses does not need to be contigous + and in fact can (and will) move around over time. To manage the sector + chains, the SMARTFS layer adds a "chain header" after the sector's "sector + header". This is a 5-byte header which contains the chain type (file or + directory), a "next logical sector" entry and the count of bytes actually + used within the sector. + + Files are stored in directories, which are sector chains that have a + specific data format to track file names and "first" logical sector + numbers. Each file in the directory has a fixed-size "directory entry" + that has bits to indicate if it is still active or has been deleted, file + permission bits, first sector number, date (utc stamp), and filename. The + filename length is set from the CONFIG_SMARTFS_NAMLEN config value at the + time the mksmartfs command is executed. Changes to the + CONFIG_SMARTFS_NAMLEN parameter will not be reflected on the volume + unless it is reformatted. The same is true of the sector size parameter. + + Subdirectories are supported by creating a new sector chain (of type + directory) and creating a standard directory entry for it in it's parent + directory. Then files and additional sub-directories can be added to + that directory chain. As such, each directory on the volume will occupy + a minimum of one sector on the device. Subdirectories can be deleted + only if they are "empty" (i.e they reference no active entries). There + are no provision made for performing a recursive directory delete. + + New files and subdirectories can be added to a directory without needing + to copy and release the original directory sector. This is done by + writing only the new entry data to the sector and ignoring the "bytes + used" field of the chain header for directories. Updates (modifying + existing data) or appending to a sector for regular files requires copying + the file data to a new sector and releasing the old one. + +SMARTFS organization +==================== + + The following example assumes 2 logical blocks per FLASH erase block. The + actual relationship is determined by the FLASH geometry reported by the MTD + driver. + + ERASE LOGICAL Sectors begin with a sector header. Sectors may + BLOCK SECTOR CONTENTS be marked as "released," pending garbage collection + n 2*n --+---------------+ + Sector Hdr |LLLLLLLLLLLLLLL| Logical sector number (2 bytes) + |QQQQQQQQQQQQQQQ| Sequence number (2 bytes) + |SSSSSSSSSSSSSSS| Status bits (1 byte) + +---------------+ + FS Hdr |TTTTTTTTTTTTTTT| Sector Type (dir or file) (1 byte) + |NNNNNNNNNNNNNNN| Number of next logical sector in chain + |UUUUUUUUUUUUUUU| Number of bytes used in this sector + | | + | | + | (Sector Data) | + | | + | | + 2*n+1 --+---------------+ + Sector Hdr |LLLLLLLLLLLLLLL| Logical sector number (2 bytes) + |QQQQQQQQQQQQQQQ| Sequence number (2 bytes) + |SSSSSSSSSSSSSSS| Status bits (1 byte) + +---------------+ + FS Hdr |TTTTTTTTTTTTTTT| Sector Type (dir or file) (1 byte) + |NNNNNNNNNNNNNNN| Number of next logical sector in chain + |UUUUUUUUUUUUUUU| Number of bytes used in this sector + | | + | | + | (Sector Data) | + | | + | | + n+1 2*(n+1) --+---------------+ + Sector Hdr |LLLLLLLLLLLLLLL| Logical sector number (2 bytes) + |QQQQQQQQQQQQQQQ| Sequence number (2 bytes) + |SSSSSSSSSSSSSSS| Status bits (1 byte) + +---------------+ + FS Hdr |TTTTTTTTTTTTTTT| Sector Type (dir or file) (1 byte) + |NNNNNNNNNNNNNNN| Number of next logical sector in chain + |UUUUUUUUUUUUUUU| Number of bytes used in this sector + | | + | | + | (Sector Data) | + | | + | | + --+---------------+ + + +Headers +======= + SECTOR HEADER: + Each sector contains a header (currently 5 bytes) for identifying the + status of the sector. The header contains the sector's logical sector + number mapping, an incrementing sequence number to manage changes to + logical sector data, and sector flags (committed, released, version, etc.). + At the block level, there is no notion of sector chaining, only + allocated sectors within erase blocks. + + FORMAT HEADER: + Contains information regarding the format on the volume, including + a format signature, formatted block size, name length within the directory + chains, etc. + + CHAIN HEADER: + The file system header (next 5 bytes) tracks file and directory sector + chains and actual sector usage (number of bytes that are valid in the + sector). Also indicates the type of chain (file or directory). + +Multiple Mount Points +===================== + + Typically, a volume contains a single root directory entry (logical sector + number 1) and all files and subdirectories are "children" of that root + directory. This is a traditional scheme and allows the volume to + be mounted in a single location within the VFS. As a configuration + option, when the volume is formatted via the mksmartfs command, multiple + root directory entries can be created instead. The number of entries to + be created is an added parameter to the mksmartfs command in this + configuration. + + When this option has been enabled in the configuration and specified + during the format, then the volume will have multiple root directories + and can support a mount point in the VFS for each. In this mode, + the device entries reported in the /dev directory will have a directory + number postfixed to the name, such as: + + /dev/smart0d1 + /dev/smart0d2 + /dev/smart1p1d1 + /dev/smart1p2d2 + etc. + + Each device entry can then be mounted at different locations, such as: + + /dev/smart0d1 --> /usr + /dev/smart0d2 --> /home + etc. + + Using multiple mount points is slightly different from using partitions + on the volume in that each mount point has the potential to use the + entire space on the volume vs. having a pre-allocated reservation of + space defined by the partition sizes. Also, all files and directories + of all mount-points will be physically "mixed in" with data from the + other mount-points (though files from one will never logically "appear" + in the others). Each directory structure is isolated from the others, + they simply share the same physical media for storage. + +SMARTFS Limitations +=================== + +This implementation has several limitations that you should be aware +before opting to use SMARTFS: + +1. No wear leveling has been implemented. The allocation scheme has a + bit of inherent wear-leveling since it automatically distributes + sector allocations across the device, but no provisions exist to + guarantee equal wearing. + +2. There is no CRC or checksum calculations performed on the data stored + to FLASH, so no error detection has been implemented. This could be + added by "stealing" one of the sequence number bytes in the sector + header and incrementing the sector version number. + +3. There is currently no FLASH bad-block management code. The reason for + this is that the FS was geared for Serial NOR FLASH parts. To use + SMARTFS with a NAND FLASH, bad block management would need to be added. + +4. The released-sector garbage collection process occurs only during a write + when there are no free FLASH sectors. Thus, occasionally, file writing + may take a long time. This typically isn't noticable unless the volume + is very full and multiple copy / erase cycles must be performed to + complete the garbage collection. + +5. The total number of logical sectors on the device must be less than 65534. + The number of logical sectors is based on the total device / partition + size and the selected sector size. For larger flash parts, a larger + sector size would need to be used to meet this requirement. This + restriction exists because: + + a. The logical sector number is a 16-bit field (i.e. 65535 is the max). + b. The SMART MTD layer reserves 1 logical sector for a format sector. + c. Logical sector number 65535 (0xFFFF) is reerved as this is typically + the "erased state" of the FLASH. + +ioctls +====== + + BIOC_LLFORMAT + Performs a SMART low-level format on the volume. This erases the volume + and writes the FORMAT HEADER to the first physical sector on the volume. + + BIOC_GETFORMAT + Returns information about the format found on the volume during the + "scan" operation which is performed when the volume is mounted. + + BIOC_ALLOCSECT + Allocates a logical sector on the device. + + BIOC_FREESECT + Frees a logical sector that had been previously allocated. This + causes the sector to be marked as "released" and possibly causes the + erase block to be erased if it is the last active sector in the + it's erase block. + + BIOC_READSECT + Reads data from a logial sector. This uses a structure to identify + the offset and count of data to be read. + + BIOC_WRITESECT + Writes data to a logical sector. This uses a structure to identify + the offset and count of data to be written. May cause a logical + sector to be physically relocated and may cause garbage collection + if needed when moving data to a new physical sector. + + +Things to Do +============ + +- Add file permission checking to open / read / write routines. +- Add reporting of actual FLASH usage for directories (each directory + occupies one or more physical sectors, yet the size is reported as + zero for directories). +- Add sector aging to provide some degree of wear-leveling. +- Possibly steal a byte from the sector header's sequence number and + implement a sector data verification scheme using a 1-byte CRC. + +