Embedded Linux: Using Compressed File Systems

http://lwn.net/Articles/219827/

This series is all about making small systems, from the kernel on up. Inthe first part I covered theTinyLinux project and its eventual integration into the kernel to help reduce kernel sizes for small systems.Inthe second part, I lookedat the use of the Initramfs and its role in providing a root file system (directly or indirectly) for an embeddedsystem.

Now it's time to look at getting applications and utilities into thesystem, still keeping an eye on size. The most direct approach is to useas few utilities as possible, even replacing/sbin/init with a singleapplication. This is possible in very small systems but, generally speaking,if you only have a single application to run you probably didn't need thecomplexity of a multitasking system like Linux to run it anyway. There areother, smaller operating systems that might be better suited in that case.

There are a number of ways to keep application layer tools small.If you have multiple applications and/or require the facilities in Linux,then you can (and should, for production systems) consider stripping yourbinaries of all symbols. The symbols are useful for debugging purposes butwon't be of much value to your users. Additionally, using compile-timefeatures to reduce size is another option, and will be the focus of thefinal article in this series. For now, we'll consider yet another option:using a compressed file system.

Compressed File Systems

File systems provide the structure for managing files on storage media,such as disks or tapes. While a device driver knows how to get data to andfrom those devices, file system provide the logical structure of that data.There are a huge number of file systems types, ranging from the standardext3 you'll find on many Linux systems to parallel and clusteredfile systems, to steganographic file systems that can both encrypt and hidedata on the media.(Note that Wikipedia hasa nice long listof file systems).

A compressed file system is one that uncompresses data as it is retrievedand may or may not compresses data as it goes into the storage media.Working with compressed files is an obvious benefit for saving space onsmall systems. The decision to use a compressed file system is usuallybased on the storage media you'll use in your system. A ram-disk basedsystem, for example, might copy data from flash into the ramdisk. SinceRAM is essential for system operation the size of the ram disk wouldprobably be best kept small. Compact flash or hard disk based systems, onthe other hand, offer more storage but may still be too small to fit allthe required files without some sort of compression.

While compressed file systems offer you more space for files, they also mayaffect performance. There may be unacceptable overhead in managing the decompression of large files at run time. And compressing files on the fly iscomputationally expensive; random writes of compressed data isdifficult to achieve. Therefore it is far more common for compressedfile systems to be read-only.

Compressing data is a common practice for live CD distributions, which usecompression to squeeze a more complete distribution onto the limited sizeof a CD or DVD. But many of the live CD distributions don't actually use acompressed file system, instead using an conventional file system image madeup of compressed blocks which are uncompressed when read using the "cloop",or compressed loopback, device. But this isn't a compressed file system. It's ablock level device handling compressed data.

The Knoppix distribution popularized the use of cloop when its author, Klaus Knopper, picked up support of the driver. Manyother live CDs followed suit. One advantage of using this kind ofcompressed image is that, since the blocks are compressed independently, itis possible to seek to specific blocks without uncompressing all theblocks. The disadvantage of such a device is that the entire image mustfit into memory in order to be uncompressed.

An example of a real compressed file system is CramFS, a file system popularwith embedded users of the 2.4 kernel for use with the initrd image. Thisfile system actually has compressed files with uncompressed metadata. Thefiles are placed in the file system from a standard directory using themkcramfs program, which compresses the files one page at a time. This isdone, for example, when creating an initrd image.

Another example of a compressed file system is e2compr. This is actually aset of patches to make the well known EXT2 file system handle on-the-flycompression and decompression. It supports both 2.4 and 2.6 kernels, buthas not been submitted for inclusion in either because of the complexity ofthe patches. As with CramFS, metadata in e2compr is not compressed.

SquashFS

A more recent (and more actively supported, the last updates coming in midJanuary 2007) compressed file system isSquashFS. SquashFS is a kindof successor to CramFS because it aims at the same target audience whileproviding a similar process for creation and use of the file system.What makes SquashFS an improvement over CramFS is best stated by PhillipLougher in a linux-kernel mailing list post:"SquashFS basically gives better compression, bigger files/file systemsupport, and more inode information."

Both SquashFS and CramFS use zlib compression. However, CramFS uses afixed size 4KB block while SquashFS supports from 0.5KB to 64KB. Thisvariable block size allows for much larger file systems under SquashFS,something desirable for complex embedded systems like digital videorecorders. Also SquashFS supports compression of both the metadata and block fragments while CramFSdoes not. And, while CramFS is integrated with the kernel source, SquashFSis not. It comes as a set of kernel patches and the driver module.The CELinux Forum provides somecomparisons of SquashFS against other file systems (compressedand uncompressed).

JFFS2

Another compressed file system is JFFS2, the Journaling Flashfile system, version 2. It was designed specifically for use with both NOR and NANDflash devices, and recently received an update via David Woodhouse for theNAND flash memory being used in the OLPC project. JFFS2 is actually a bitmore sophisticated than SquashFS because it providesmechanisms for plugging in different compression algorithms, including not using anycompression at all. But unlike SquashFS, JFFS2 is integrated into thekernel.

So if you're building an embedded system with flash storage, wouldn't yoube better with JFFS2? Not necessarily.

Accordingto the OpenWRT project, which uses both SquashFS and JFFS2, SquashFS provides better performance than JFFS2. Additionally, at leastin the case of the few files that need to be updated for a productionversion of the project, there is little advantage to using a read/writeJFFS2 compressed root file system with respect to the performance hit it incursvs a read-only SquashFS root file system used with a writable JFFS2 file systemfor stored files.

JFFS2 is a read/write file system while SquashFS is a read-only file system.A runtime system very often needs to write to its root file system.Imagine making updates to/etc/hosts, for example, as you might with aembedded video recorder client trying to access a server backend on a local network.If writing to the file system is required for an embedded system, how couldyou use SquashFS at all?

Some projects, like OpenWRT, use a hybrid system that uses a read-only rootfile system mixed with a read/write file system for saving files. In such ahybrid you might use special configurations or modified applications toaccess read/write file systems, but that doesn't help if you need writeaccess to /etc/hosts on a read-only file system. What you need is a methodof having parts of the directory structure writable while other parts areread-only. What you need is a stackable file system like UnionFS.

Using UnionFS: BusyBox and SquashFS together

UnionFS is a mechanism for mounting two directories from differentfile systems under the same name. For example, I could have a read-onlySquashFS file system and a read/write JFFS2 file system mounted togetherunder the root directory so that the JFFS2 would be /tmp and/etc while the SquashFS might be everything else.

So how might you use this with a compressed file system and our BusyBoxbased utilities we created in the last article? First, we build our kernelwith SquashFS patches and then build the UnionFS driver as a loadable module.Next, we build BusyBox with all the runtime utilities we need and installthe result to a local directory on the build machine, let's call it"/tmp/busybox". Next, we package those files into a compressed SquashFSfile system:

    mksquashfs /tmp/busybox /tmp/busybox.sqfs -info 

This command takes the contents of /tmp/busybox and compresses it into a file systemimage in/tmp called busybox.sqfs. The -info optionincreases verbosity, printing the filenames, original size and compressionratio as they are processed.

We then create an initramfs with another build of BusyBox that has onlyminimal utilities - enough to do mounting of the loopback device and loadingkernel modules, plus the UnionFS module we built previously (which wemanually copy into the directory after we rebuild BusyBox). We might addsupport for other devices like a CDROM if we store the SquashFS file thereor JFFS2 and support for flash memory if we store the SquashFS file there.

At runtime, I need a writable file system to go with my read-only SquashFSfile system. I'll use the tmpfs file system which puts all the files I'llwrite at runtime in virtual memory. In my init script for my initramfs, Iadd:

    mkdir /.tmpfs    mount -w -t tmpfs -o size=90% tmpfs /.tmpfs    mkdir /.tmpfs/.overlay

The overlay directory will be used to store data written by my embeddedsystem.

When you boot your 2.6 kernel, you'll have a BusyBox based initramfs withan init script and your SquashFS file system (or a way to get to thatfile system via commands in your init script). I'm mounting thebusybox.sqfs file from the root directory of a CD over the loopbackdevice onto a directory in my initramfs, so I add the following to the initscript:

mkdir /.tmpfs/.cdrommount -r -t iso9660 /dev/cdrom /.tmpfs/.cdromlosetup /dev/loop0 /.tmpfs/.cdrom/root.sqfs

Then I can mount the loopback device as a SquashFS file system to anotherdirectory I've created in my tmpfs:

mkdir /.tmpfs/.sqfsmount -r -t squashfs /dev/loop0 /.tmpfs/.sqfs

UnionFS mounts multiple directories, in either read-only or read-writemode, onto a single directory. In the init script, I place threedirectories side by side under a single UnionFS directory:

mount -w -t unionfs -o \dirs=/.tmpfs/.overlay=rw:/.tmpfs/.cdrom=ro:/.tmpfs/.sqfs=ro \unionfs /.union

What this does is place all three directory structures, which are referredto as branches under UnionFS, under/.union; any conflicting directorynames are resolved by taking the first one found, searching the branches left toright. So if there is an/.tmpfs/.overlay/etc/hosts (a file we'vecreated at runtime, for example), it takes precedence over/.tmpfs/.sqfs/etc/hosts.

With this command, when you write to /.union (which later becomes the rootdirectory due to a switch_root in the init script), the writes go to theread/write directory which is on the tmpfs file system. But this writablespace is in memory and won't survive reboots. If you need to save databetween boots, you could mount a compact flash drive under/.tmpfs/cf anduse that instead of /.tmpfs/.overlay in the previous mount command.

Which directory gets the write if there are two read-write branches?UnionFS uses "copy-up", which causes any attempt to write to a read-onlybranch to be written to the next read-write branch on its left. Imaginecreating a SquashFS for/etc, one for /var and one for everything else inyour root partition. Then if you had 2 compact flashes you could use onefor writes to/etc and one for writes to /var simply by ordering thesecorrectly when you mounted them under the UnionFS file system.

UnionFS is considered by some to be too buggy for production use, thoughI've never had much trouble with it when building live CDs. If youexperience problems using UnionFS, you might considerAuFS as an alternative. AuFS started out as arewrite of UnionFS but has since evolved into its own file system. SLAX, aSlackware based live CD that originally used UnionFS, has migrated to AuFS.In fact, a bug bounty was offered by SLAX for a bug and the winner of thatbounty, Junjiro Okajima, is the author of AuFS.

Next in the series: uClibc

This long running series (it's taken me awhile to write each of the threearticles so far) has one piece left: using uClibc to reduce program size.This is a reduced size version of the standard glibc library, specificallybuilt for small footprint systems.