MiniRTL-V2.3 (Kernel 2.2.14)
by Nicholas McGuire
Finite State Machine Labs, Inc.
System software used in embedded systems covers a wide range, spreading from micro-kernels at a size of 40 KB up to full-featured operating systems such as Linux with a run time kernel in the range of 1-6 MB. In this document we are concerned with the high end of the embedded systems domain. More precisely, we deal here with embedded systems based on Linux, referred to as Linux embedded user system, which try to provide the embedded systems with as close to a complete user environment as possible.
Expressing the size of such systems in terms of memory is hard. They start somewhere in the range of 2 MB disk and 2 MB RAM requirements for Paul Gortmaker's 1.0.9-ELF kernels (linux.about.com/pub/Linux/kernel/ at Metalab), Linux Lite (please drop me a note if you know the official home-page url), and 2 MB disk with 4 MB RAM for 2.0.x or 2.2.x kernels. The 2.4.X kernels look bad at the moment with concern to size. If these systems are to do more than the absolute minimum, however, then a 4 MB disk and 4 MB RAM mark the bottom line at which such a system can be identified as Linux embedded user system. If an 8 MB RAM requirement is acceptable, this kind of embedded system can be used for any application or project. In particular, these systems are appropriate for mobile units, a small series of special devices or systems that need extensive maintenance and remote monitoring.
Traditionally, system software for embedded systems has been developed independently of general purpose operating systems. So why take mainstream Linux as a basis and drop it to an embedded system? The answer is actually quite simple, though not purely technical: GNU/Linux is a stable desktop system which provides many development tools. Thus, it makes life very much easier, as it offers the possibility of doing development on a desktop system and, with some minor restrictions, dropping the result to the embedded system without modification. Debugging, optimizing and all development-related work can be done on the desktop system. There is no need for any special development environments and no need for proprietary tools. The decision to implement the current kernels, although they are quite large, is based upon three tenets:
Sticking with the mainstream kernel makes it easy to ensure compatibility with the standard desktop GNU/Linux system. Realtime Linux development is also tightly coupled to the current kernel development in many essential aspects. Although it might not always be necessary to have the newest version for a particular application, full SMP support requires current RTLinux kernel/patches, and support for flash-disk and disk-on-chip is just emerging in 2.4.0.
RTLinux for embedded platforms is in general intended for high-end projects and for projects that want to utilize the advantage of using commodity-component PCs. Even if most people don't recognize a mini-tower equipped with an old i386/i486 as an embedded system, from a functional standpoint this is often a very interesting alternative for low-cost and prototyping systems. And at the least, many jobs can be done in this way.
Before going into the details of a Linux embedded user system and its sample implementation, Mini-RTL, an introduction to the boot process and Linux system initialization is given.
Linux Boot Process
The description of the Linux boot process presented here (drawn from the source files of the 2.2.14 Kernel) is short and definitely not complete. It is informative, however, even if subject to change. First, some general remarks will define its borders and limits. For an in-depth description of the Linux kernel, refer to linux-doc-project/linux-kernel/ at Metalab), and The Linux Kernel by David A. Rusling.
The Linux boot process will refer to a ``normal'' desktop system -- steps pertaining to RAM-disk based systems only will be marked as such. An embedded system need not necessarily be RAM-disk based, but Mini-RTL was implemented this way and so we include this information. Some of these notes apply to RTLinux-specific modifications and are mentioned only to put them in context. No complete description of the RTLinux-specific modifications should be anticipated here. We also limit the discussion to the Intel processor-family -- although real-time Linux ports to other hardware are in-progress at the time of writing, no such (non-beta) systems are available yet.
The filenames given here refer to a freshly unpacked 2.2.14 kernel as obtained from kernel.org. In whatever location of the file system you unpack the source, you will end up with a directory called linux and all paths are relative to this directory. E.g., if you unpack the kernel tree in /tmp you will get a /tmp/linux directory. As convention, the file rd.c say, located in /tmp/linux/drivers/block, will be referenced as drivers/block/rd.c.
Chronological Boot Sequence
bootsect, uncompressed: exactly 512 bytes of 8086 machine code. This code loads Setup and the system, and is only executed if no OS-loader is present (e.g., Linux-only desktop systems).
setup, uncompressed: 8086 machine code. It contains setup code and the decompression routines.
system, compressed: 80386 code for the actual system (note that every Intel PC boots as a single processor 8086).
device_setup()-- so the kernel knows how to talk to the device.
filesystem_setup()-- registers the file system with the kernel. Now the kernel knows how to handle the directory entries and file attributes, pe, etc., on the minix filesystem.
mount_root(),-- mounts it. Now it is known to the VFS, and can be accessed.
basic system directories
basic system links (/tmp, /var/tmp)
busybox links via the list in config/root.bb.links
grep links (grep is a shell-script!)
system devices
realtime device files
/proc and the mount table (/etc/mtab)
permissions on a few files
Now the system is ready to drop in the rest of the executables/lib/modules. The next step is to enable the appropriate capabilities in the kernel (if not compiled in).
load fat.o + msdos.o
load floppy.o
mount /dev/fd0 on /config/mnt (previously created dir)
At this point the minimum system is already accessing the file system and running as Linux, comparable to a desktop booted into a shell.
Mapping the Boot Sequence to Files
From a chronological point of view, the boot sequence is easy to understand. It is more difficult to follow how this procedure is implemented by functions, mapped in a file structure, and finally compiled into the kernel. To further illustrate the boot process, we have picked out some kernel files and will show how the distribution of functions fits quite well with the kernel source directory structure. The functional split presented here is harder to clarify -- because all drivers can be built into the kernel, it is hard to say what is in the kernel and what is in a driver. We therefore assume here that everything that can be built as a module is built as a module, leaving us with what we would call the kernel proper.
Bootstrap Files
arch/i386/kernel/trampoline.S. This is the spark of life in your box, and it does what the filename says: It jumps to the first bytes of the kernel and gets the boot-process going. This is loaded by your OS loader (lilo, syslinux).
arch/i386/boot/compressed/head.S. The first bytes of the kernel contain basic setup routines, which are in the uncompressed part of your zImage or vmlinuz. The file arch/i386/boot/compressed/misc.c contains the routine to decompress the kernel image. After decompression, the runtime kernel is in place for startup. The code executed up to this point will be overwritten and is not part of the kernel runtime functionality. None of this will be present at system run time; it's the kernel bootstrap only. (You can see the message "Freeing unused kernel memory: 44k freed" at system boot, indicating that this initial code is being dropped.)
The First Task
The code contained by the files listed here are not called before main.cbut rather as part of the main routine. We discuss them init/main.c only because they contain the first steps of system setup. Note that the path arch/i386/kernel contains all Intel hardware-related portions of code (which is very little) and that these routines (especially the ones in assembler) are the core hardware abstraction layer. This will change with different platforms, but the basic structure will probably remain the same.
arch/i386/kernel/head.S. The first steps of the kernel are in here: setting up the stack- pointers, organizing your memory and silly tasks like finding your CPU type.
arch/i386/kernel/entry.S. Handling interrupts, fault-occurrences and task-switches go through here. All system calls have their entry-point via the sys-call table which is initialized here at boot-up. This is one of the important files that is modified to achieve hard-realtime capabilities in the RTLinux Kernel.
arch/i386/kernel/irq.c. The filename says it all. This is the point where the real-time kernel will intercept all interrupts (IRQs) and make them soft IRQs. The rest of the Linux kernel will not notice. It simply never sees a "real" interrupt--it only sees the interrupts RTLinux decides belong to a non-realtime process--so processing will occur in the non-real-time realm if the interrupt is not destined for a real-time job. A realtime thread will be invoked if the interrupt was requested by a hard-realtime task. This interception is what guarantees that no non-real-time job can disturb hard real-time executives in RTLinux.
arch/i386/kernel/timer.c.Timing functions, which are naturally critical to a real-time environment, are also modified in real-time Linux. Notably, interception by the timer interrupt is disabled and nanosecond resolution is added to timer functions (restricted, naturally, to usable hardware resolution).
arch/i386/kernel/init_task.c. The task structure in which every task is represented internally in the kernel. This task structure will be initialized by init_task for all non-realtime tasks running under Linux. Realtime tasks/threads are allocated when the realtime module is inserted and register with the realtime scheduler only.
init/main.c. The kernel's main routine. Function names in parentheses are the names by which you can find them in init/main.c. The main routine, naturally enough, does lots of work, such as: calls to hardware initialization functions, initialization of CPUs in the system (smp_init), calibration of the delay loop (calibrate_delay), handling of boot-options (parse_options), initialization of interrupts (trap_init, init_IRQ) and then the enabling of them. It also initializes the system console so the system can report what's going on. The detected memory is then set up and assigned to the different memory types (uidcache_init, filecache_init, dcache_init, vma_init, buffer_init, signals_init, inode_init and file_table_init). At this point, the kernel can start using hardware abstraction layers like file systems and devices (insofar as they exist at this point and are not compiled as modules).
For RAM-disks, this is now set up (drivers/block/rd.c) in buffer cache by calling ramdisk_start_setup. A file system is set up on the RAM-disk (drivers/block/rd.mkminix.c). The root archive is then decompressed with the kernel's built-in gzip function (arch/i386/boot/misc.c). At this point, the real root is set because the ROOT_DEV at boot-up pointed at the boot-media (which is not necessarily the root device of the runtime system). The root device is initialized by preparing the device_driver (device_setup) and initializing the executable formats that will be available (binfmt_setup, fs/exec.c). This is what tells the kernel, for example, that a file starting with #!/bin/bash is a shell-script and needs to be treated appropriately, and registers the file systems available with the kernel. Now that the kernel knows how to handle file systems, knows what to do with an executable and has been informed of the runtime root-device, this root device can be mounted (mount_root).
The RAM-disk startup then unpacks your root tarball (drivers/block/rd.untar.c) which was loaded into RAM by ldlinux.sys. This could not be done any earlier since the kernel did not know how to handle a file system, and where the root file system actually is located, until after the previous step. The unpacking of the core OS tarball must be done by the kernel because there are no accessible executables yet. Now we open the <linuxrc.tty as the console device (we don't have a real console available yet, as there is no /dev set up at this point) and call linuxrc to set up the file system content. The file system is already set, so linuxrc now populates it with files and directories (that is, the files and directories that are in the other *.rtl packages on the floppy) and sets up device files (finally giving the system a real console to be opened a little later).
A few things still need initialization. If mtrr was enabled it is setup now (mtrr_init), sysctl is initialized (sysctl_init), followed by the setup of available buses (pci_init, sbus_init, etc) and then networking capabilities (sock_init). This part of init/main.c is very configuration-dependent, so you will find lots of #ifdef / #endif pairs here. To understand exactly what's going on, looking at the .config file of the top-level kernel source directory will show you the way.
This is where we set up of the real-time capabilities, which is done by rtl_init and then re-enabling interrupts (sti), making the system capable of doing hard real-time jobs.
Up to this point we were not using a real console, but now the real one (/dev/console) can now be opened. Since it was already set up by linuxrc, the kernel had the capabilities to handle a console, but it was missing the device file, so syslinux only had to setup this "hook"--not really the console as such. The kernel proper is now fully booted.
init/version.c. This gives the system a name, so you can have multiple kernels on one system, and when booting an alternative kernel it will know which modules to use. It will also complain if you try to load modules that were built for a different kernel (version mismatch).
Architecture Specific Dependencies
We list the '.o' files here, not the '.c', because during the kernel build you will see these files, and I would guess that describing their content is more instructive than trying to go through all the '.c' files in 68 MB of source code.
arch/i386/kernel/kernel.o. The platform-dependent specifics for the kernel calls, interrupts, traps and system calls. Although it's not quite correct, you might think of it as being the hardware abstraction layer for signals and system calls.
arch/i386/mm/mm.o This file contains the platform-specific part of memory handling and handles protection, allocation of memory ranges and alignment, so this will be very different on the PPC, Alpha or whatever. It's the hardware abstraction layer for the system memory. Note that here only the basic memory routines are defined, and there are advanced memory-related topics associated with cache (mtrr) and special memory "devices" (especially in the 2.4.0 kernel) that allow for more hardware-specific tuning/optimization.
kernel/kernel.o. Basic kernel high-level functionality: fork, panic, etc. The resource management functions handle I/O mapping, timing, interrupts, signals and the tasks that do these jobs.
Basic System Resources
These are the basic resources we need in an OS and are not really hardware-specific. There may be drivers built into the kernel, but they are not part of the kernel proper.
mm/mm.o. The high-level memory functions: allocation, paging, swapping, mapping, etc. The virtual memory layer is coded in mm/*.
fs/fs.o. File system abstraction layer. This is what makes it possible to access all the different types of file systems through a common layer (readdir, read-write and so on). The abstraction layer between hardware drivers and actual files is provided here, such as block devices, pipes and fifos (non-rt fifos, that is).
ipc/ipc.o This represents the multi-tasking in Linux, with its basic process communication routines, messages, semaphores, shared memory functions and process accounting, again only for the normal linux kernel. These capabilities in the rt-realm are handled by runtime-inserted rtl_ modules.
Everything we've discussed up to this point is what we need in the kernel just to boot up and wait with a blinking cursor, doing nothing more. This is the same situation as when you have a kernel on your hard disk and nothing else installed. You would actually end up at panic: No init found, which is a fully initialized system doing nothing.
Kernel Modules
An operating system is not only a process and memory resource handler, it also needs access to (manage) hardware and further abstraction layers, like network protocols and high-level services that autonomously manage resources (automounter, nfs-server). These are all initialized by init and are not really kernel/boot related (although some, like knfsd and autofs, are crawling into kernel space... waiting for kvi to appear). From the call to init on, the system is really what one thinks of as a Linux system. Many of these advanced resources, including real-time capabilities, are added at run time by inserting kernel modules (the kernel needs only to contain the appropriate hooks), so a brief description of kernel module insertion is appropriate here.
Kernel modules are a.out or ELF code , compiled but built as relocatable code (I guess you will not find any a.out modules any more. At least, I've never seen one...). So, there are no absolute addresses in kernel modules, instead there are symbolic references. When a module is loaded (kernel/module.c), its code is patched up, with symbolic names being replaced by addresses which are supplied by the kernel from its exported symbol table. Symbols from the new module also can be exported and become normal kernel symbols, which in turn may be used by further modules (also called module-stacking). This is why module-dependencies exist. Modules once inserted into the kernel are not distinct from the monolithic part of the kernel, except for having an initialization code segment and a cleanup code segment, and being allocated a little differently.
The symbols init_module, cleanup_module are not exported, but the kernel knows where they are to be found, so it can access them when removing the module. To make the usage of modules by the kernel possible, the module_list is initialized at boot-up (init_module() in init/main.c) with a pseudo-module entry (the empty module list entry). To this module_list, new modules may then be dynamically added.
MISSING (An explanation of insmod/rmmod and the basic module buildup is to follow.)
Designing a Minimum System
Minimizing disk usage in a RAM-disk based system is performance-critical. Most embedded systems are low-memory systems, at least by desktop standards. A 2.2.14 kernel will boot and run in 4 MB, but there is little room for applications in this setup. Linux is quite sensitive to low-RAM conditions. We might be exaggerating only slightly by stating that doubling RAM on low memory systems will increase overall performance just as much as doubling CPU speed.
A RAM-disk resides in buffer cache, so increasing disk usage will reduce available memory. This implies that we should review strategies for optimizing performance, and also makes clear why dynamic disk usage during boot-up is not really critical (as long as you never try to exceed available total ram, which can happen if you try to put the entire system into a single archive file). So, there is little necessity to reduce the image size of the booting system, as only run-time size is relevant.
Reduce the system size
Designing the system requires the most time, because it is hard to tell what one can safely drop and what one really needs in a "minimum" system. Fortunately, there are some measures that can be taken to reduce system size, as discussed in the following.
Exploiting redundancy
Linux is very redundant when it comes to executables: you can do the job of displaying a file on the console with cat, tail, more, less, grep and even dd. So finding redundant executables and defining the scope really required is the first step. Some questions to ask here are:
Deletion
One point here is probably unique to minimum systems, and so deserves a little more detail: A normal system has its executables in place, and these will be used at the next system boot. Embedded systems that run in RAM-disk are running off an image copy, so they don't need to preserve files if they are not required for the system's operation. They can simply be removed. Candidates for removal are:Compression
Many executables/modules may reside on the file system in compressed form, because they will only be needed for specific purposes during maintenance. It is essential, though, to make sure that these compressed executables are not called from scripts without decompressing them first. This sounds obvious, but the dependencies in even a 2MB system can be quite complex. It is in general not a trivial task to ensure this. Candidates for compression are all executables that will only be used by administrative personnel and not called by scripts (after boot up that is!), such as editor(s), maintenance programs (such as ifconfig, ping, rmmod) and file utilities (such as ctar, ae).
Compressed filesystems like the compressed fat filesystem are not realy a solution for such a system , the filesystem modules them selfe are quite large and the compression is relatively bad. Also this approach reduces storage-device bandwith substantially thus seems to me to be an inadequate solution for embedded systems and especially realtime systems.<\P>
All these measures can reduce system runtime size by 30 to 40 percent without any black magic. A 40 percent filesystem size reduction means a substantial increase in RAM available at run-time.
Reduce run time sizes
The above size reduction was achieved by using compression, which is limited to the file system size of executables. The run time size is not affected in the same manner, so the next step, naturally, is to reduce run time size as well. This does not always correlate with a reduction in file system size!
There are lots of provisions made in the regular Linux kernel that assume operation as a full multi-user multi-tasking system, with no real limits to the number of processes and users. As a consequence, more resources are allocated than a minimal system needs. As a first step, therefore, optimization of the Linux kernel for minimum systems consists of no more than throwing out some of these resources (pty's, tty's, hdX and lots of hooks in /dev).
Real optimization, in the sense of modifying kernel behavior to better use specific resources, is not intended here (and is probably not that easy either). It's also counterproductive, due to the fact that there is a new Linux kernel out there every few weeks. Removing resources, however, is simple and requires no real kernel hacking. Starting points may be found in include/linux/*.h. Be careful with the limits defined there, though, for not everything can be reduced without side-effects. If you modify any #define statement, you will need to grep through the kernel tree to find if it is referenced anywhere else, thus creating side-effects if modified.
For example, there will rarely be a need for 63 console devices on an embedded system, so the maximum number of consoles in include/linux/tty.h can be reduced to 3 (leaving you with four consoles). Other good candidates are device files. You will not need to create all of hda* through hdh*, as it is difficult to imagine an embedded system running with eight IDE devices hooked up.
A significant drawback to compile-time modifications is that you will have to dig around in the kernel source every time a new kernel comes out. This means that you will have a job to do every few weeks if you want to keep up with the current kernel. Putting minimization flags in the config process would be a nice solution, but I doubt that it will find its way into the main stream kernel.
Many executables that you use on a day-to-day basis on your desktop system have many options that you will rarely need and could either spare completely or substitute with generic commands. Going through the executables you have designated for your minimum system and stripping out options, error messages and built-in help is an easy way of reducing such files.
Also, consider compile-time options, as they also influence size. On desktop systems this isn't a big deal, so often the Makefiles will provide the safest options, which are not necessarily the most efficient with respect to size. As a general way to reduce size, don't include the default libs, but explicitly include the libs you need. To do so, use the -nostdlib and -nostartfiles flags with gcc, and include the libc with -lc.
Whenever programs on a minimum system produce data, you might consider outputting it in a compressed format from the beginning. This will not only save scarce disk-space; downloading to a server or pushing data over an NFS-mount can also be substantially improved if a compressed format is used. Alternatively, data files and log-files may be compressed periodically by a simple cronjob, especially for log-files on embedded systems where in many cases only recent events will be relevant.
Table: Compiler options and executable size
|
-O2 |
33439 |
read_lpt |
|
striped (or with -s) |
7420 |
read_lpt |
|
-O2 -g |
36259 |
read_lpt |
|
-O2 -nostdlib -lc |
2772 |
read_lpt |
|
-O2 -nostdlib -s -lc |
2008 |
read_lpt |
|
-O2 |
33585 |
monitor |
|
no optimization |
33649 |
monitor |
Compiling with glibcX.X.X When building programs for a MiniRTL system, naturally only
libs available on the MiniRTL system may be used. Since glibc-2.0.7 is not
necessarily up to date, and it is to be expected that most Linux desktop
system are running newer versions of glibc, a short introduction on how to
compile with a given set of libs instead of with your default set is given
here: Get glibc-2.0.7 from the official glibc site at GNU.org, or grab a tarball of the Mini
RTL libs. Assuming you got your tarball from thinkingnerds.com, do the
following:
Don't add this directory to /etc/ld.so.conf, since the
lib names would conflict with the newer libs of the same name, and they
would either not be used (since they are listed below the path to the new
libs), or your system would segfault horribly (if you list the old libs
before the new libs). Compiling programs to run on MiniRTL now requires
that you explicitly name the directory to the compiler and include ALL libs
explicitly, including libc itself.
The -nostdlib flag to cc has the effect
that no libs are included at all by default, which is also why you need to
list libc as -lc on the command line! The side effect of this is
that gcc mutters a little about some start symbol that ld can't resolve,
but that should be okay. At least, I've had no problems with this up to
now. You will see something like:
Libraries
Some of the library problems were mentioned above; libc is a very large and powerful library, but for minimum systems it is a problem since it is very resource consuming. Nevertheless, we stick with full libc, because reducing its size is not only complicated (you must figure out all function calls that are unused and cleanly remove them), but also because it poses a compatibility problem. If you try to optimize by modifying libraries, you lose compatibility with your desktop system. At the same time, it means maintaining a private version of the library, and you don't want to maintain your own libc track!
Stripped libraries are dramatically smaller, and since debugging can comfortably be done on the full desktop-system, there is no need to include debug symbols on the minimum system. The same holds for executables that can be stripped, thereby massively reducing size. To reduce the number of required libraries, it is best to define a set of libraries for the minimum system and then strictly build on those, and then only incorporate software that runs with those libraries. This is not such a big problem. Due to the vast amount of software/sources on the Internet, it is quite easy to find editors, scripting languages and the like that will not need any special libraries. Naturally, the system will have a little bit of an archaic touch, but that's ok; you're not expected to work full time with ash and ae as your shell and editor. For debugging purposes or administrative jobs, you can get used to it.
The minimum list of libraries (libc-2.0.7pre6) (assuming network support) is:
libc-2.0.7.so
libcrypt-2.0.7.so
libdl-2.0.7. so
libncurses.so.4
libnsl-2.0.7.so
libnss_db-2.0.7.so
libnss_dns-2.0 .7.so
libnss_files-2.0.7.so
libresolv-2.0.7.so
libss.so.2.0
libutil- 2.0.7.so
libuuid.so.1.1.
The Busybox Concept
An interesting concept which packs lots of functionality into a minimum system is the so-called busybox. It is a monolithic collection of base functions in a single executable, with function selection by the name of the calling process. This is handled by symbolically linking functions names to the busybox executable.
The advantage is that there is little overhead in the executable for argument handling and the like, since there is only one parser for all linked executables. Also, functionality in the busybox is reduced to a minimum, though this comes at a price: some standard functions behave a little differently than one might expect, with some options missing and others reduced in scope. Adding functions to the busybox is quite simple, but it requires recompiling the entire suite, and modifications to the system initialization, to set up the required links appropriately.
The Busybox vs. Standard ToolsFirst, let us look at the file size of the busybox (0.28) when compared to the sum of the sizes of the corresponding standard Linux tools. The busybox occupies 65KB, whereas the sum of the Linux tools occupies 608KB. Even the stripped Linux tools still require 592KB (egcs-2.91.66, glibc 2.0.7).
This comparison might not seem fair, since many options available in the standard tools are not available in the busybox, but the essence of the busybox concept is that you can pack lots of functionality into a tiny executable, if reduced to the minimum needs of administration. The real challenge is deciding precisely what is needed and what can be removed. With all the dependencies within an OS like Linux (and the use of shell-scripts for many jobs), this is not easily decided. This leads us back to one of the motivations for such a minimum system: reduced system complexity, which eases understanding of such dependencies.
The second comparison is the hello world program in C compiled with gcc (egcs-2.91.66) as an ELF-executable and the size difference of a hello world function added to busybox called via a link named hello. The ELF-executable comes in at 33KB raw and 7KB stripped. The increase in size of the busybox, however, is a nominal 161 bytes.
This shows how efficiently the overhead reduction is achieved by having one main routine and calling everything else as a named function of the busybox.
Statically Linked Binaries
A question that might arise in connection with putting libraries on a minimum system is: Are statically linked executables an alternative if my program needs some particular library which is not available on the system? Although this is done by some install disks (e.g., RH 6.1/6.2 has the installer statically linked with python libs) the answer is probably "no." The following comparison should make this clear: the hello world program statically linked under glibc 2.0.7 weighs in at a mighty 416KB. Stripped, it still exceeds 105KB.
This might seem unfair, but it is a devastating example. Indeed, we know of no example where a statically linked executable is smaller than the size difference caused by adding it to the busybox, and in most cases it is just about as big as adding the ELF-executable and the lib together.
Run-Time Optimization Strategies
The optimization strategies presented here are only applicable in very special situations and might seem a little weird at first glance. Yet consider that many embedded systems run for months without any human intervention. For these systems, optimization strategies that are repaid by an increase in administrative complexity might be acceptable.
Kernel Resource Optimization
The modular structure of the Linux kernel permits optimization of run time size. Modules for network-support may be loaded, enabling the embedded system to drop log/data-files to a central server system, and then be removed from the kernel until needed again. The same holds for file system support. The size reduction may be performance-relevant on low-memory systems.
A further kernel optimization may result from reducing allocated resources, either by kernel parameters at boot-up or by reducing resources in the kernel source. This does not really require kernel hacking, merely going through kernel source files and stripping down resources that you will not need for embedded systems (e.g., by setting the maximum number of IDE devices to 2 instead of 8, reducing number of consoles to 4 and so on). These modifications can be done without going very deep into the kernel and are not too hard to apply again when a new kernel comes out. This stripping of resources will not substantially reduce the compressed kernel image but it will reduce the run time kernel memory trace.
Many kernel resources can be tuned on your desktop system via the sysctl interface operating via /proc. By tuning system settings on the full system you can find an optimized setting for your kernel. On a minimum system sysctl would be a real waste since it is very large, so the optimized values for free pages pagecache, etc. must be set in the kernel source and the kernel recompiled. This is not as bad as it sounds, since optimization is rarely so application-specific that it would change within a running system.
File Systems
Many file systems are designed for very big systems. Embedded systems rarely need a directory depth of 1024 directories, or maximum filename length of 1800 characters. Memory can be saved if the correct file systems are used with the appropriate options. A comparison is shown in Table 11-1.
Table: Comparative File System Sizes
|
1K blocks |
Used |
Available |
% Use |
FS creation command |
|
1423 |
0 |
1423 |
0% |
mformat a: |
|
1421 |
1 |
1420 |
0% |
mkfs.minix |
|
1435 |
1 |
1434 |
0% |
mkfs.minix -l14 -i32 |
|
1412 |
13 |
1327 |
1% |
mkfs.ext2 |
|
1412 > |
13 |
1399 |
1% |
mkfs.ext2 -m0 |
|
1431 |
13 |
1418 |
1% |
mkfs.ext2 -m0 -N32 |
Keep in mind that every restriction on the file system (DOS 8.3 filename length or mkfs.minix -n14 giving you 14-character maximum filename length) is paid for by having to be careful of file conversions when moving files around (notably, msdos-fs is a real limitation). This easily can lead to hard-to-detect side-effects.
On-demand Loading
It is not necessary for all executables to be resident on the system. You can have administrative tools (like an editor or some additional kernel modules only required for administration) on the remote site and simply upload them on demand, removing them when the tasks have completed. This is performance-critical since, as noted above, the file system resides in buffer cache. Alternatively, you can have modules with nfs support on the embedded system, which can temporarily increase disk space availability tremendously. You can even swap via nfs, but this is not suitable via a modem-line! Of course, nfs has security implications that need to be considered if running such a system outside a secure network.
The point of all this is that there are very few resources really needed on-site. Much can be moved off-site as long as there are mechanisms available to hook them up on demand (via scripts, cron etc.).
Building a System
Building a minimum system is actually quite simple. The real work is sorting out what you really need and getting all of this down to a size that will fit on your media. Setting up your media (floppy, flash-disk or whatever), is a four-step process:
For DOS disks, steps one through three can be done with one program: syslinux. To play with such media you don't need to really crank away at a floppy for days -- you can create a 1.44 MB DOS/minix disk image and mount this into your system. As soon as you are happy with your disk image, you then drop it to the real media.
Black Magic on Image Files
To create such an image file, you dump nothing to a file (making sure it's the right amount of nothing!), otherwise it will flood your system:
% dd if=/dev/zero of=/tmp/empty_floppy.img bs=512 count=2880 (1.44MB)
Then, make a file system on this nothing:
% mkfs.minix /tmp/empty_floppy.img
And now you can simply mount it like a regular device:
% mount -t minix -o loop /tmp/empty_floppy.img /mnt
The advantage of this, clearly, is that you have full bandwidth on this 'floppy,' and you can create image files of arbitrary size for test purposes. After you are happy with your virtual floppy, you can umount it and dump it to a real floppy like so (assuming your floppy is /dev/fd0):
% dd if=/tmp/empty_floppy.img of=/dev/fd0
For flash-disks, the procedure of dumping the image to the media is not standard, so this may vary.
MISSING (how to set it up on flash disks, comments/reports appreciated)
Redundant Systems
For embedded systems it is important to ensure system stability without requiring foreign intervention. An interesting possibility for ensuring this is to have a fully-redundant system, not in the sense of a redundant runtime system (RAID hot-swap processors etc.), but in the sense that a reboot of the system will put the system in a defined (sane) state.
An example of such an embedded system is Mini-RTL, explained later in detail, which will boot off media (floppy or flash-disk) and after that will not use this media during operation. This is accomplished by setting up a RAM-disk and loading the system into this volatile media. This has a clear disadvantage concerning permanent modification of the system, but guarantees that a system reboot will put it in a defined state.
Doing this via floppy is very handy during development of a system, since such a system can be tested on regular PC hardware. For a real embedded system, flash-disks are a better solution because they offer far better boot-up time than from a (cranking) floppy, and they are not as fragile. Updating such a system is still possible by mounting the boot media in the running system, uploading the modified image to the system via network and placing it on the boot-media. A slight disadvantage of such a solution is that it requires additional RAM (one can create temporary disk-space via nfs-mounts, but that's not applicable to all systems--especially with concern for security constraints). Basically, if the boot-media is mounted, you can also scp/tftp the image-files directly to the boot-media, provided changes don't effect the os-loader or filesystem of the boot-media.
To minimize this disadvantage, the RAM-disk is placed in buffer cache. Therefore, the actually used RAM is not the size of the RAM-disk as defined at boot time. This (boot time) size defines the maximum that the RAM-disk can occupy, but not the size of the file system running in the RAM-disk at any given time.
One aspect of using an initial RAM-disk which must be taken into consideration is that during boot up of the system there are certain dynamic disk space requirements for unpacking the archives. To minimize this dynamic space requirement, the archive size and order of unpacking is crucial. Splitting packages into separate sub-systems will ease maintenance, and unpacking with decreasing sizes at boot-up will keep dynamic disk-usage at a minimum. Generally, unpacking a tarball requires the full tar file size during unpacking. So, by having small packages and unpacking the largest ones first (when more disk space is still free), and processing the smaller packages as the file system gets populated, the minimum free disk space requirement is brought down to 1 MB. In general, this is acceptable for such a system, since this amount of disk space is required for log-files and temporary data storage anyway.
Mini-RTL, A Sample System
Mini-RTL is a fully operational standalone Linux system, compacted on a 1.44 MB floppy, with the capability to boot as a networked system (www.thinkingnerds.com/ projects/minirtl.html). This system successfully implements the optimization strategies described in the previous sections. It could be used as a basis for somewhat larger systems, or as a usable starting point for even smaller systems. Also, it is far easier to understand system dependencies if you have 1.44 MB to figure out and not a 4 GB desktop system.
Mini-RTL was originally based on the Linux router project. Naturally, it is a little archaic, and you should not expect GNU/emacs as the system default editor. The main features available on this minimum system are not really specific to this system, but reflect rather simply the standard Linux features. We list them here because these capabilities might not have been taken into account when considering embedded system design:
You might not need all this, and you might be missing something, but Mini-RTL should give you a good idea of what can be squeezed onto 1.44 MB! Considering this, a 2 MB flash-disk based system gives you lots of space to play with.
Kernel Modifications
The kernel modifications described here were not done as part of this project, but were rather the basis for it. Describing them all here in detail would not be possible, so we will only give a brief overview of the concepts involved.
To boot off the initial RAM-disk, you need to modify the standard kernel a bit. Basically, it is no more difficult than replacing the hook in init/main.c that points to the regular root device and making it point to the initial RAM-disk. Using raw images, this would be all that is required to run a system, yet running raw images is not very comfortable during development (and also quite messy to modify). Since raw images are required to tell the kernel exactly where to jump, raw images have no file system hook. So, to permit the usage of standard tar.gz archives, some additional modifications are added to create an initial minix file system on the RAM-disk. The kernel then unpacks all it needs into the file system.
One problem with this is that the initial console does not exist at the time when the kernel is actually ready to boot up the system, because it has not been created yet. This is solved in a pragmatic way by having a special device packed on the disk (called linuxrc.tty) which is a minimalist tty, only used at boot time.
In drivers/block/rd.c, the RAM-disks are created by the kernel at boot time. This is where the code for creating the minix file system on /dev/ram0 and unpacking the root archive from tar.gz form is inserted. This is essential because now the root image is no longer a raw image, but simply a standard tar.gz file, making manipulation more easily done. Due to this add-on, it is possible for any Linux user to create the modified archive files. All you need to do is pack everything you would like to have in a tar archive (provided you can get it on a 1.44 MB floppy) and gzip it, then tell linuxrc to unpack it via the "commandline options" in syslinux.cfg
Besides the above modifications, the normal RTL patches were applied, with no modifications, as of v1.1 (kernel 2.0.36) and v2.0/2.2 (kernel 2.2.13/14). The front-end application interfaces are restricted in the libs available on the Mini-RTL system. For the standard demo-apps, no modifications were required, but application design for a minimum system must take the libs restrictions into account.
Resource optimization
The suggested strategies for reducing kernel runtime memory trace were applied to the MiniRTL system, resulting in a small kernel footprint (408k kernel code, 412k reserved, 276k data, 24k init). Systems running in 2MB-RAM + 2MB-RAM disk with a 2.2.13/2.2.14 Kernel have been built successfully (without network support though), but are currently very restricted. With a consequent reduction of allocated resources, a stable system that can actually do work in 4MB total should be achievable using a current main-stream Linux Kernel.
Running MiniRTL
Note on minirtl2.0....
If you enable sshd on miniRTL2.0, it will only be stable on a 9MB box. With 8MB, bus-errors occur and it will be unstable. So, for miniRTL2.0 the memory limit should be set to 10MB (6RAM + 4RAM-DISK). The ramdisk is located in buffer-cache -- therefore log or datafiles that fill up the ramdisk dynamically reduce the available RAM. Therefore, a safe RAM/RAMDISK setup is 4MB RAMDISK + 6MB RAM. This system will stay functional and stable, even if the RAMDISK is filled to 100% (provided programs can handle the "Disk-full" situation correctly). I have not found the cause for this, but the simplest solution is to upgrade to minirtl2.2 or 2.3. For SMP boxes, it is a little hard to say what the bottom line really is, but with 16MB you should be safe on SMP.
Requirements:
You will need the following packages on your system:
You can find a (hopefully up-to-date) collection of the necessary tools at rtlinux.org.
You will need root-privileges, or you will need your admin to give you access to a 1.44MB DOS partition on the system (or a loop mounted floppy image).
This can be done by giving you access to the floppy device with the following entry in /etc/fstab:/dev/fd0 /floppy auto noauto,user 0 0
The user mount option does the trick, allowing a normal user to mount/umount the floppy device. For loop mounting, I know of no simple trick to grant users the required privileges without giving them more or less full system access as root.
Quick start in three steps:
You can "optimize" dd by giving it a larger blocksize:
This will do the job somewhat faster on most systems, though not on all.
As long as the network setting is not correct, the launch of syslog/klog may be delayed until it gets the timeouts and decides to continue without the network. This problem goes away as soon as the network is up and running, and other than this bothersome delay at boot-time there is no penalty for a misconfigured network.
RT Example Programs on MiniRTL:
The following list of example realtime linux programs are available on MiniRTL-V2.3, located in /lib/modules. Kernel modules are listed as .o, frontend programs are listed without extensions:
In addition to these examples, the basic rtlinux modules are also on-disk:
More information on rtlinux, and on writing and understanding realtime code can be found at rtlinux.org. Following are descriptions of the examples sorted by directories. (These are the directories under examples in the rtlinux top-level directory under which you can find the sources of these programs.)
Hello world in realtime -- examples/hello:
One of the first C programs to master is "hello world", a simple program that will print "hello world" on your screen. In the realtime linux world it's not much different. This simple "realtime hello world" will write "hello world" from the rt-side to the non-rt-side of your running rtlinux box. To see the messages hello.o is writing to you, you must call the Linux command dmesg, which will print the kernel messages to stdout. This is because on the realtime side of your box you can't directly access the console, so hello.o drops a message to the linux kernel and that in turn prints it via printk.
files:
hello.cclose to the simples module possible , init_module , cleanup_module as with all modules, and start_routine ,the actual task ,which is only a periodic rtl_printk of a message and any arguments received. After inserting hello.o with insmod, simply execute dmesg, and it will talk to you.
hello.c shows the basic structure of many simple realtime modules:
usage:
Sound with a pc-speaker -- examples/sound:
The PC-speaker is driven by a programmable timer/counter chip in your PC. It will generate a periodic signal that can be thought of as a "sine wave," or simply a periodic signal of some shape, or simply a beep. If this beep is turned on and off in a reasonably fast and precisely timed fashion then one can produce "sound". This condition is what makes it clear why such a speaker driver would not properly work when executed from regular non-realtime linux -- on an unloaded box it might work more or less, but with increasing load the distortion would become unacceptable.
The data used to control the "on" or "off" state of the speaker is 8-bit mu-law encoded audio data (regular .au file). Itis reduced to 1-bit, and this is then used to turn the speaker on or off, depending on what's left of the 8-bit after running it through a simple filter function.
files:
sound.c is a little different from the other sample programs, in that it is not scheduled (and thus managed by the realtime scheduler), but is simply assigned as the interrupt handler of interrupt 8. This is done for performance reasons, but is restricted to a one-task situation. init module will save the CMOS settings, then assign the "sound" routine as the interrupt handler for interrupt 8, program the CMOS clock to generate interrupts at 8KHz and then exit. So this is basically what you would do on a DOS box to run a routine as an interrupt service routine directly. This interrupt service routine then reads the data in from the fifo and filters it from 8-bit to 1-bit. filter() then activates and deactivates the speaker according to the filter output. cleanup_module will reset the state of the CMOS clock, remove the fifo and free the RTC interrupt.
usage:
Parallel Port -- examples/parallel:
Files in this directory are for playing with the parallel port and rtlinux. To use them all you will need a standard PLIP-cable. For details on this type of cable, please check PLIP-pinout.
Note:
The values of LPT are not defined via common.h for read_lpt.c and lpt_irq.c, since I assume that you are running these two executables on the second realtime or non-realtime box. You might have to recompile on that second box manually if the installation is not the same (especially the libs). To manually compile read_lpt.c and lpt_irq.c, use:
Note:
To compile these programs for MiniRTL you must take the lib restrictions into account. That is, if you installed the glibc-2.0.7 package, then you need to issue the compile command as:
(If you install the glibc2.0.7 libs in a different directory you need to pass that location.)
files:
rt_irq_gen.c: wait for an input on the parallel port. This polls the parrallel port in a busy-wait loop until something occurs. It will "freeze" your rtlinux box until something comes in or it is timed out. If you remove the timeout in rt_irq_gen.c then your rtlinux box will only stay "alive" as long as you run the lpt_irq program on your second linux box, connecting it via a plip-cable to your rtlinux box.
sched_toggle.c: This toggles the pins D0-D7 of parallel-port. With an oscilloscope you can directly measure the scheduling jitter of RTLinux! With the program read_lpt running on a second linux box (rt or non-rt), you can watch the pins toggle (but you will get no timings).
rectangle.c: The name says it. This is two threads. One sets D0-D7 high, the other sets it low.
read_lpt.c: This is a small program that will simply read the status pins of the parallel-port in a loop. You can use this instead of an oscilloscope just to see what is going on. To do so, start this program on a second Linux pc and connect the rtlinux box via standard PLIP-cable. (They are not expensive, and it's not worth making your own -- if you connect wrong pins you can damage your parallel port.)
lpt_irq.c: This pools for ACK to go low on LPT and then toggles pins D0-D7 to produce an ACK. Again, this runs on the second linux box and is connected to the rtlinux box via PLIP-cable.
Simple Realtime Multitasking -- examples/frank:
files:
Accessing memory form both rt and non-rt -- examples/mmap:
files:
Process synchronization with mutex -- examples/mutex:
files:
Making changes in MiniRTL:
Note:
if your kernel does not support msdos and loop, then compile it in or build the modules and insert them now (insmod fat, insmod msdos and insmod loop). (Alternatively, if you don't have root-privileges on the box you are working on, you can ask your admin to produce a empty 1.44MB msdos image and permanently loopmount it in the system so that you can build your images there.)
Create a mountpoint:
Mount the image file as follows. (A diskimage can be mounted just like a device even though it is a file on your harddrive.)
df should now show you something like the following:
Filesystem 1024-blocks Used Available Capacity Mounted on /dev/hda1 127918 119715 1377 99% / /dev/hda5 67706 33701 30392 53% /var /dev/hda6 67706 11174 52919 17% /tmp /dev/hda7 3555552 3317142 54468 98% /usr /usr/minilinux/minirtl.img 1423 1378 45 97% /rtlimg
Now you can copy the compressed images of the minirtl system to a local file:
Uncompress:
And run tar:
Now, ls in /usr/minilinux/etc/ will show you:
This etc is now the /etc of the minirtl system, and modifications can be made to the files here ( /usr/minilinux/etc/etc ). Remove the etc.tar -- you will not need it and when compressing your changes you don't want the old etc.tar to go into the image. The tar unpacked it in ./ and not in any subdirectory like a proper tar archive should, but this is necessary in this case because of the bootprocess of the minilinux system, where the tar must create the directories in / !
Network settings:
The main changes will be to network.conf. It sets up the IP/netmask/name/gw/etc.:
/etc/network.conf
Naturally yo u need to change /etc/resolv.conf, /etc/hosts, /etc/hostname, etc., but those are standard files and syntax.
If you test the MiniRTL system on a PC that is in your network and you give it a different IP from that which the PC regularly has, you might encounter a problem due to ARP caching. If you can't reach the MiniRTL system from another PC but you can ping to the outer world from the MiniRTL system, then you should manually clear your arp cache; then you should be set. If you have an environment with statically set IP/MAC addresses on your switch, then you MUST give the MiniRTL system a IP address that the PC you are testing on is known for,or reset the switch (which is probably not a very good way to make your network admin an rtlinux fan :).
Putting it back on disk:
Changing root's password (good idea...):
The minirtl.img has a rootpassword set to nopasswd, and it is probably a good idea to change this. To do so, you need to modify /usr/minilinux/etc/etc/shadow. Simply copy the crypt string from your user account in /etc/shadow into this file (for access to /etc/shadows you will need root privileges, though). You need root-access via network if your box has no local display -- there is no su or sudo on the disk (!). So, you need root access, or you need to make insmod setuid 0, or you will not be able to do much. No security stuff has been done yet -- that's the next step for this system. If security is of concern to you, disable telnet access completely (or inetd for that matter) and only use ssh/scp to access the MiniRTL system and transfer files.
There is one non-privileged account called rtadmin on the MiniRTL disk. It also has the password "nopasswd". There is currently no su available on MiniRTL, so to do more then check status you must login as root. Alternatively, you could set lsmod/rmmod and the like toSUID-root, but then you might as well log in as root.
Note:
ssh and scp reduce but don't eliminate the security problem. Since the key on MiniRTL is not saved before reboot, it will generate a new key on every reboot and thus send this key on first ssh/scp connection to the MiniRTL system. This means it potentially could be sniffed.
Now put it back on the disk (as above).
Changing the modules loaded at bootup:
The default MiniRTL system is built for a 3c509 or NE2000(ISA) NIC and has no special hardware support other than that. (I don't know exactly which image has which card set.) To set up the system for your hardware you need to make changes in the modules.rtl package only (unless you need the modules to access your bootmedia.
Compile the modules for your network card, scsi controller or whatever hardware you intend to access from MiniRTL:
(They all are in one directory , so it's a little messy.)
Now modify the module settings and parameters:
Edit the file modules. It simply contains a list of modules to be inserted at boot time.
/usr/minilinux/modules/etc/modules
And put it back on disk.
That should be all you need to get the system up and running. The rest is compiling and putting the realtime modules on the disk, and insmoding them after bootup. There is no depmod or modprobe on the disk, so you need to insmod in the correct order! The coolest thing about MiniRTL is that if you freeze the box with a bad module and must reset the box, NO FSCK is needed :)
Modifying the web-server:
The thttpd is a separate bundle (thttpd.rtl). If you would like to modify the server content, then boot the default server, modify any *.html or cgi-script you want changed, and upload it via tftp (see below). Once everything is in place, copy the thttpd.rtl to thttp.tar.gz on the harddisk of your linux-desktop-pc and unpack it. (I will assume you choose a directory named thtppd). The source for the server is in thttpd/var/www/. Change anything in here and then drop it back to the loop mounted image as described above. There is a brute force loop-script in thttpd/var/www/cgi-bin that will update the content of sys.html every 10 seconds. This is started by the thttpd initialization script in thttpd/etc/init.d/thttpd , you can modify the script. Simply comment out its launch to disable it. Real cgi-bin support is available now too, but the scripts must be written as ash-script -- NOT BASH! Even though there is a link bash->ash, this is only because often you have /bin/bash in /etc/passwd. This link is to prevent login-denial due to a missing shell. Writing ash-scripts is not difficult, but the syntax differences to bash are sometimes easy to overlook. It's similar, but not the same (which is probably harder than writing a completely different syntax).
Uploading via tftp:
Tftp is not very frequently used, so I assume most readers are not familiar with using it. This is only here for convenience, as it's not really MiniRTL related. Once you are up and running on the network, you can upload/download files to the MiniRTL system via tftp. To do so, first touch the files on the MiniRTL box first and give them world read/write permission (or tftpd will not write them). Then you simply:
Tftp's home directory is /tmp, so all uploaded files land there by default. Log into the system via telnet, and get the uploaded file from the /tmp directory. The tftp upload uses no authentication other than that the filename must be known and the perms set appropriately on /tmp, so I would not recommend using tftp on a site connected to any real network.
To at least reduce these security problems, uploads via scp are now supported (miniRTL2.X.img). One problem that occurs is that MiniRTL doesn't save the encryption seed at shutdown like a normal system would, so it will have a new public key after every reboot! This will make ssh/scp mutter about a key change. If you are using the public key, you simply say "yes" when ssh/scp asks you, or you remove the entry for the MiniRTL system in $HOME/.ssh/. It must be warned again that security has not yet been properly targeted in the current MiniRTL distributions. This is THE next big subject for embedded systems though, so this will follow.
mini_httpd (thttpd):
There is a simple sample script (endless loop) in /var/www/cgi-bin, which is a very brute force method of producing status reports, but it may be useful. Besides this script, cgi-bin support now works, so you can query status and/or execute commands via cgi-bin scripts (also supplied in /var/www/cgi-bin). They are not too wild, but I guess they show what can be done and how it works in a sufficient manner. This is glibc 2.0.7, though, and there are numerous known security problems with this lib. Most are related to buffer overflows, so these scripts need to be designed carefully if they should be secure.
ide-disks as modules:
This is just noted here because probably most Linux users don't compile ide support as modules, so using them mightnot be so common. You need three modules for loadable ide-disk support:
None of them should need any special arguments as long as you use "standard" ide port/irq settings. After inserting ide support, you need to add the filesystem support if the disk is not minix or dos (which are already in the kernel at boot-time). Ide-modules and filesystem modules can be found in the modules directories on the ftp-sites, and either uploaded via tftp/scp or put on the disk instead of something else. You will not be able to add them on without removing something else, though.
Please take into account that as long as you don't insert any hard-disk drivers, MiniRTL will not touch your harddisks and there is no risk of damaging data on them, but as soon as you insert harddisk support, and access the partitions on them, this can potentially lead to problems.