Linux: Development for ARM modules

Cross-Compilation

In the embedded world, software development is almost always done on a system other than the one on which the software will eventually run. It is common nowadays to use x86 desktop workstations to develop software for ARM-based embedded products. Such an approach requires cross-compilation tools and environment.

Building a cross compiler became recently relatively simple thanks to such tools as crosstool and crosstool-ng. But working cross-compiler is not enough. Modern applications rely on variety of libraries, frameworks and utilities. All these has to be cross-compiled also. Most Linux and Open Source software projects use a configure script to configure their software for compilation. Configure is meant to ease configuring the software for compilation and its default assumption is that the software will be run in the same environment and processor in which it was compiled and run from the place in which it was installed. The second problem of many configure scripts, is that most application developers don't take cross-compilation into account, so configure ends up cross-compiling test code and trying to run it on the build host, which breaks the configuration. Thus, cross compilation tools are not enough for embedded application development and complete cross-development environment is required.

Nowadays, there are several popular cross-development environments: Scratchbox, OpenEmbedded, it's spin-off Poky Linux, and Embedded Linux Development Kit.

Scratchbox

Scratchbox is a chrooted cross-compilation environment. This means one has to log into the Scratchbox like one would log into a real machine. Once you logged in, you develop your application as if you were developing it on native ARM machine, but with computational power of modern x86 workstation.

OpenEmbedded

OpenEmbedded is a development environment allowing users to target a wide variety of devices and configurations. OpenEmbedded enables you to build an entire Linux distribution optimized for specific device needs. Additionally, it includes devshell addon that provides you with an interactive shell with all the appropriate variables set for cross-compiling.

Poky Linux

Poky is an embedded Linux build system, distribution and developer environment which builds upon OpenEmbedded technologies. It is less generic than OpenEmbedded, but on the other side, you may find it more tuned and optimized for such devices as PDAs and smartphones.

ELDK

Embedded Linux Development Kit (ELDK) is a collection of GNU cross development tools, such as compilers, binutils, gdb, etc., and a number of pre-built target tools and libraries necessary to provide some functionality on the target system. It is not full-featured cross-development environment like Scratchbox, OpenEmbedded, or Poky, but it is known to have stable and reliable cross compilation tools.

Chroot with QEMU

Cross-compilation is the fastest way to build applications for ARM platforms. However, cross-compilation environment setup is relatively difficult even with advanced frameworks such as Scratchbox and OpenEmbedded. QEMU emulator allows development of ARM applications on x86 and x86-64 desktop system without setup of cross-compilation environment. It is possible to use statically build QEMU and Linux kernel binfmt capabilities to chroot into ARM root filesystem on the host workstation.

CompuLab provides minimal Debian filesystem that can be used as a basis for custom filesystem creation.

Binfmt_misc

The binfmt_misc Linux kernel feature allows running of different binary formats provided existence of appropriate interpreter. In conjunction with statically built qemu-arm or qemu-armeb the binfmt_misc provides ability to run ARM binaries on x86 systems.

Most modern Linux distributions enable the binfmt_misc support by default

The binfmt_misc should be enabled in the Linux kernel configuration in "Executable file formats / Emulations" menu:

.config - Linux Kernel v2.6.27-gentoo-r8 Configuration
──────────────────────────────────────────────────────────────────────────────
 ┌───────────────── Executable file formats / Emulations ──────────────────┐
 │  Arrow keys navigate the menu.  <Enter> selects submenus --->.          │
 │  Highlighted letters are hotkeys.  Pressing <Y> includes, <N> excludes, │
 │  <M> modularizes features.  Press <Esc><Esc> to exit, <?> for Help, </> │
 │  for Search.  Legend: [*] built-in  [ ] excluded  <M> module  < >       │
 │ ┌─────────────────────────────────────────────────────────────────────┐ │
 │ │    [*] Kernel support for ELF binaries                              │ │
 │ │    <M> Kernel support for MISC binaries                             │ │
 │ │    [*] IA32 Emulation                                               │ │
 │ │    < >   IA32 a.out support                                         │ │
 │ │                                                                     │ │
 │ │                                                                     │ │
 │ │                                                                     │ │
 │ │                                                                     │ │
 │ │                                                                     │ │
 │ │                                                                     │ │
 │ └─────────────────────────────────────────────────────────────────────┘ │
 ├─────────────────────────────────────────────────────────────────────────┤
 │                    <Select>    < Exit >    < Help >                     │
 └─────────────────────────────────────────────────────────────────────────┘

QEMU

Statically build QEMU ARM emulator should reside in the target (ARM) filesystem to allow invocation of ARM binaries on x86 system using binfmt_misc.

Ubuntu Karmic Koala, Debian Sid and Squeeze and probably other Linux distributions provide pre-built packages for static QEMU

If the binary package is not available in your distribution, building static version of QEMU is straightforward:

wget http://download.savannah.gnu.org/releases/qemu/qemu-0.11.1.tar.gz
tar xzf qemu-0.11.1.tar.gz
cd qemu-0.11.1
./configure --static --target-list="armeb-linux-user arm-linux-user"
make
cp arm-linux-user/qemu-arm armeb-linux-user/qemu-armeb /path/to/target/rootfs/user/local/bin

qemu-arm and qemu-armeb have to be registered with binfmt_misc as ARM binary interpreters:

echo ":qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\
\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:\
/usr/local/bin/qemu-arm:" > /proc/sys/fs/binfmt_misc/register
echo ":qemu-armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\
\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:\
/usr/local/bin/qemu-armeb:" > /proc/sys/fs/binfmt_misc/register

After the above setup is complete it is possible to execute

chroot /path/to/target/rootfs /bin/sh

and use the chroot'ed filesystem for application development

Networked Root Filesystem

It is convenient to use the same filesystem on host workstation and target device. Because the filesystem exists on the workstation, it is easy to use it for cross-development. And because the workspace is mounted on the target device also, we can test code developed on the host by running it on the target device.

The host workstation need to have NFS server packages installed. The package name depends on the distribution you use:

Ubuntu/Debian

apt-get install nfs-kernel-server nfs-common portmap

Fedora Core

yum install nfs-utils

Gentoo

emerge -av nfs-utils

After the NFS tools are installed on the host workstation, you can use images/Angstrom-image.tar.bz2 from Linux package for CM-T3530, images/angstrom-em-x270.tar.bz2 from EM-X270 Linux package and debian-image.tar.bz2 from CM-X270 Linux package, or CM-X300 Linux package for networked root filesystem. Extract the filesystem archive to some directory:

mkdir /home/development/module/rootfs
cd /home/development/module/rootfs

For CM-T3530:

tar xjf /path/to/cm-t3530-package/images/Angstrom-image.tar.bz2

For EM-X270:

tar xjf /path/to/em-x270-package/images/angstrom-em-x270.tar.bz2

For CM-X300:

tar xjf /path/to/cm-x300-package/images/debian-image.tar.bz2

And for CM-X270:

tar xjf /path/to/cm-x270-package/debian-image.tar.bz2

Afterwards add this directory to /etc/exports:

/home/development/module/rootfs \
        192.168.0.0/255.255.255.0(no_root_squash,sync,rw)

and make the NFS server export this directory:

/usr/sbin/exportfs -a

Linux kernel has Root on NFS feature, allowing mounting root filesystem form a server rather than from local storage device. Default kernels shipped by CompuLab have this feature built in. If you are planing to use custom kernel you can enable Root on NFS as follows: in the kernel configuration choose "Networking -> Networking Options" menu and enable "IP: kernel level autoconfiguration"

  +-------------------------- Networking options ---------------------------+
  |  Arrow keys navigate the menu.  <Enter> selects submenus --->.          |
  |  Highlighted letters are hotkeys.  Pressing <Y> includes, <N> excludes, |
  |  <M> modularizes features.  Press <Esc><Esc> to exit, <|> for Help, </> |
  |  for Search.  Legend: [*] built-in  [ ] excluded  <M> module  < >       |
  | +---------------------------------------------------------------------+ |
  | |    [ ] Network packet debugging                                     | |
  | |    <*> Packet socket                                                | |
  | |    [ ]   Packet socket: mmapped IO                                  | |
  | |    <*> Unix domain sockets                                          | |
  | |    < > Transformation user configuration interface                  | |
  | |    [ ] Transformation sub policy support (EXPERIMENTAL)             | |
  | |    < > PF_KEY sockets                                               | |
  | |    [*] TCP/IP networking                                            | |
  | |    [ ]   IP: multicasting                                           | |
  | |    [ ]   IP: advanced router                                        | |
  | |    [*]   IP: kernel level autoconfiguration                         | |
  | |    [*]     IP: DHCP support (NEW)                                   | |
  | |    [*]     IP: BOOTP support (NEW)                                  | |
  | |    [ ]     IP: RARP support (NEW)                                   | |
  | |    < >   IP: tunneling                                              | |
  | |    < >   IP: GRE tunnels over IP                                    | |
  | |    [ ]   IP: ARP daemon support (EXPERIMENTAL)                      | |
  | |    [ ]   IP: TCP syncookie support (disabled per default)           | |
  | +----v(+)-------------------------------------------------------------+ |
  +-------------------------------------------------------------------------+
  |                    <Select>    < Exit >    < Help >                     |
  +-------------------------------------------------------------------------+

Afterwards choose "File Systems -> Network File Systems" menu and enable NFS options:

  +------------------------- Network File Systems --------------------------+
  |  Arrow keys navigate the menu.  <Enter> selects submenus --->.          |
  |  Highlighted letters are hotkeys.  Pressing <Y> includes, <N> excludes, |
  |  <M> modularizes features.  Press <Esc><Esc> to exit, <|> for Help, </> |
  |  for Search.  Legend: [*] built-in  [ ] excluded  <M> module  < >       |
  | +---------------------------------------------------------------------+ |
  | |    <*> NFS file system support                                      | |
  | |    [*]   Provide NFSv3 client support                               | |
  | |    [ ]     Provide client support for the NFSv3 ACL protocol extensi| |
  | |    [ ]   Provide NFSv4 client support (EXPERIMENTAL)                | |
  | |    [*]   Allow direct I/O on NFS files                              | |
  | |    <M> NFS server support                                           | |
  | |    [*]   Provide NFSv3 server support                               | |
  | |    [ ]     Provide server support for the NFSv3 ACL protocol extensi| |
  | |    [ ]     Provide NFSv4 server support (EXPERIMENTAL)              | |
  | |    [*]   Provide NFS server over TCP support                        | |
  │ │    [*]   Root file system on NFS                                    │ │
  | +----v(+)-------------------------------------------------------------+ |
  +-------------------------------------------------------------------------+
  |                    <Select>    < Exit >    < Help >                     |
  +-------------------------------------------------------------------------+

After the kernel is built it can be loaded with

root=/dev/nfs nfsroot=<serverip>:/path/to/nfsroot

options and access the filesystem on the host workstation and used for software development. More information about how to use NFS root can be found in Documentation/filesystems/nfsroot.txt in the Linux kernel source tree.

If you use Angstrom Linux distribution, you need to make sure that startup scripts do not change primary Ethernet interface configuration as set up by the Linux kernel. Possible way to achieve this would be to set eth0 configuration to 'manual' iface eth0 inet manual in /etc/network/interfaces on the target filesystem

Building the Linux kernel for XScale modules

There may be cases when default kernel for device does not fit your system requirements and you would like to build your own kernel. Both EM-X270 and CM-X270 platforms are supported by vanilla kernel starting from version 2.6.23 and 2.6.24 respectively. Yet, the support present there is incomplete and does not include several features. This features can be enabled if you apply EM-X270 (or CM-X270) patchset to vanilla kernel and make proper kernel configuration.

Keep in mind that to build Linux kernel for CM-X270 and EM-X270 you need cross-compilation toolchain and for EM-X270 you will also need mkimage utility from U-Boot.

Below is a brief description of Linux kernel build process for CompuLab Xscale modules along with an example commands sequence.

• Download Linux kernel source from preferred mirror
• Unpack it to some directory
• Apply EM-X270 or CM-X270 patchset
• Set environment variables ARCH, and CROSS_COMPILE so that kernel build system cross compile the kernel
• Build the kernel, and modules
• Install the modules

Building the Linux kernel for EM-X270

mkdir /home/development/em-x270/kernel
cd /home/development/em-x270/kernel
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.tar.bz2
tar xjf linux-2.6.23.tar.bz2
cd linux-2.6.23
cat /path/to/x270em-linux-img/kernel/linux-2.6.23-em-x270.patch | patch -p1
export ARCH=arm
export CROSS_COMPILE=arm-linux-
make em_x270_defconfig
make menuconfig
make && make uImage && \
     INSTALL_MOD_PATH=/home/development/em-x270/rootfs make modules_install

In the above example, the Linux kernel image (uImage) will be created in /home/development/em-x270/kernel/linux-2.6.23/arch/arm/boot, and the loadable kernel modules will be installed into the /home/development/em-x270/rootfs/lib/modules directory.

Now, if you boot EM-X270 with newly created kernel image and with networked root filesysem at /home/development/em-x270/rootfs/ the system will be able to properly load kernel modules and you avoid modules versioning problems.

Building the Linux kernel for CM-X270

mkdir /home/development/cm-x270/kernel
cd /home/development/cm-x270/kernel
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.24.tar.bz2
tar xjf linux-2.6.24.tar.bz2
cd linux-2.6.24
cat /path/to/x270cm-linux-drv/2.6.24/linux-2.6.24-cm-x270.patch | patch -p1
export ARCH=arm
export CROSS_COMPILE=arm-linux-
make cm_x270_defconfig
make menuconfig
make && \
     INSTALL_MOD_PATH=/home/development/cm-x270/rootfs make modules_install

In the above example, the Linux kernel image (zImage) will be created in /home/development/cm-x270/kernel/linux-2.6.24/arch/arm/boot, and the loadable kernel modules will be installed into the /home/development/cm-x270/rootfs/lib/modules directory.

Now, if you boot CM-X270 with newly created kernel image and with networked root filesysem at /home/development/cm-x270/rootfs/ the system will be able to properly load kernel modules and you avoid modules versioning problems.

Creating NAND flash images

MTD utils

For NAND flash images management, you will need to have MTD utilities installed on your host workstation and on any root filesystem, used to boot the CompuLab XScale modules. MTD utilities are available as binary package in Debian/Ubuntu and probably other distributions, or can be downloaded directly from MTD website or MTD utils repository.

If you want to use mtd-utils package from MTD utils repository, you may need to install some development libraries as described in MTD's official FAQ.

After the MTD utilities are set up, proceed to UBIFS images or JFFS2 images creation guides.

UBIFS images

This guide describes procedure for creating default UBIFS and UBI volume images for CM-X300 with 512MB NAND flash. For installation instructions, please see the "Getting started with Linux on CM-X300" article. For more information and custom image creation, please see the UBI/UBIFS official documentation and FAQ links below.

UBIFS image creation could be split into two parts: UBIFS filesystem image creation and UBI volume image creation. UBIFS filesystem image can be used with ubiupdatevol utility for existing UBI volume updates. UBI volume image can be put onto the MTD partition using the ubiformat utility.

UBIFS filesystem image creation is done with mkfs.ubifs utility. The resulting image file is passed to ubinize utility along with image.cfg configuration file. The ubinize utility converts the UBIFS filesystem image to UBI volume image.

Typical image.cfg file contains:

[ubi-image]
mode=ubi
image=tmp_image.ubifs
vol_id=0
vol_size=450MiB
vol_type=dynamic
vol_name=ubi-image
vol_flags=autoresize

The following command sequence is used to create UBI volume image for CM-X300 module:

mkfs.ubifs --root=/home/development/cm-x300/rootfs --min-io-size=2048 --leb-size=126976 --max-leb-cnt=4094 \
           --output=/path/to/tmp_image.ubifs
ubinize --output=image.ubi --min-io-size=2048 --vid-hdr-offset=2048 --peb-size=128KiB image.cfg

The image.ubi is ready to be transferred to the device NAND flash using ubiformat command.

min-io-size parameter has to be the NAND page size.

JFFS2 images

The following command sequence is used to create JFFS2 filesystem image:

mkfs.jffs2 --no-cleanmarkers --pad --eraseblock=0x20000 --root=/home/development/cm-x270/rootfs --output=/path/to/tmp_image.jffs2
sumtool --no-cleanmarkers --pad --eraseblock=0x20000 --input=/path/to/tmp_image.jffs2 --output=/path/to/image.jffs2

Here eraseblock parameter may vary depending on your NAND flash and the layer used, for example: default CM-X300 kernel configuration enables UBI volumes on the NAND flash. To use the JFFS2 image on top of UBI volume the eraseblock size parameter should be set to 0x1f000. More information on this could be found at JFFS2 images over UBI layer page.

The image.jffs2 is ready to be transferred to the device NAND flash.

CM-X270 Legacy images

cd /path/to/new/image/

# Create temporary directories
mkdir –p /tmp/destimg

# Create a placeholder for the new image
# Make image ~500 MB to fit it to NAND devices with bad blocks
dd if=/dev/zero of=512mb.img count=500 bs=1024k

# Format it using ext3 file system
mke2fs –F -j 512mb.img

# mount the new (empty) image
mount –t ext3 -o loop 512mb.img /tmp/destimg

# copy files preserving device nodes, ownership, permissions and
# symbolic links
cp -a /home/development/cm-x270/rootfs/* /tmp/destimg

# unmount the image
umount /tmp/destimg

# flush disk caches to the physical storage 
sync

# clean up
rm -rf /tmp/destimg 

After running this script, the image /path/to/new/image/512mb.img will contain a file system suitable for a 512 MB NAND flash. This image can be installed on the CM-X270 using the procedure described in the Legacy image installation section of "Getting started with Linux on CM-X270" article.

See also

• Getting started with Linux on CM-X270
• Getting started with Linux on CM-X300
• Getting started with Linux on EM-X270
• Linux Kernel for CM-X270
• Linux Kernel for CM-X300
• Linux kernel for Exeda
• Angstrom Linux for EM-X270
• Debian Linux for CM-X270
• Debian Linux for CM-X300
• Linux MTD drivers on CM-X270
• JFFS2 images over UBI layer
• U-Boot images
• UBI official documentation
• UBI official FAQ and HOWTO
• UBIFS official documentation
• UBIFS official FAQ and HOWTO