Embedded Linux update
Why
I don’t think I need to explain why possibility of system update is important. Nearly every day I read about another device filled with security holes.
While bugs are inevitable – every system has some. It is extremely important to be able to fix them.
While some (most maybe) devices are containing full blown Linux distribution, with pacman/apt support, there are also devices working on minimal distribution built using Yocto or Buildroot. These distributions are working more close to the metal, without resources required to run decent package manager.
Update process itself is little different comparing to regular Linux machine. User usually cannot supervise it. He cant make decisions.
Firmware update contents are under full control of device maker. It is the manufacturer who decides what will be updated.
It is devices makers responsibility to ensure that update won’t „brick” the device.
For who
I expect intermediate knowledge about Linux and Yocto build system.
I am using Yocto, but solution can be used on any NAND-based embedded Linux project.
Hardware
I am using custom board based on Atmel’s SAMA5. You can use this article to build update system for any U-boot/NAND based board.
I will port this solution to NXP SOM’s devices in future.
Definitions
Yocto
Yocto is a widely used system for building complete embedded Linux images.
Unfortunately, learning curve is extremely steep. While documentation is rich and detailed, there are number of „gotchas”, which are making development process painful.
Device tree
ARM-based devices does not have a BIOS. There are no system which could tell Linux Kernel about hardware.
That is why Device Tree was introduced.
Device tree is a binary encoded hardware description. Linux kernel parse and analyze device tree.
Device tree usually is loaded in to RAM by bootloader, then bootloader loads and starts Linux Kernel.
NAND flash
While most of today using embedded system are using miniSD card or some kind of eMMC chip. There are some using NAND flash.
MiniSD and eMMC can be described as NAND with additional controller – specialized chip that can balance NAND flash sectors wearing. Also controller is able to detect bad sectors.
Because NAND flash has some special properties – ordinary file system cannot be used on them, that is why specialized file systems was invented (UBI, YAFFS and more)
NAND partitions
NAND flash can be divided into to partitions, similar to hard disk. But unlike hard disk – NAND does not contain partition table.
NAND partitions are defined using Device Tree or command line parameters.
UBI and UBIFS
UBIFS is relatively new file system. Designed to run on top of UBI image.
UBI (unsorted block image) is a layer placed on top of NAND partition.
UBI can be described as image that can contain many volumes.
UBI is much safer then raw NAND flash. It balances writes, and is able to detect bad sectors and restore its data. That is why it is safe to store volume table inside UBI.
Bootloader
Bootloader is a program thats job is to start operating system.
Bootloaders can be divided in to:
- First stage – those one initializes hardware and loads second stage bootloader.
- Second stage, these job is to start operating system.
Bootstrap
Bootstrap is a first stage bootloader made by Atmel.
It’s job is to initialize hardware, and load second stage bootloader.
U-boot
U-boot is a popular second stage bootloader.
Many hardware vendors are creating patches, to U-boot could support their chips.
U-boot has command-line interface. User can modify default behavior by modifying environment variables.
Environment variables are stored in NAND partitions. Usually these partitions are redundant. Even if one is damaged, second one should allow device boot.
U-boot default environment variables, location of environment variables partitions(s), NAND partitions layout etc. are compiled-in into U-boot binary.
In simplest scenario, U-boot loads kernel and device tree from NAND partitions into memory. Then starts kernel.
FIT
FIT is a single-file format supported by U-boot that can be used to store both kernel and device tree.
FIT also can be signed, U-boot will not run image without proper signature (optional).
Default Atmel SAMA5D3 NAND layout
Lets analyze Atmel SAMA5D3 default boot process.
Here is NAND layout:
+--------------------------------------+ | Bootstrap | +--------------------------------------+ | U-boot | +--------------------------------------+ | U-boot environment variables 1 | +--------------------------------------+ | U-boot environment variables 2 | +--------------------------------------+ | Device tree | +--------------------------------------+ | Kernel | +--------------------------------------+ | UBI/UBIFS rootfs | +--------------------------------------+
Boot process
* Bootstrap initializes hardware
* Bootstrap loads and starts U-boot.
* U-boot starts,
* Loads its environments variables from NAND partition (or uses defaults, if no valid environment variables partitions has been found).
* Executes /bootcmd/ environment variable commands which loads device tree and kernel into RAM.
* U-boot starts Linux kernel using comand-line parameters defined in /bootargs/ environment variable
Goal
My goal is to create redundant banks for kernel and root.
The bank is a pair of volumes. One for kernel, second for root.
There are two banks. Only one is in use. The other one is used during update process.
Update script writes kernel/rootfs images in to free bank volumes.
Then, U-Boot environment variables are modified, so on the next reboot, bootloader will start Linux using different bank.
Device configuration after modifications
This is how NAND/UBI partitions and volumes is going to look.
+--------------------------------------+ | Bootstrap | +--------------------------------------+ | U-boot | +--------------------------------------+ | U-boot environment variables 1 | +--------------------------------------+ | U-boot environment variables 2 | +--------------------------------------+ | UBI | | +----------------------------------+ | | | Kernel 1 (FIT) (Bank 1) | | | +----------------------------------+ | | | Kernel 2 (FIT) (Bank 2) | | | +----------------------------------+ | | | Rootfs 1 (UBIFS) (Bank 1) | | | +----------------------------------+ | | | Rootfs 2 (UBIFS) (Bank 2) | | | +----------------------------------+ | | | Data (UBIFS) | | | +----------------------------------+ | +--------------------------------------+
Modifications required
Overview of modifications required to implement
NAND
NAND layout has to be changed.
The problem is – there are more then one place where NAND partitions are defined. All of those definitions has to modified.
U-boot
U-Boot requires full information about NAND flash partitions layout. Also it has to know UBI volumes labels for kernel and rootfs.
Additionally, U-Boot has to be configured to support FIT files.
All modifications requires rather serious U-boot code modifications.
I suggest to fork U-boot git repository, and make changes.
Changing git repository locations requires altering Yocto recipe behavior by creating append file (.bbappend)
UBI volumes
By default UBI image with single volume is created.
Image creation process is implemented in image_types.bbclass file. Unfortunately modifying behavior defined in .bbclass file is not as easy as .bbrecipe. Recipes have simple inheritance mechanism, classes are not. Luckily there is a hack we can use to get the work done.
To create UBI image containing five volumes original image_types.bbclass file has to be copied and changed.
UBI image should contain five volumes. Each volume is created using UBIFS image file or raw image.
Kernel
Kernel can remain unchanged. But Yocto has to be instructed to generate FIT file containing both Kernel and device tree.
Rootfs
Rootfs should contain additional software packages:
- UBI volumes manipulation
- NAND partitions manipulation
- U-Boot environment variables manipulation
These packages needs to be added to image,
DATA
Rootfs should be read only. Usually we need some kind of read/write storage. That is why another UBI volume is needed. DATA volume should contain UBIFS file system, and has to be mounted a boot time.
Bootstrap
Bootstrap may be left unchanged.
Lets get to work
Yocto configuration
Custom layer
To customize way some Yocto recipes work we need custom layer.
Lets call it meta-arek. Create folder named ‚meta-arek’ next to other layers in yocto directory.
You should know how to create ‚build’ directory.
Inside build/conf directory find, and open bblayers.conf file. Add path to meta-arek layer directory to BBLAYERS variable.
Path to meta-arek should be first on list. I’ll explain later why.
Link to Yocto documentation:
https://www.yoctoproject.org/documentation
UBI image and volumes
We need to change image creation process. The code that creates images can be found in meta/classes/image_types.bbclass file. Unfortunately we cant override behavior of .bbclass files. That is why I’m using little hack.
Copy file to meta-arek/classes directory. Because ‚meta-arek’ layer directory is higher then ‚meta’, it’s files will be preferred by bitbake.
Now, lets modify meta-arek/classes/image_types.bbclass file
multiubi_mkfs() {
local mkubifs_args="$1"
local ubinize_args="$2"
if [ -z "$3" ]; then
local vname=""
else
local vname="_$3"
fi
echo -n > ubinize${vname}.cfg
echo \[kernel1\] >> ubinize${vname}.cfg
echo mode=ubi >> ubinize${vname}.cfg
echo image=${DEPLOY_DIR_IMAGE}/fitImage >> ubinize${vname}.cfg
echo vol_id=1 >> ubinize${vname}.cfg
echo vol_type=dynamic >> ubinize${vname}.cfg
echo vol_name=kernel1 >> ubinize${vname}.cfg
echo vol_size=10MiB >> ubinize${vname}.cfg
echo \[kernel2\] >> ubinize${vname}.cfg
echo mode=ubi >> ubinize${vname}.cfg
echo image=${DEPLOY_DIR_IMAGE}/fitImage >> ubinize${vname}.cfg
echo vol_id=2 >> ubinize${vname}.cfg
echo vol_type=dynamic >> ubinize${vname}.cfg
echo vol_name=kernel2 >> ubinize${vname}.cfg
echo vol_size=10MiB >> ubinize${vname}.cfg
echo \[root1\] >> ubinize${vname}.cfg
echo mode=ubi >> ubinize${vname}.cfg
echo image=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${vname}.rootfs.ubifs >> ubinize${vname}.cfg
echo vol_id=3 >> ubinize${vname}.cfg
echo vol_type=dynamic >> ubinize${vname}.cfg
echo vol_name=root1 >> ubinize${vname}.cfg
echo vol_size=30MiB >> ubinize${vname}.cfg
echo \[root2\] >> ubinize${vname}.cfg
echo mode=ubi >> ubinize${vname}.cfg
echo image=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${vname}.rootfs.ubifs >> ubinize${vname}.cfg
echo vol_id=4 >> ubinize${vname}.cfg
echo vol_type=dynamic >> ubinize${vname}.cfg
echo vol_name=root2 >> ubinize${vname}.cfg
echo vol_size=30MiB >> ubinize${vname}.cfg
echo \[data\] >> ubinize${vname}.cfg
echo mode=ubi >> ubinize${vname}.cfg
echo vol_id=5 >> ubinize${vname}.cfg
echo image=${DEPLOY_DIR_IMAGE}/empty.ubifs >> ubinize${vname}.cfg
echo vol_type=dynamic >> ubinize${vname}.cfg
echo vol_name=data >> ubinize${vname}.cfg
echo vol_size=30MiB >> ubinize${vname}.cfg
echo vol_flags=autoresize >> ubinize${vname}.cfg
rm -rf ${DEPLOY_DIR_IMAGE}/empty/*
mkdir -p ${DEPLOY_DIR_IMAGE}/empty
mkfs.ubifs -r ${DEPLOY_DIR_IMAGE}/empty -o ${DEPLOY_DIR_IMAGE}/empty.ubifs ${mkubifs_args}
mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${vname}.rootfs.ubifs ${mkubifs_args}
ubinize -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${vname}.rootfs.ubi ${ubinize_args} ubinize${vname}.cfg
# Cleanup cfg file
mv ubinize${vname}.cfg ${DEPLOY_DIR_IMAGE}/
# Create own symlinks for 'named' volumes
if [ -n "$vname" ]; then
cd ${DEPLOY_DIR_IMAGE}
if [ -e ${IMAGE_NAME}${vname}.rootfs.ubifs ]; then
ln -sf ${IMAGE_NAME}${vname}.rootfs.ubifs \
${IMAGE_LINK_NAME}${vname}.ubifs
fi
if [ -e ${IMAGE_NAME}${vname}.rootfs.ubi ]; then
ln -sf ${IMAGE_NAME}${vname}.rootfs.ubi \
${IMAGE_LINK_NAME}${vname}.ubi
fi
cd -
fi
}
As you see, we have changed way file ubinize.cfg is generated. Ubinize.cfg file is used by ‚ubinize’ tool, which creating UBI image. Ubuinize.cfg contains definitions of UBI volumes, and some NAND specific configuration values.
U-boot
I spent couple of days, searching for simple solution to modify U-boot. I did not found any.
Atmel’s manual for U-boot setup is terribly outdated. So I have to found solution myself.
U-boot configuration is probably hardest part. We have to modify U-boot source files.
That is why I recommend to fork U-boot repository, make changes in forked version, then change repository URI using by Bitbake to fetch U-boot source code.
If you forked U-boot repository, we can start work.
Open arch/arm/mach-at91/Kconfig file.
add:
config TARGET_SAMA5D3_AREK
bool "SAMA5D3 Xplained board"
select CPU_V7
select SUPPORT_SPL
And, at the end of arch/arm/mach-at91/Kconfig file:
source "board/atmel/sama5d3_arek/Kconfig"
Create file board/atmel/sama5d3_arek/Kconfig
if TARGET_SAMA5D3_AREK
config SYS_BOARD
default "sama5d3_xplained"
config SYS_VENDOR
default "atmel"
config SYS_CONFIG_NAME
default "sama5d3_arek"
endif
Create file board/atmel/sama5d3_arek/MAINTAINERS
SAMA5D3_AREK BOARD M: Arek Marud <a.marud@post.pl> S: Maintained F: board/atmel/sama5d3_arek/ F: include/configs/sama5d3_arek.h F: configs/sama5d3_arek_defconfig F: configs/sama5d3_arek_defconfig
Create file configs/sama5d3_arek_defconfig
CONFIG_ARM=y CONFIG_ARCH_AT91=y CONFIG_TARGET_SAMA5D3_AREK=y CONFIG_SPL=y CONFIG_FIT=y CONFIG_SYS_EXTRA_OPTIONS="SAMA5D3,SYS_USE_NANDFLASH"
Create file include/configs/sama5d3_arek.h
#include "sama5d3_xplained.h" #define MTDIDS_DEFAULT "nand0=nand_flash" #define MTDPARTS_DEFAULT "mtdparts=nand_flash:256k(bootstrap)ro,512k(uboot)ro,256k(env1),256k(env2),-(sys)" #define CONFIG_BOOTARGS "ubi.mtd=4 root=ubi0:root1 rootfstype=ubifs console=ttyS0,115200 earlyprintk mtdparts=atmel_nand:256k(bootstrap)ro,512k(uboot)ro,256k(env1),256k(env2),-(sys)" #define CONFIG_BOOTCOMMAND "mtdparts default;ubi part sys;ubi read 0x22000000 kernel1;bootm 0x22000000#conf@1"
That’s it.
Please notice contents of file ‚include/configs/same5d3_arek.h’, MTDPARTS_DEFAULT describes NAND partitions layout for U-boot. CONFIG_BOOTARGS environment variable defines kernel command line parameters, and NAND partitions.
Finally CONFIG_BOOTCOMMAND variable stores command for kernel start.
U-boot Yocto changes
To make our changes work, we have to change git repository URI.
Create file ‚meta-arek/recipes-bsp/u-boot/u-boot-at91_git.bbappend’.
Add single line:
SRC_URI="git://gitserver.com/u-boot-custom.git.git;branch=master;protocol=http"
Of course you have to enter URI for your repository.
Machine
Default settings for image are stored in machine file. Let’s create one.
First we need to copy all required include files.
Copy files:
at91sam9.inc bootloaders.inc sama5d2.inc sama5d3.inc sama5d4.inc sama5.inc
to meta-arek/conf/machine/include directory.
Files can be found inside meta-atmel directory.
Create file meta-arek/conf/machine/mymachine.conf
require include/sama5d3.inc MACHINE_FEATURES = "kernel26 apm ext2 ext3 usbhost usbgadget camera ppp wifi iptables" # used by sysvinit_2 SERIAL_CONSOLES ?= "115200;ttyS0" ROOT_FLASH_SIZE = "256" IMAGE_FSTYPES += " ubi tar.gz" # NAND MKUBIFS_ARGS ?= " -e 0x1f000 -c 2048 -m 0x800 -x lzo" UBINIZE_ARGS ?= " -m 0x800 -p 0x20000 -s 2048" UBI_VOLNAME = "rootfs" UBOOT_MACHINE ?= "sama5d3_arek_config" UBOOT_ENTRYPOINT = "0x20008000" UBOOT_LOADADDRESS = "0x20008000" AT91BOOTSTRAP_MACHINE ?= "sama5d3_xplained" PREFERRED_PROVIDER_virtual/kernel = "linux-at91" PREFERRED_VERSION_linux-at91= "4.%" KERNEL_CLASSES += "kernel-fitimage" KERNEL_IMAGETYPE = "fitImage"
Last two lines forces FIT file creation. FIT file will be used during UBI image creation.
Test build
bitbake core-image-minimal
Directory build/tmp/deploy/images/mymachine should be populated
flash
It is time to flash files in to NAND flash memory.
Do to that, Atemel’s SAM-BA flashing tool is required. Download and unpack sam-ba. Find directory with sam-ba_64 file.
Create flash.tcl file:
global target
puts "=== Initialize the NAND access ==="
NANDFLASH::Init
puts "=== Erase the NAND access ==="
NANDFLASH::EraseAll
puts "=== Send SPL ==="
NANDFLASH::SendBootFileCmd "at91bootstrap.bin"
puts "=== Send u-boot.bin ==="
send_file {NandFlash} "u-boot.bin" 0x40000 0
puts "=== Send rootfs ==="
send_file {NandFlash} "rootfs.ubi" 0x00140000 0
puts "=== DONE ==="
Create flash.sh file:
#!/bin/bash ./sam-ba_64 /dev/ttyACM0 at91sama5d3x-xplained flash.tcl
Make flash.sh executable:
chmod u+x flash.sh
Connect your SAMA5 based board to USB port.
Remember to enable NAND access mode (JP5 jumper on SAMA5D3 Xplained).
Power the device (with jumper connected). Disconnect jumper after 3-4 seconds and start ‚flash.sh’ script. Wait for flashing process to finish.
Restart your device. Linux should boot.
Linux modifications
Now, our system has to have ability to update itself. That means that Linux needs to have access to:
* U-boot environment variables. These variables has to be changed to "switch" active kernel and rootfs.
* UBI volumes. Kernel and rootfs images will be copied in to them.
Lets start with U-Boot environment variables.
First, lets add „u-boot-fw-utils” package in to the rootfs. The package contains utilities, that allows to modify U-Boot environment variables.
local.config
IMAGE_INSTALL_APPEND += " u-boot-fw-utils"
The package „u-boot-fw-utils” is built using main U-Boot source repository. Because we changed U-Boot repository to different server, it is recommended to do this also for „u-boot-fw-utils” package.
To modify git repository address, create file u-boot-fw-utils_2015.07.bbappend in directory meta-arek/recipes-bsp/u-boot
SRCREV = "<git comit hash>" LIC_FILES_CHKSUM = "file://Licenses/README;md5=a2c678cfd4a4d97135585cad908541c6" SRC_URI="git://yourgit.server.com/repository.git;branch=master;protocol=http"
Unfortunately u-boot-fw-utils_2015.07.bbrecipe file is using SRCREV parameter. I did not found the way to „nullify” SRCREV in bbappend file. So, the value has to be changed each time some significant commit was pushed in to git repository.
What SRCREV does, is to force usage of specified commit instead the last one.
SRC_URI should be the same like u-boot-at91_git.bbappend file.
OK. So now we have tool to manipulate U-Boot environment variables installed. Now we need to configure it.
Program „u-boot-fw-utils” requires information about placement of U-Boot environment variables. Environment variables are stored in MTD partitions. Because we added MTD partitions layout information to the kernel command line, Kernel should create device files for each partition. Files are named /dev/mtd1 /dev/mtd2 and so on.
Configuration for u-boot-fw-utils is stored in /etc/fw_env.config file.
Here is valid configuration for MTD layout:
/dev/mtd2 0x0 0x20000 0x20000 1 /dev/mtd3 0x0 0x20000 0x20000 1
Default content for /etc/fw_env.config file is stored in U-Boot repository, it has to be changed.
Easiest way to do it, is to modify U-Boot repository. Find default file, and modify it.
Unfortunately because git repository was changed, SRCREV in u-boot-fw-utils_2015.07.bbappend file has to be updated. Change SRCREV value to valid commit hash.
UBI volumes manipulation
One of the greatest Linux strengths is the „everything is a file” philosophy. Each hard disk, partition on that disk has its own device file.
That rule applies also to UBI volumes – where kernel and rootfs are stored.
Essentially, during update process rootfs and kernel image files are copied in to UBI volume device file.
But UBI volumes are not ordinary Linux block device, ‚dd’ command cannot be used. We need tool for this task.
Package „mtd-utils-ubifs” contains everything we need.
To add „mtd-utils-ubifs” to image, add:
IMAGE_INSTALL_APPEND += " mtd-utils-ubifs"
to local.config
Now, command ubiupdatevol can be used. To copy kernel image file to /dev/ubi0_1 UBI volume device file try command:
ubiupdatevol /dev/ubi0_1 kernel
Now we have all tools we need to update Linux.
There are, however few scripts that needs to be created.
Create update package file
The easiest way to do that, is to create .tar.gz file containing rootfs and kernel files.
Unpack update package
I think best location is somewhere in /tmp directory.
Determine currently used UBI volumes.
This can be done by parsing U-Boot environment variables, or kernel command line parameters contained in /proc/ cmdline.
Overwrite UBI volumes
Using ubiupdatevol, copy contents of kernel and rootfs files in to UBI volumes.
Modify bootloader
Use fw_setenv command to modify bootloader environment variables.
Variable „bootcmd” contains kernel location.
Variable „bootargs” contains rootfs location.
Unfortunately we need to change two variables. That means that „switch” will not be an atomic operation.
Mount „data” volume
Rootfs contents should not be changed. Best solution would be read-only rootfs, I tried that, but number of problems that needs to be fixes is huge.
System settings directory – /etc/ can be mounted using unionfs over tmpfs volume. But some services (dropbear for example) are changing files in other directories.
Anyway, all changes made on rootfs will be lost after update. That is why we need a place where modified files (configuration data for example) will be stored. There is UBI volume named ‚data’ for that purpose.
We can:
* Mount contents of ‚data’ volume in /mnt/data directory
* Mount unionfs over /etc and /mnt/data/etc directories to merge them. All modifications made on /etc will be stored in ‚data’ volume. That means that all settings will remain unchanged after update process.
Unfortunately /etc/fstab can’t be used to mount UBI volume. To mount volume create script that will be executed during boot time, and mount volume ‚by hand’ using ‚mount’ command.
Firmware update file
Easiest way to make firmware update package, is to place files named rootfs and kernel inside tar.gz file.
Archive can be extracted in to /tmp directory (/tmp in Yocto minimal image is mounted using tmpfs, it is basically a ramdisk).
I suggest to sign package using OpenSSL private key. Add public key to your rootfs, and verify update file before performing any modifications.
Scripts
At least one script is required to perform update process.
Script should:
* Verify package file against public key (optionally)
* Extract update package archive
* Check rootfs and kernel files existence
* Check currently used UBI volumes (read U-Boot environment variables using fw_printenv command)
* Use ubiupdatevol to modify UBI volumes
* Use fw_setenv command to modify U-Boot environment variables, to use new kernel and rootfs locations.
Arkadiusz Marud 