4. Software Guide

This chapter provides instructions for compiling and deploying the BSP (Board Support Package) software to the Q7 module.

4.1. Architecture Overview

The BSP consists of several parts. They run on different parts of the CPU and each play their role in the boot process. Because the CPU contains cores running different instruction sets (ARMv6-M and ARMv8-A), two different compilers are needed. The table below list the parts and their instruction set.

BSP Part Architecture
Cortex-M0 power management firmware ARMv6-M
ATF (ARM Trusted Firmware) ARMv8-A
U-Boot bootloader ARMv8-A
The Linux kernel ARMv8-A
Debian user-space ARMv8-A

The next section explains how to install suitable cross-compilers for both instruction sets.

The section “Compiling Linux Applications” provides guidance for compiling user-space applications for the RK3399.

4.2. Prerequisites

You need a recent x86_64 Linux installation to run the cross-compiler on and at least 10GB of disk space. The cross-compiler requires libc version 2.2.5. Distributions shipping this version are, among others:

  • Ubuntu 16.04 “Xenial”
  • Debian 8 “Jessie”
  • Debian 9 “Stretch”

We recommend Debian 9 “Stretch” or Ubuntu 16.04 “Xenial”. Please install the following packages to set up the common build infrastructure:

sudo apt install device-tree-compiler u-boot-tools build-essential git bc debootstrap qemu-user-static libssl-dev

4.2.1. ARMv6-M Compiler

The “GNU Embedded Toolchain for ARM” is suitable for compiling the Cortex-M0 power management firmware. It can be downloaded from https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/downloads .

For this manual, it is assumed that version 6-2017-q1 is used. Direct link to the file:

https://developer.arm.com/-/media/Files/downloads/gnu-rm/6_1-2017q1/gcc-arm-none-eabi-6-2017-q1-update-linux.tar.bz2

Extract the tar.bz2 archive to /opt:

sudo tar -xf gcc-arm-none-eabi-6-2017-q1-update-linux.tar.bz2 -C /opt

4.2.2. ARMv8-A Compiler

The Linaro aarch64-linux-gnu toolchain is suitable for compiling all other parts of the BSP. It is also suitable for compiling user-space applications. You can download ready-to-use binaries from Linaro: https://releases.linaro.org/components/toolchain/binaries/6.3-2017.02/aarch64-linux-gnu/ .

Direct link to the file:

https://releases.linaro.org/components/toolchain/binaries/6.3-2017.02/aarch64-linux-gnu/gcc-linaro-6.3.1-2017.02-x86_64_aarch64-linux-gnu.tar.xz

Extract the tar.xz archive to /opt:

sudo tar -xf gcc-linaro-6.3.1-2017.02-x86_64_aarch64-linux-gnu.tar.xz -C /opt

4.3. Compile the Cortex-M0 power management firmware

The Cortex-M0 firmware runs inside a microcontroller embedded in the CPU IC. It implements power-management functionality and helpers (e.g. DRAM frequency switching support).

Set up environment variables to make use of the ARMv6-M compiler, then download the source code and compile:

export ARCH=arm64
export CROSS_COMPILE=/opt/gcc-arm-none-eabi-6-2017-q1-update/bin/arm-none-eabi-
git clone https://git.theobroma-systems.com/rk3399-cortex-m0.git
cd rk3399-cortex-m0
make
cd ..

4.3.1. Optional: Compile the cross-compiler

As an alternative to using a ready-made compiler, the firmware repository has a mechanism to compile the ARMv6-M-compiler as a part of the build process. This is called “internal toolchain”.

If you want to use the internal toolchain instead you will also need the following packages:

sudo apt install libssl-dev autoconf gperf bison flex texinfo help2man gawk libncurses5-dev

Then to use the internal toolchain, specify “USE_INTERNAL_TOOLCHAIN=1” as part of your invocation to make.:

make USE_INTERNAL_TOOLCHAIN=1

4.4. Compile the ATF

Download the source code and compile using:

export CROSS_COMPILE=/opt/gcc-linaro-6.3.1-2017.02-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
git clone https://git.theobroma-systems.com/arm-trusted-firmware.git
cd arm-trusted-firmware
make PLAT=rk3399 bl31
cd ..

4.5. Compile U-Boot

U-Boot is used as the bootloader on the RK3399-Q7 module.

Download the source code using:

git clone https://git.theobroma-systems.com/puma-u-boot.git

The U-Boot build process uses the files generated in the previous steps Copy the previously generated files rk3399m0.bin from the Cortex-M0 firmware and bl31.bin from the ATF to the puma-u-boot directory. Recent U-Boot releases expect the bl31.bin file under the name bl31-rk3399.bin. To support all variants, the file is copied twice:

cp rk3399-cortex-m0/rk3399m0.bin puma-u-boot
cp arm-trusted-firmware/build/rk3399/release/bl31.bin puma-u-boot/bl31.bin
cp arm-trusted-firmware/build/rk3399/release/bl31.bin puma-u-boot/bl31-rk3399.bin

Then you are ready to compile U-Boot:

cd puma-u-boot
export ARCH=arm64
export CROSS_COMPILE=/opt/gcc-linaro-6.3.1-2017.02-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
make puma-rk3399_defconfig
make -j4
tools/mkimage -n rk3399 -T rksd  -d spl/u-boot-spl.bin spl_sd.img
tools/mkimage -n rk3399 -T rkspi -d spl/u-boot-spl.bin spl_spi.img
make u-boot.itb
cd ..

The resulting bootloader consists of three files: spl_sd.img, spl_spi.img and u-boot.itb, but only one of the spl files is used at a time. The file spl_sd.img is only used when booting from SD-card or eMMC, while spl_spi.img is only used when booting from SPI NOR flash.

4.6. Compile the Boot Script

The U-Boot boot sequence is controlled by a file called boot.scr. This file is generated from a plain-text file called boot.cmd.

Download the repository and generate boot.scr using:

git clone https://git.theobroma-systems.com/som-tools.git
cd som-tools
make -C boot-script
cd ..

4.7. Compile the Linux Kernel

The kernel source code can be cloned with:

git clone https://git.theobroma-systems.com/puma-linux.git

Compile using:

cd puma-linux
export ARCH=arm64
export CROSS_COMPILE=/opt/gcc-linaro-6.3.1-2017.02-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
make puma-rk3399_defconfig
make -j4 rockchip/rk3399-puma.dtb Image

This will create the two files needed for booting with U-Boot (paths are relative to the puma-linux directory):

  • The device tree: arch/arm64/boot/dts/rockchip/rk3399-puma.dtb
  • The kernel image: arch/arm64/boot/Image

4.8. Building the root filesystem

A filesystem can be created using Debootstrap, specifying arm64 as architecture in the command line.

Supposing the target dir is called rk3399-rootfs and the chosen distribution is Debian 9 “Stretch” (recommended):

export targetdir=/opt/rk3399-rootfs
sudo mkdir -p $targetdir
sudo debootstrap --arch=arm64 --foreign stretch $targetdir http://deb.debian.org/debian/

Next, copy the qemu-arm-static binary into the right place for the binfmt packages to find it and copy the resolv.conf file from the host system:

sudo cp /usr/bin/qemu-aarch64-static $targetdir/usr/bin/
sudo cp /etc/resolv.conf $targetdir/etc

This will provide a very basic arm64 rootfs in the targetdir. For the next stages, we chroot to the target dir:

sudo chroot $targetdir

Second stage of debootstrap inside the new root dir:

/debootstrap/debootstrap --second-stage

Set up the apt package sources:

cat << EOT > /etc/apt/sources.list
deb http://deb.debian.org/debian stretch main contrib non-free
deb http://deb.debian.org/debian stretch-updates main contrib non-free
deb http://security.debian.org/ stretch/updates main contrib non-free
EOT

We can now pull the latest apt database from the Debian mirrors and install locales:

apt update
apt install locales
echo "en_US.UTF-8 UTF-8" > /etc/locale.gen
locale-gen

Install any additional packages inside the chroot. An ssh server and sudo are recommended:

apt install openssh-server sudo

Set the root password for logging in via serial console:

passwd

To log in over ssh, create another user besides root (root login over ssh is not permitted by default):

adduser user

Add the new user to the sudo group so you can switch to root privileges if needed:

adduser user sudo

Set up a basic network configuration file with DHCP via eth0 and enable automatic DNS configuration through systemd-resolved:

cat << EOT > /etc/systemd/network/eth0.network
[Match]
Name=eth0
[Network]
DHCP=yes
EOT
systemctl enable systemd-networkd
systemctl enable systemd-resolved
ln -f -s /lib/systemd/resolv.conf /etc/resolv.conf

When executing systemctl enable you may get the message qemu: Unsupported syscall: 278. The operation still succeeds and the message can be safely ignored.

Set the hostname and exit from the chroot:

echo rk3399-q7 > /etc/hostname
exit

We now have a root filesystem which can be deployed to the SD card.

4.9. Deploy on SD Card

4.9.1. Partition Setup

Both U-Boot and Linux will be located on the same SD card. The layout of the card after setup is as follows:

Offset Contents Files
0 Partition table  
32kiB U-Boot SPL spl_sd.img
240kiB U-Boot environment  
256kiB U-Boot + ATF + Cortex-M0 firmware u-boot.itb
2MiB Partition 1 (ext4 - Linux root fs) boot.scr, Image, rk3399-puma.dtb, defaultEnv.txt and rootfs

To setup a SD card for booting you first need to create partitions. Partitions can be created using fdisk (assuming the SD card is mapped to /dev/sd``X`` , where X should be replaced with your corresponding device-letter) and has no partitions (this can be checked using the p command):

sudo fdisk /dev/sdX
> p

This should show an empty partition table, for example:

Disk /dev/sdX: 3980 MB, 3980394496 bytes
123 heads, 62 sectors/track, 1019 cylinders, total 7774208 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdbbd45c7

Device Boot      Start         End      Blocks   Id  System

If there are partitions on the sdcard, they can be deleted with o.

The required partition can be created with the command n, then accepting the defaults, except for First sector, where we use 4096:

> n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): <ENTER>
Partition number (1-4, default 1): <ENTER>
First sector (2048-7774207, default 2048): 4096
Last sector, +sectors or +size{K,M,G} (...): <ENTER>

This will create a primary partition at offset 2MiB. Enter w to write the new partition table to the disk:

> w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Now we format the partition as ext4:

sudo /sbin/mkfs.ext4 -E lazy_itable_init=0 /dev/sdX1

The option lazy_itable_init=0 speeds up the first boot because it initializes the inode tables in advance.

The SD card is now ready to have the U-Boot bootloader and Linux deployed.

4.9.2. Deploy U-Boot

The U-Boot images spl_sd.img and u-boot.itb are written to the SD card. Assuming the SD card is mapped to /dev/sdX:

sudo dd if=puma-u-boot/spl_sd.img of=/dev/sdX bs=1k seek=32  conv=nocreat
sudo dd if=puma-u-boot/u-boot.itb of=/dev/sdX bs=1k seek=256 conv=nocreat

4.9.3. Deploy the Linux Kernel and the Root Filesystem

Mount the SD card partition and copy the rootfs (assuming that the rootfs is located at /opt/rk3399-rootfs and the sd card at /dev/sdX1):

sudo mkdir -p /mnt/sdcard
sudo mount /dev/sdX1 /mnt/sdcard
sudo cp -av /opt/rk3399-rootfs/* /mnt/sdcard

Copy kernel image, device tree and boot script into the boot directory:

sudo cp -r som-tools/boot-script/boot/{boot.scr,puma_rk3399} /mnt/sdcard/boot
sudo cp puma-linux/arch/arm64/boot/dts/rockchip/rk3399-puma.dtb /mnt/sdcard/boot/puma_rk3399
sudo cp puma-linux/arch/arm64/boot/Image /mnt/sdcard/boot/puma_rk3399

Finally, unmount the SD card:

sudo umount /mnt/sdcard

The SD card is ready for booting.

4.9.4. U-Boot Customization

The boot script /boot/boot.scr handles the boot sequence. Unless you want to customize the sequence, no further action is required.

If you want to step through the sequence manually or customize it, you can execute the following commands on the U-Boot prompt:

setenv bootargs root=/dev/mmcblk0p1 rw rootwait
ext4load mmc 1:1 $kernel_addr_r boot/puma_rk3399/Image
ext4load mmc 1:1 $fdt_addr_r boot/puma_rk3399/rk3399-puma.dtb
booti $kernel_addr_r - $fdt_addr_r

Optionally, these commands can be compiled together in a single command and saved so it is performed on every subsequent boot:

setenv bootargs root=/dev/mmcblk0p1 rw rootwait
setenv boot_sd "ext4load mmc 1:1 $kernel_addr_r boot/puma_rk3399/Image && \
ext4load mmc 1:1 $fdt_addr_r boot/puma_rk3399/rk3399-puma.dtb && \
booti $kernel_addr_r - $fdt_addr_r"
setenv bootcmd run boot_sd
saveenv

To reset the U-Boot settings to default, execute:

env default -f -a
saveenv

Note

root=/dev/mmcblk0p1 and ext4load mmc 1:1 refer to the SD Card. Use root=/dev/mmcblk1p1 and ext4load mmc 0:1 instead to boot from eMMC.

4.10. Deploy on SPI NOR-flash

To have a reliable boot sequence even if eMMC and/or SD-card fail, U-Boot can be written in to the onboard NOR-flash. While u-boot.itb is used for both SD-card and SPI-flash boot, spl_spi.img is used instead of spl_sd.img.

From U-boot:

sf probe
load mmc 1 $kernel_addr_r root/spl_spi.img
sf erase 0 +$filesize
sf write $kernel_addr_r 0 $filesize

load mmc 1 $kernel_addr_r root/u-boot.itb
sf erase 0x40000 +$filesize
sf write $kernel_addr_r 0x40000 $filesize

From Linux:

dd if=spl_spi.img of=/dev/mtdblock0 bs=256k seek=0 conv=nocreat
dd if=u-boot.itb  of=/dev/mtdblock0 bs=256k seek=1 conv=nocreat

4.11. Deploy on On-Board eMMC storage

As the eMMC storage is only accessible from the module itself, you must first boot the RK3399-Q7 from SD card.

Partition and format the eMMC storage as described in 4.9.1 Partition Setup, but using the device /dev/mmcblk1.

Mount the eMMC partition and copy the contents of the SD card to the eMMC storage. The copy process will take about 30 seconds:

sudo mkdir -p /mnt/emmc
sudo mount /dev/mmcblk1p1 /mnt/emmc
sudo cp -ax / /mnt/emmc
sudo umount /mnt/emmc

The final step is copying the bootloader to the eMMC:

dd if=/boot/puma_rk3399/spl_sd.img of=/dev/mmcblk1 bs=1k seek=32  conv=nocreat
dd if=/boot/puma_rk3399/u-boot.itb of=/dev/mmcblk1 bs=1k seek=256 conv=nocreat

Shut down the board (poweroff command) and remove the SD card. Make sure the boot selector switch is set to “Normal Boot”. The next boot will run U-Boot off the internal eMMC storage.

4.12. Compiling Linux Applications

The easiest option is to compile your applications directly on a running RK3399-Q7 module. Just install the gcc package and related utilities and you are good to go:

sudo apt-get install build-essential

The second option is to cross-compile your applications. The ARMv8-A compiler that was installed earlier is suitable to compile applications for the RK3399-Q7.

4.13. Serial Number

Each RK3399-Q7 module has a unique serial number that can be read by software.

In U-Boot, the serial number is contained in the environment variable serial#. You can print it using the command:

printenv serial#

Under Linux, it is represented by a simple text file in /sys:

cat /sys/firmware/devicetree/base/serial-number

The serial number is fixed in hardware (derived from the SoC CPU ID) and cannot be modified.

4.14. MAC Address

By default, the MAC address of each RK3399-Q7 module is a random value derived from the serial number. The properties of this default MAC address are:

  • It is a Locally Administered Address: The U/L bit of the MAC address is set to 1
  • It is not guaranteed to be globally unique
  • The address is fixed for each RK3399-Q7 module. It stays constant across reboots as it is deterministically derived from the serial number

To set your own Universally Administered Address, you overwrite the U-Boot environment variable ethaddr. On the U-Boot prompt, with XX:XX:XX:XX:XX:XX replaced by your MAC address:

setenv ethaddr XX:XX:XX:XX:XX:XX
saveenv

The MAC address can be queried from the U-Boot prompt using:

printenv ethaddr

To reset the MAC address to the default value, run:

env delete ethaddr
saveenv