Debugging Keystone in QEmu

2025-05-21 909 words 5 minutes

Contents

This post has been written with the help of @OussamaELmnaouri2001.

The goal of this blogpost if to show how we can play with Keystone in QEMu in three ways:

Running the default framework in QEmu
Debugging from the Security Monitor point of view
Debugging from an enclave binary point of view

Requirements

It is assumed that the Keystone framework repository has been cloned on the host:

        
KEYSTONE_REPO=$PWD/keystone
git clone https://github.com/keystone-enclave/keystone
cd keystone
git checkout 88c49ee
git submodule update --init --recursive

Running the default framework in QEmu

Compiling and running QEmu

        
cd KEYSTONE_REPO
make -j$(nproc) 
make run

Running default examples in the emulator

Once QEmu is launched, it will boot a Buildroot-based kernel:

        
        
        
    
Welcome to Buildroot
buildroot login: root
Password: sifive
$ modprobe keystone-driver
$ /usr/share/keystone/examples/hello.ke      
Verifying archive integrity... MD5 checksums are OK. All good.
Uncompressing Keystone Enclave Package
hello, world!

Debugging from the Security Monitor point of view

In order to debug an application, we will need two terminal windows:

One for the emulator.
Another one for the debugger.

In the first terminal, let’s start QEmu:

        
$ cd $KEYSTONE_REPO
$ KEYSTONE_DEBUG=y make run       
>>>  (INF) Starting QEMU
/home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/host/bin/qemu-system-riscv64 -m 2G -smp 4 -nographic -machine virt,rom=/home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/bootrom.bin -bios /home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/fw_jump.elf -kernel /home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/Image -drive file=/home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/rootfs.ext2,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -append "console=ttyS0 ro root=/dev/vda" -netdev user,id=net0,net=192.168.100.1/24,dhcpstart=192.168.100.128,hostfwd=tcp::9821-:22 -device virtio-net-device,netdev=net0 -device virtio-rng-pci  -gdb tcp::9822 -S

GDB is now available from port 9822 (be aware of this port number, it may be different in other cases). For the debugger, the toolchain has been compiled and installed from the official repository.

The idea is to run the debugger with the Security Monitor binary (fw_jump.elf). The following command prints functions available in the binary with pmp in their names:

        
        
        
    
nm -n build-generic64/buildroot.build/images/fw_jump.elf| grep pmp
0000000080005f98 T pmp_set
00000000800060d4 T pmp_get
0000000080009f6c T sbi_hart_pmp_count
0000000080009f94 T sbi_hart_pmp_granularity
0000000080009fbc T sbi_hart_pmp_addrbits
000000008000a00c T sbi_hart_pmp_configure
0000000080019088 T osm_pmp_set
000000008001efc4 T pmp_detect_region_overlap_atomic
000000008001f030 T pmp_unset_global
000000008001f0a8 T pmp_set_global
000000008001f120 T pmp_init
000000008001f330 T pmp_set_keystone
000000008001fb08 T pmp_unset
000000008002028c T pmp_region_free_atomic
00000000800203a8 T pmp_region_init
000000008002081c T pmp_region_init_atomic
00000000800208b8 T pmp_region_get_addrDebugging from an enclave binary - Existing example
000000008002090c T pmp_region_get_size
0000000080021214 T sbi_pmp_ipi_local_update
0000000080021260 T send_and_sync_pmp_ipi
0000000080063680 b pmp_lock

In a second terminal, let’s say we want to debug the pmp_set_keystone function:

        
$ cd $KEYSTONE_REPO
$ riscv64-unknown-linux-gnu-gdb build-generic64/buildroot.build/images/fw_jump.elf

In GDB:

        
        
        
    
Reading symbols from build-generic64/buildroot.build/images/fw_jump.elf...

(gdb) target remote localhost:9822
Remote debugging using localhost:9822
0x0000000000001000 in ?? ()

(gdb) set riscv use-compressed-breakpoints no

(gdb) b pmp_set_keystone
Breakpoint 1 at 0x8001f330: file ../keystone-sm-de08c214516f19c8/src/pmp.c, line 114.

(gdb) c
Continuing.

Thread 1 hit Breakpoint 1, pmp_set_keystone (region_idx=0, perm=perm@entry=0 '\000') at ../keystone-sm-de08c214516f19c8/src/pmp.c:114
114	  return TEST_BIT(region_def_bitmap, region_idx);

(gdb) info registers
ra             0x800191d8	0x800191d8 <sm_init+80>
sp             0x8006aec0	0x8006aec0
gp             0x0	0x0
tp             0x8006b000	0x8006b000
t0             0x80000460	2147484768
t1             0xffffffffffff00ff	-65281
t2             0x8	8
fp             0x8006af00	0x8006af00
s1             0x80060128	2147877160
a0             0x0	0
a1             0x0	0
a2             0x8002b5a8	2147661224
a3             0x1f1800	2037760
a4             0x0	0
a5             0x1f1800	2037760
a6             0x7	7
a7             0x0	0
s2             0xffffffffffffffff	-1
s3             0x1	1
s4             0x8006aecc	2147921612
s5             0x80038000	2147713024
s6             0x0	0
s7             0x4	4
s8             0x2000	8192
s9             0x800586a8	2147845800
s10            0x0	0
s11            0x0	0
t3             0xffffffffff00ffff	-16711681
t4             0xffffff00ffffffff	-1095216660481
t5             0xffff00ffffffffff	-280375465082881
t6             0xff00ffffffffffff	-71776119061217281
pc             0x8001f330	0x8001f330 <pmp_set_keystone>

(gdb) source scripts/gdb/pmp.py   
-0x7fffffffffebee53

(gdb) pmp-dump
PMP reg 1 NAPOT
	cfg	0x18 
	addr	0x0 = 0x0 -> 0x8
PMP reg 2 NAPOT
	cfg	0x1f RWX
	addr	0x0 = 0x0 -> 0x8

GDB can be used as usual. Keystone provides a pmp.py script which allows to view PMP registers in a reader friendly view with the pmp-dump command.

Debugging from an enclave binary point of view

In this second example, the goal is to debug an existing example. We will start with the “Hello world” binary and we will set a breakpoint on the main function. Applications are not compiled with debug symbols by default, we need to modify the CMakeLists.txt to add debug symbols on the eapp (the enclave binary) and the host:

        
        
        
    
@@ -9,11 +9,13 @@ set(eyrie_plugins "io_syscall linux_syscall env_setup")
 # eapp
 a bin
 add_executable(${eapp_bin} ${eapp_src})
+target_compile_options(${eapp_bin} PRIVATE -g -O0) 
 target_link_libraries(${eapp_bin} "-static")
 
 # host
 
 add_executable(${host_bin} ${host_src})a bin
+target_compile_options(${host_bin} PRIVATE -g -O0) 
 target_link_libraries(${host_bin} ${KEYSTONE_LIB_HOST} ${KEYSTONE_LIB_EDGE})
 
 # add target for Eyrie runtime (see keystone.cmake)

Now, we just need to compile the image again. It will be shorter as it only recompiles Keystone examples.

        
$ cd $KEYSTONE_REPO
$ make -j$(nproc)

In a first terminal:

        
$ cd $KEYSTONE_REPO
$ KEYSTONE_DEBUG=y make run       
>>>  (INF) Starting QEMU
/home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/host/bin/qemu-system-riscv64 -m 2G -smp 4 -nographic -machine virt,rom=/home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/bootrom.bin -bios /home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/fw_jump.elf -kernel /home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/Image -drive file=/home/test/Documents/sandbox/keystone//build-generic64/buildroot.build/images/rootfs.ext2,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -append "console=ttyS0 ro root=/dev/vda" -netdev user,id=net0,net=192.168.100.1/24,dhcpstart=192.168.100.128,hostfwd=tcp::9821-:22 -device virtio-net-device,netdev=net0 -device virtio-rng-pci  -gdb tcp::9822 -S

In a second terminal, we need to use GDB with the hello binary. It is located at build-generic64/buildroot.build/build/keystone-examples-<some_hex_string> (it usually creates another directory compared to a default compilation) :

        
riscv64-unknown-linux-gnu-gdb build-generic64/buildroot.build/build/keystone-examples-2f54c3f0cb1b5fa6/hello/hello

GDB can be used as usual. Keystone provides a pmp.py script which allows to view PMP registers in a reader friendly view with the pmp-dump:

        
        
        
    
Reading symbols from build-generic64/buildroot.build/build/keystone-examples-2f54c3f0cb1b5fa6/hello/hello...

(gdb) target remote localhost:9822
Remote debugging using localhost:9822
0x0000000000001000 in ?? ()

(gdb) set riscv use-compressed-breakpoints no

(gdb) b main  
Breakpoint 1 at 0x106d0: file eapp/hello.c, line 5.

(gdb) c
Continuing.

Thread 1 hit Breakpoint 1, main () at eapp/hello.c:5
5	  printf("hello, world!\n");

(gdb) source scripts/gdb/pmp.py  
-0x7fffffffffebee53

(gdb) pmp-dump
PMP reg 0 NAPOT
	cfg	0x18 
	addr	0x2003ffff = 0x80000000 -> 0x80200000
PMP reg 1 NAPOT
	cfg	0x1f RWX
	addr	0x20cbffff = 0x83200000 -> 0x83400000
PMP reg 2 NAPOT
	cfg	0x1f RWX
	addr	0x0 = 0x0 -> 0x8
PMP reg 7 NAPOT
	cfg	0x1f RWX
	addr	0x20c77fff = 0x831c0000 -> 0x83200000