🐞 linux内核调试

(🗒️)

环境

编译Busybox系统环境

macOS下面编译Busybox有点问题，换成同架构的Linux系统进行编译

创建文件系统

下载BusyBox源码并编译：

1
2
3
4
5
sudo dnf install make gcc -y
wget https://busybox.net/downloads/busybox-1.37.0.tar.bz2
tar -xvf busybox-1.37.0.tar.bz2
cd busybox-1.37.0
make menuconfig

常见错误:

1
2
3
4
5
6
7
8
# fix: fatal error: curses.h: No such file or directory
sudo dnf install ncurses-devel -y
sudo dnf install ncurses -y

# fix: Install ncurses (ncurses-devel) and try again
vim scripts/kconfig/lxdialog/Makefile
# hash out this line
always         := $(hostprogs-y) dochecklxdialog

勾选Build static binary (no shared libs)

静态链接

创建磁盘镜像

1
2
3
4
5
dd if=/dev/zero of=rootfs.img bs=1M count=2048
sudo dnf install e2fsprogs -y # mkfs.ext4
mkfs.ext4 rootfs.img
mkdir root
sudo mount -t ext4 -o loop rootfs.img ./root

常见错误:

1
libbb/hash_md5_sha.c:1316:35: error: ‘sha1_process_block64_shaNI’ undeclared (first use in this function); did you mean ‘sha1_process_block64’?

使用vim编辑替换符号:%s#sha1_process_block64_shaNI#sha1_process_block64#g

1
networking/tc.c:236:27: error: ‘TCA_CBQ_MAX’ undeclared (first use in this function); did you mean ‘TCA_CBS_MAX’?

删除networking/tc.c文件，见Bug

1
2
3
4
/usr/sbin/ld: cannot find -lm: No such file or directory
/usr/sbin/ld: have you installed the static version of the m library ?
/usr/sbin/ld: cannot find -lresolv: No such file or directory
/usr/sbin/ld: have you installed the static version of the resolv library ?

安装glibc-static和libgcc:

1
2
sudo dnf install glibc-static -y
sudo dnf install libgcc -y

拷贝BusyBox到根文件系统

1
sudo make install CONFIG_PREFIX=./root

修改根文件系统

1
2
3
4
5
6
cd root
sudo mkdir proc dev etc home mnt
sudo cp -r ../examples/bootfloppy/etc/* etc/
sudo chmod -R 777 .
cd ..
sudo umount ./root

启动内核

内核文件和文件系统

使用QEMU启动内核和文件系统：

1
brew install qemu

启动命令

1
qemu-system-aarch64 -smp 4 -M virt,accel=hvf -cpu host -nographic -m 2048M -kernel linux-6.7.9/arch/arm64/boot/Image -hda busybox-1.37.0/rootfs.img -append "root=/dev/vda rw console=ttyAMA0" -s

内核启动成功

关闭内核地址随机化

关闭内核地址随机化

其他问题:

1
qemu-system-aarch64: Error: r = HV_BAD_ARGUMENT (0xfae94003, at ../target/arm/hvf/hvf.c:2234)

解决办法:
问题已修复，需要更新QEMU版本。

预编译二进制没有发版，可以源码编译QEMU

1
2
3
4
5
git clone https://gitlab.com/qemu-project/qemu.git
cd qemu
./configure --target-list=aarch64-softmmu,riscv64-softmmu
make -j$(nproc)
codesign -d --entitlements accel/hvf/entitlements.plist --force -s -  ./build/qemu-system-aarch64-unsigned
1
/Volumes/Work/qemu/build/qemu-system-aarch64-unsigned -smp 4 -M virt,accel=hvf -cpu host -nographic -m 2048M -kernel linux-6.7.9/arch/arm64/boot/Image -hda busybox-1.37.0/rootfs.img -append "root=/dev/vda rw console=ttyAMA0" -s

Xcode调试

打开Xcode自定义LLDB命令支持

1
defaults write com.apple.dt.Xcode IDEDebuggerFeatureSetting 12

启用自定义lldb命令

使用下面的LLDB命令连接QEMU调试：

1
2
target create $PROJECT_DIR/vmlinux
gdb-remote localhost:1234

调试配置

调试信息

生成Xcode项目

把下面的内容保存为project.yml文件，然后使用xcodegen生成Xcode项目：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
name: Linux
options:
  bundleIdPrefix: top.ourfor.opensource
settings:
  HEADER_SEARCH_PATHS: $SRCROOT/**
targets:
  Kernel:
    type: tool
    platform: macOS
    deploymentTarget: 10.15
    preBuildScripts:
      - script: export PATH="/opt/homebrew/bin:$PATH" && lkmake ARCH=arm64 LLVM=1 -j $(nproc)
        name: Makefile Build
    postCompileScripts:
      - script: cp $PROJECT_DIR/vmlinux $BUILT_PRODUCTS_DIR/$EXECUTABLE_NAME
        name: Copy Executable
      - script: |
          export PATH="/opt/homebrew/bin:/Volumes/Work/qemu/build:$PATH"
          cd /Volumes/Work
          # 检查并关闭已存在的 qemu 会话
          tmux has-session -t qemu 2>/dev/null && tmux kill-session -t qemu
          # 在新的 tmux 会话中启动 QEMU
          tmux new-session -d -s qemu '/Volumes/Work/qemu/build/qemu-system-aarch64-unsigned \
            -smp 4 \
            -M virt,accel=hvf \
            -cpu host \
            -nographic \
            -m 2048M \
            -kernel linux-6.7.9/arch/arm64/boot/Image \
            -hda busybox-1.37.0/rootfs.img \
            -append "root=/dev/vda rw console=ttyAMA0" \
            -s'
          sleep 1
          echo "QEMU started in tmux session 'qemu'"
          echo "Use 'tmux attach -t qemu' to connect to QEMU console"
        name: Launch QEMU
        runOnlyWhenInstalling: false
  Dummy:
    type: tool
    platform: macOS
    deploymentTarget: 10.15
    sources:
      - path: arch
        includes: ["**/*.[chsS]"]
      - path: block
        includes: ["**/*.[chsS]"]
      - path: crypto
        includes: ["**/*.[chsS]"]
      - path: drivers
        includes: ["**/*.[chsS]"]
      - path: fs
        includes: ["**/*.[chsS]"]
      - path: include
        includes: ["**/*.[chsS]"]
      - path: init
        includes: ["**/*.[chsS]"]
      - path: io_uring
        includes: ["**/*.[chsS]"]
      - path: ipc
        includes: ["**/*.[chsS]"]
      - path: kernel
        includes: ["**/*.[chsS]"]
      - path: lib
        includes: ["**/*.[chsS]"]
      - path: mm
        includes: ["**/*.[chsS]"]
      - path: net
        includes: ["**/*.[chsS]"]
      - path: security
        includes: ["**/*.[chsS]"]
      - path: sound
        includes: ["**/*.[chsS]"]
      - path: virt
        includes: ["**/*.[chsS]"]
schemes:
  Linux:
    build:
      targets:
        Kernel: [run]
    run:
      debugEnabled: true
      executable: none
      customLLDBInitFile: |
        target create $PROJECT_DIR/vmlinux
        gdb-remote localhost:1234

修改源码并调试

修改后编译，打开终端，执行tmux a -t qemu查看Linux系统输出

修改源码并调试

查看局部变量

查看局部变量