Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add minimal arm64 kernel build configuration #4568

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

bulwahn
Copy link
Contributor

@bulwahn bulwahn commented Mar 12, 2024

The new configs upstream-arm64-min-{kasan,kcsan,leak} are intended to create fuzzing campaigns, where the absolute number of uncovered lines of code is minimal. These configs might also be useful for config bisections and understanding which non-reproducible warnings appear even on kernels with very minimal kernel configurations.

They are based on allnoconfig, then further configs are enabled to boot a qemu virt generic virtual platform, see https://www.qemu.org/docs/master/system/arm/virt.html.

The kernels built with these configs boot with the following qemu invocation on an ARM Neoverse-N1 system:

qemu-system-aarch64 -m 8G -smp 4 -cpu host,kvm-no-adjvtime=yes -machine virt,gic-version=host
-enable-kvm -append 'root=/dev/vda rw net.ifnames=0'
-kernel "kernel-image" -hda "bookworm-image"
-net user,hostfwd=tcp:127.0.0.1:10088-:22 -net nic,model=virtio-net-pci
--nographic --snapshot -monitor none

Before we add those configs, we add some configs to base.yml that have been implicitly part of many of the configs already and which some of the configs in base.yml already depend on.

Further, finally add myself as contributor, as I had time to do some own investigation and development---after being only a product owner to some contributing teams to syzkaller in the past.

In order to prepare adding a minimal config, make a few configs explicit in base.yml. Consequently,
drop those from net.yml and subsystems.yml.

CGROUPS was always implicitly in all configs, as CGROUP_PIDS and MEMCG in base.yml depend on it,
and generation of the configs would have failed. PCI was always implicitly in all configs, as
PCI_HOST_GENERIC in base.yml depends on it, and generation of the configs would have failed.
Further configs are needed for the Debian bullseye and bookworm image to start properly.

None of these additions to base.yml lead to any effective change in the current configs.
When starting up the bookworm user-space image, systemd complains:

  systemctl status proc-sys-fs-binfmt_misc.mount
  mount: /proc/sys/fs/binfmt_misc: mount point does not exist

Add BINFMT_MISC to base to address this issue.

After regeneration, the configs are modified as expected. Fuzzing with these configs should not be
significantly impacted due to this config addition.
In order to prepare adding a minimal config, make JUMP_LABEL explicit in kfence.yml. Consequently,
drop it from subsystems.yml. JUMP_LABEL is needed for KFENCE_STATIC_KEYS.

The changes to upstream-arm-{full,kasan}.config are probably without any significant impact to the
current fuzzing.
The new configs upstream-arm64-min-{kasan,kcsan,leak} are intended to create fuzzing campaigns,
where the absolute number of uncovered lines of code is minimal. These configs might also be useful
for config bisections and understanding which non-reproducible warnings appear even on kernels with
very minimal kernel configurations.

They are based on allnoconfig, then further configs are enabled to boot a qemu virt generic
virtual platform, see https://www.qemu.org/docs/master/system/arm/virt.html.

The kernels built with these configs boot with the following qemu invocation on a ARM Neoverse-N1
system:

  qemu-system-aarch64 -m 8G -smp 4 -cpu host,kvm-no-adjvtime=yes -machine virt,gic-version=host
    -enable-kvm -append 'root=/dev/vda rw net.ifnames=0'
    -kernel <kernel> -hda <bookworm-image>
    -net user,hostfwd=tcp:127.0.0.1:10088-:22 -net nic,model=virtio-net-pci
    --nographic --snapshot -monitor none
@bulwahn bulwahn changed the title Add minimal arm64 config Add minimal arm64 kernel build configuration Mar 12, 2024
Copy link

codecov bot commented Mar 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.1%. Comparing base (c35c26e) to head (d700cf5).

Additional details and impacted files

see 1 file with indirect coverage changes

@a-nogikh
Copy link
Collaborator

If I'm not mistaken (Cc @dvyukov), that's what our -base.config configs are effectively for -- they don't include many subsystems that the full kernel config does and they are used as the baseline for kernel config minimization (it's actually the only way we actually use them on syzbot).

So I think that instead of introducing new config files it'd be much better to make the existing -base.config configs smaller (and also base them on allnoconfig?). Probably our only requirement for them is that they keep on being bootable and fuzzable on GCE and qemu.

@bulwahn
Copy link
Contributor Author

bulwahn commented Mar 19, 2024

@a-nogikh Okay, I can try to reduce the '-base.config' configs and see where we get with that.

From my memory of the previous review of the yml config bits and my future plans, I see the following potential changes and challenges to the current base configs to get closer what I am proposing here:

  • as already noted, we start with allnoconfig instead of defconfig.
  • we change to boot with qemu-virt-arm64 instead of versatile. I believe virt-arm64 is more minimal (with regards to needed drivers) and more generic (with regards to cpu architectures, so it can run natively with kvm host support on various arm64 server machines, now and in the future); but I am not the qemu and qemu arm board support expert.
  • I can check if it boots in qemu, but I am sure if I can check if it boots on GCE yet. For now, I just have a personal google cloud account for experimentation and as soon as my initial free budget is gone, I need to check what checking to boot on GCE costs me (given that as of now, I would pay from my own pockets), besides the challenge that the GCE setup is yet another setup I first need to get familiar with---but documentation in syzkaller seems pretty good to guide me.
  • I want to reduce the config even more from what is proposed now. The next reduction would probably require to build a super minimal user-space image, e.g., with buildroot or dracut, that is just an init ramdisk and still can fuzz the kernel. Then the kernel would not even require ext4 file system support. And then reduce even further, e.g., use ssh and rpc over predefined vsocks to throw out the whole TCP/IP stack. I am not sure if that is still fitting to the -base config use case you have in mind, especially as then the deployment may have more different variants, e.g., user-space images based on Debian, some based on minimal init ram disks, etc.

Let me explore and make a proposal to refine the -base configs. Then we can see how far the changes to base configs are acceptable and at which point we would really say that some even more reduced configs simply should be a config by their own, and if they deserve to be maintained in the upstream repository or should be maintained out of tree elsewhere---with getting as much into the mainline repository as possible to reduce the churn.

@a-nogikh Let me know if my thoughts are going in the wrong directions, or if some of the changes to base configs are not acceptable from your point of view and the deployment you have.

@a-nogikh
Copy link
Collaborator

we change to boot with qemu-virt-arm64 instead of versatile.

Does the resulting kernel still boot on versatile?

I can check if it boots in qemu, but I am sure if I can check if it boots on GCE yet

I can check it once needed, though if it does not boot, this process may become trickier.

The docs list some parameters:
https://cloud.google.com/compute/docs/images/building-custom-os#kernelrequired

Though AFAIK in our case it usually boiled down to CONFIG_GVE.

The next reduction would probably require to build a super minimal user-space image, e.g., with buildroot or dracut

This is okay, we use buildroot for all Linux instances.

The next reduction would probably require to build a super minimal user-space image, e.g., with buildroot or dracut
throw out the whole TCP/IP stack.

This is already out of the -base scope indeed.

Will you need to use all these differently minimal configs (with ext4 and net, without ext4, without net, etc) simultaneously? Or will you will always fully switch to the most minimal?

Let me explore and make a proposal to refine the -base configs. Then we can see how far the changes to base configs are acceptable and at which point we would really say that some even more reduced configs simply should be a config by their own, and if they deserve to be maintained in the upstream repository or should be maintained out of tree elsewhere---with getting as much into the mainline repository as possible to reduce the churn.

Sounds good to me!
Let's check how well what you need is achievable with -base and then decide what we can do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants