GSoC progress report (1)

Qubes-OS accepted to run under Google’s oss-fuzz.

What to fuzz?

My mentor and I decided that fuzzing the qubes file-copy service would be the best to get started with since it essentially involves file I/O.

Dockerfile for project’s base image

This took up a lot of time mainly because I started off in a wrong direction and ended up spending the initial 4-5 days on it.

My mentor and I had decided to keep the build process as close to the regular build process as possible. The ideal way would be to build using the qubes-builder itself. So, I wrote the Dockerfile, build scripts etc. The travis builds helped a lot. Once the build process was complete, I tried running the docker container through the oss-fuzz helper script, but it didn’t build successfully. The issue was that the qubes-builder builds and packages the qubes components inside chroot directories. While doing so, the builder also mounts the procfs and sysfs of the host machine. Unfortunately, mount being a privileged command, doesn’t run in Docker containers unless they are run with the --privileged flag which is not the case with oss-fuzz. So, all the effort and time was spent in vain.

After some discussion with @marmarek, building the components directly seemed to be the best way. I then started from scratch and wrote the Dockerfile, this time focussing on directly buidling only one individual component: qubes-linux-utils

FROM gcr.io/oss-fuzz-base/base-builder
MAINTAINER [email protected]

RUN apt-get update && apt-get -y install build-essential automake libtool git python

WORKDIR qubes-os

RUN git clone --depth 1 https://github.com/qubesos/qubes-builder-debian.git $SRC/qubes-os/builder-debian && \
    echo "deb [arch=amd64] http://deb.qubes-os.org/r4.0/vm stretch main" >> /etc/apt/sources.list && \
    echo "deb [arch=amd64] http://deb.qubes-os.org/r4.0/vm stretch-testing main" >> /etc/apt/sources.list && \
    apt-key add $SRC/qubes-os/builder-debian/keys/qubes-debian-r4.0.asc && \
    apt-get update

RUN git clone https://github.com/paraschetal/qubes-linux-utils.git $SRC/qubes-os/linux-utils && \
    $SRC/qubes-os/builder-debian/scripts/debian-parser control --build-depends $SRC/qubes-os/linux-utils/debian/control | xargs apt-get -y install && \
    $SRC/qubes-os/builder-debian/scripts/debian-parser control --qubes-build-depends debian $SRC/qubes-os/linux-utils/debian/control | xargs apt-get -y install && \
    $SRC/qubes-os/builder-debian/scripts/debian-parser control --qubes-build-depends stretch $SRC/qubes-os/linux-utils/debian/control | xargs apt-get -y install

COPY build.sh $SRC/

This sets up the base image for qubes-linux-utils by installing all the dependencies.

build.sh for fuzz target of libqubes-rpc-filecopy

The next thing to do was to build libqubes-rpc-filecopy, and link it with the fuzzing engine and fuzz targets. This took longer than I had expected because I was not very familiar with the oss-fuzz environment and the debugging process took a long time (building the docker image took ~15 minutes each time). Also, I wanted to make as few changes to the code of the actual components as possible. I also fixed a tiny issue in the qubes-linux-utils.

#!/bin/bash

export BACKEND_VMM=xen


cd $SRC/qubes-os/linux-utils/
git checkout origin/fuzz

cd qrexec-lib

$CC $CFLAGS -c ioall.c
$CC $CFLAGS -c copy-file.c
$CC $CFLAGS -c crc32.c
$CC $CFLAGS -c pack.c
$CC $CFLAGS -c unpack.c
ar rcs libqubes-rpc-filecopy.a ioall.o copy-file.o crc32.o unpack.o pack.o

$CXX $CXXFLAGS -o $OUT/fuzzer-qubes-rpc-filecopy -I. -I./fuzzer fuzzer/fuzzer.cc -lFuzzingEngine libqubes-rpc-filecopy.a
#include <stdio.h>
#include <unistd.h>
#include <stdint.h>
#include <stdlib.h>
#include "fuzzer.h"
#include "libqubes-rpc-filecopy.h"

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
  //fuzz
  return 0;
}
#ifndef FUZZER_H
#define FUZZER_H

#ifdef __cplusplus
extern "C" {
#endif
typedef void (notify_progress_t)(int, int);
typedef void (error_handler_t)(const char *fmt, va_list args);
void register_notify_progress(notify_progress_t *func);
void register_error_handler(error_handler_t *func);

/* common functions */
int copy_file(int outfd, int infd, long long size, unsigned long *crc32);
const char *copy_file_status_to_str(int status);
void set_size_limit(unsigned long long new_bytes_limit, unsigned long long new_files_limit);
void set_verbose(int value);
/* register open fd to /proc/PID/fd of this process */
void set_procfs_fd(int value);
int write_all(int fd, const void *buf, int size);
int read_all(int fd, void *buf, int size);
int copy_fd_all(int fdout, int fdin);
void set_nonblock(int fd);
void set_block(int fd);

/* unpacking */
extern unsigned long Crc32_ComputeBuf( unsigned long inCrc32, const void *buf,
        size_t bufLen );
extern int do_unpack(void);

/* packing */
int single_file_processor(const char *filename, const struct stat *st);
int do_fs_walk(const char *file, int ignore_symlinks);
/* used in tar2qfile to alter only headers, but keep original file stream */
void write_headers(const struct file_header *hdr, const char *filename);
int copy_file_with_crc(int outfd, int infd, long long size);
/* MUST be called before first do_fs_walk/single_file_processor */
void qfile_pack_init(void);
void set_ignore_quota_error(int value);
/* those two will call registered error handler if needed */
void wait_for_result(void);
void notify_end_and_wait_for_result(void);

#ifdef __cplusplus
}
#endif

#endif

Building fuzz target locally

[email protected] ~/p/oss-fuzz> python infra/helper.py build_fuzzers --engine libfuzzer --sanitizer address  $PROJECT_NAME                                    qubes-linux-utils
Running: docker build -t gcr.io/oss-fuzz/qubes-os projects/qubes-os
Sending build context to Docker daemon  5.12 kB
Step 1 : FROM gcr.io/oss-fuzz-base/base-builder
 ---> b49026d90ee6
Step 2 : MAINTAINER [email protected]
 ---> Using cache
 ---> e9b4718508a9
Step 3 : RUN apt-get update && apt-get -y install build-essential automake libtool git python
 ---> Using cache
 ---> 4dead403c402
Step 4 : WORKDIR qubes-os
 ---> Using cache
 ---> 3d57a7ba20b2
Step 5 : RUN git clone --depth 1 https://github.com/qubesos/qubes-builder-debian.git $SRC/qubes-os/builder-debian &&     echo "deb [arch=amd64] http://deb.qubes-os.org/r4.0/vm stretch main" >> /etc/apt/sources.list &&     echo "deb [arch=amd64] http://deb.qubes-os.org/r4.0/vm stretch-testing main" >> /etc/apt/sources.list &&     apt-key add $SRC/qubes-os/builder-debian/keys/qubes-debian-r4.0.asc &&     apt-get update
 ---> Using cache
 ---> 956918347127
Step 6 : RUN git clone https://github.com/paraschetal/qubes-linux-utils.git $SRC/qubes-os/linux-utils &&     $SRC/qubes-os/builder-debian/scripts/debian-parser control --build-depends $SRC/qubes-os/linux-utils/debian/control | xargs apt-get -y install &&     $SRC/qubes-os/builder-debian/scripts/debian-parser control --qubes-build-depends debian $SRC/qubes-os/linux-utils/debian/control | xargs apt-get -y install &&     $SRC/qubes-os/builder-debian/scripts/debian-parser control --qubes-build-depends stretch $SRC/qubes-os/linux-utils/debian/control | xargs apt-get -y install
 ---> Using cache
 ---> 3fe96d1c2849
Step 7 : COPY build.sh $SRC/
 ---> Using cache
 ---> 2b13369d7524
Successfully built 2b13369d7524
Running: docker run --rm -i --cap-add SYS_PTRACE -e BUILD_UID=1000 -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -v /home/user/projects/oss-fuzz/build/out/qubes-os:/out -v /home/user/projects/oss-fuzz/build/work/qubes-os:/work -t gcr.io/oss-fuzz/qubes-os
---------------------------------------------------------------
Compiling libFuzzer to /usr/lib/libFuzzingEngine.a ...ar: creating /usr/lib/libFuzzingEngine.a
 done.
CC=clang
CXX=clang++
CFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp
CXXFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp -stdlib=libc++
---------------------------------------------------------------
Adding user `builder' ...
Adding new group `builder' (1000) ...
Adding new user `builder' (1000) with group `builder' ...
Creating home directory `/home/builder' ...
Copying files from `/etc/skel' ...
+ export BACKEND_VMM=xen
+ BACKEND_VMM=xen
+ cd /src/qubes-os/linux-utils/
+ git checkout origin/fuzz
Note: checking out 'origin/fuzz'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 1a608fe... Fix include header syntax
+ cd qrexec-lib
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp -c ioall.c
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp -c copy-file.c
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp -c crc32.c
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp -c pack.c
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp -c unpack.c
+ ar rcs libqubes-rpc-filecopy.a ioall.o copy-file.o crc32.o unpack.o pack.o
+ clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp -stdlib=libc++ -o /out/fuzzer-qubes-rpc-filecopy -I. -I./fuzzer fuzzer/fuzzer.cc -lFuzzingEngine libqubes-rpc-filecopy.a
[email protected] ~/p/oss-fuzz> 

Running fuzz target locally

The fuzz targets run locally using the test scripts by oss-fuzz for testing locally to ensure that the targets will run on their infrastructure.

[email protected] ~/p/oss-fuzz> python infra/helper.py run_fuzzer $PROJECT_NAME fuzzer-qubes-rpc-filecopy                                                     qubes-linux-utils
Running: docker run --rm -i --cap-add SYS_PTRACE -e FUZZING_ENGINE=libfuzzer -v /home/user/projects/oss-fuzz/build/out/qubes-os:/out -t gcr.io/oss-fuzz-base/base-runner run_fuzzer fuzzer-qubes-rpc-filecopy
/out/fuzzer-qubes-rpc-filecopy -rss_limit_mb=2048 -timeout=25
INFO: Seed: 814672508
INFO: Loaded 1 modules (1 guards): [0x881cd0, 0x881cd4), 
INFO: -max_len is not provided, using 64
INFO: A corpus is not provided, starting from an empty corpus
#0	READ units: 1
#1	INITED cov: 1 ft: 1 corp: 1/1b exec/s: 0 rss: 30Mb
#524288	pulse  cov: 1 ft: 1 corp: 1/1b exec/s: 262144 rss: 57Mb
#1048576	pulse  cov: 1 ft: 1 corp: 1/1b exec/s: 262144 rss: 59Mb
#2097152	pulse  cov: 1 ft: 1 corp: 1/1b exec/s: 262144 rss: 59Mb
#4194304	pulse  cov: 1 ft: 1 corp: 1/1b exec/s: 246723 rss: 59Mb
^C==7== libFuzzer: run interrupted; exiting
[email protected] ~/p/oss-fuzz> 

The next task, of course, is to maximize coverage for this target and adding other targets.

Summary

In the first ten days of GSoC:

  • Qubes OS was accepted as a project to run on Google’s oss-fuzz.
  • The Dockerfile for building the project base image is complete.
  • The build script for the fuzz target of libqubes-rpc-filecopy is complete.
  • The fuzz target successfully builds locally with the helper oss-fuzz script.
  • The fuzz target successfully runs locally with the helper oss-fuzz script.

So overall, the integration with oss-fuzz is now almost done. The first time was a bit of a hurdle, since the whole environment was new and took a lot of time to debug (building the docker images every time I had to change a line took ~15 min). From now onwards, adding more fuzz targets should be smooth sailing because now I am aware of the environment. The next task will be to improve upon the coverage of the fuzz targets and add more of them.