Set version

Resolves #17

.travis.yml

@@ -0,0 +1,30 @@
+language: go
+
+go:
+  - 1.x
+  - master
+
+os:
+  - linux
+
+dist:
+  - bionic
+
+addons:
+  apt:
+    packages:
+    - qemu
+
+services:
+  - docker
+
+env:
+  - GO111MODULE=on
+
+install: true
+
+before_script:
+  - ./tools/qemu-debian-img/bootstrap.sh
+
+script:
+  - go test -parallel 1 -v ./...

CHANGELOG.md

@@ -0,0 +1,153 @@
+# Changelog
+
+[ISO 8601](https://xkcd.com/1179/).
+
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.1.0] 2019-08-30
+
+### Added
+
+- Global configuration file (~/.out-of-tree/out-of-tree.toml) allow to
+  set up default values for settings.
+
+- rootfs generator for Ubuntu 14.04.
+
+- Parameter for setting up docker registry server.
+
+- Support for (distro-specific) custom docker commands that will be
+  executed before the base template.
+
+- Parameter for setting up a reliability threshold for exit code.
+
+- Parameter for setting up global timeout, after which no new tasks
+  will be started.
+
+### Fixed
+
+- Spelling in output.
+
+- Now kernel generation will not fail if there are no directory
+  /lib/modules inside the container.
+
+## [1.0.0] 2019-08-20
+
+### Added
+
+- New parameter `--max=X` is added for `autogen` (generate kernels
+  base on `.out-of-tree.toml` definitions) and `pew` (automated
+  runs) and allows to specify a maximum number of runs per each
+  supported kernel in module/exploit definition.
+
+- New command `genall` -- generate all kernels for specified
+  distro/version.
+
+- All logs stores in sqlite3 database. Implemented specific commands
+  for making simple queries and export data to markdown and json.
+
+- Implemented success rate calculation for previous runs.
+
+- Save of build results supported by parameter `--dist` for `pew`.
+
+- Support for generating kernels info from host system.
+
+- Support for build on host.
+
+- Support for custom kernels.
+
+- Now debugging environment is automatically looking for debug
+  kernel on the host system.
+
+- Added ability to enable/disable kaslr/smep/smap/kpti for debugging
+  by command line flags.
+
+- New parameter `--threads=N` is added for `pew` and allows to
+  specify maximum number of threads that will be used for parallel
+  build/run/test.
+
+- Tagging for runs. Tags write to log and can be used for
+  statistics.
+
+- Added non-regex way to set kernel version in .out-of-tree.toml (see
+  examples).
+
+- New command `pack` that perform tests in subdirectories.
+
+- Added ability to disable kaslr/smep/smap/kpti for in artifact
+  definition.
+
+- Added ability to change amount of memory/CPUs and set qemu timeout
+  in artifact definition (`.out-of-tree.toml`).
+
+- Now images downloading while `kernel autogen`, bootstrap is not
+  required anymore.
+
+- Support CentOS kernels.
+
+### Changed
+
+- Now if there's no base image found — out-of-tree will try to use
+  an image from closest previous version, e.g. image from Ubuntu
+  18.04 for Ubuntu 18.10.
+
+- Kernel modules tests will not be failed if there are no tests
+  exists.
+
+- Now *out-of-tree* will return negative error code if at least one
+  of the stage was failed.
+
+- Project is switch to use Go modules.
+
+- Now test.sh is used by default if copying is not implemented in
+  Makefile.
+
+- dmesg is not cleaned before the start of module/exploit anymore.
+
+- qemu/kvm will use all host cpu features.
+
+### Removed
+
+- *Kernel factory* is removed completely in favor of incremental
+  Dockerfiles.
+
+- `bootstrap` is not doing anything anymore. It'll be removed in next
+  release.
+
+### Fixed
+
+- Command `timeout` is not required anymore.
+
+- Errors is more meaningful.
+
+- Temporary files is moved to `~/.out-of-tree/tmp/` to avoid docker
+  mounting issues on some systems.
+
+## [0.2.0] - 2019-12-01
+
+The main purpose of the release is to simplify installation.
+
+### Changes
+
+- All configuration moved to `~/.out-of-tree`.
+
+- Now prebuilt images can be downloaded with bootstrap.
+
+- Ability to generate kernels specific to .out-of-tree.toml in
+  current directory. So now there's no need to wait for several
+  hours for start work on specific kernel with module/exploit.
+
+- Now there's no need to keep source tree and _out-of-tree_ can be
+  distributed in binary form.
+
+- New command: **debug**. Creates interactive environment for kernel
+  module/exploit development. Still work-in-progress.
+
+- No warning anymore if test.sh is not exists.
+
+## [0.1.0] - 2019-11-20
+
+Initial release that was never tagged.
+
+Refer to state after first public release on ZeroNights 2018
+([video](https://youtu.be/2tL7bbCdIio),
+[slides](https://2018.zeronights.ru/wp-content/uploads/materials/07-Ways-to-automate-testing-Linux-kernel-exploits.pdf)).

README.md

@@ -1,17 +1,50 @@
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/aba4aad2046b4d1a9a99cf98e22c018b)](https://app.codacy.com/app/jollheef/out-of-tree?utm_source=github.com&utm_medium=referral&utm_content=jollheef/out-of-tree&utm_campaign=Badge_Grade_Dashboard)
+[![Build Status](https://travis-ci.com/jollheef/out-of-tree.svg?branch=master)](https://travis-ci.org/jollheef/out-of-tree)
+[![Go Report Card](https://goreportcard.com/badge/code.dumpstack.io/tools/out-of-tree)](https://goreportcard.com/report/code.dumpstack.io/tools/out-of-tree)
+[![Documentation Status](https://readthedocs.org/projects/out-of-tree/badge/?version=latest)](https://out-of-tree.readthedocs.io/en/latest/?badge=latest)
+[![Donate](https://img.shields.io/badge/donate-paypal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=R8W2UQPZ5X5JE&source=url)
+[![Donate](https://img.shields.io/badge/donate-bitcoin-green.svg)](https://blockchair.com/bitcoin/address/bc1q23fyuq7kmngrgqgp6yq9hk8a5q460f39m8nv87)
+
 # [out-of-tree](https://out-of-tree.io)
 
 out-of-tree kernel {module, exploit} development tool
 
+out-of-tree is for automating some routine actions for creating development environments for debugging kernel modules and exploits, generating reliability statistics for exploits, and also provides the ability to easily integrate into CI (Continuous Integration).
+
 ![Screenshot](https://cloudflare-ipfs.com/ipfs/Qmb88fgdDjbWkxz91sWsgmoZZNfVThnCtj37u3mF2s3T3T)
 
-## Installation
+## Requirements
 
-    $ go get github.com/jollheef/out-of-tree
-    $ out-of-tree bootstrap
+[Qemu](https://www.qemu.org), [docker](https://docker.com) and [golang](https://golang.org) is required.
+
+Also do not forget to set GOPATH and PATH e.g.:
+
+    $ echo 'export GOPATH=$HOME' >> ~/.bashrc
+    $ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
+    $ source ~/.bashrc
+
+### Gentoo
+
+    # emerge app-emulation/qemu app-emulation/docker dev-lang/go
+
+### macOS
+
+    $ brew install go qemu
+    $ brew cask install docker
+
+### Fedora
+
+    $ sudo dnf install go qemu moby-engine
+
+Also check out [docker post-installation steps](https://docs.docker.com/install/linux/linux-postinstall/).
+
+## Build from source
+
+    $ go get -u code.dumpstack.io/tools/out-of-tree
 
 Then you can check it on kernel module example:
 
-    $ cd $GOPATH/github.com/jollheef/out-of-tree/examples/kernel-module
+    $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/examples/kernel-module
     $ out-of-tree kernel autogen # generate kernels based on .out-of-tree.toml
     $ out-of-tree pew
 
@@ -45,13 +78,16 @@ Use custom kernels config
 
     $ out-of-tree --kernels /path/to/kernels.toml pew
 
-## Generate all kernels
+Generate all kernels
 
-Does not required if you dont need to use `--guess`.
+    $ out-of-tree kernel genall --distro Ubuntu --ver 16.04
 
-    $ cd $GOPATH/src/github.com/jollheef/out-of-tree/tools/kernel-factory
-    $ ./bootstrap.sh # more than 6-8 hours for all kernels
-    $ export OUT_OF_TREE_KCFG=$GOPATH/src/github.com/jollheef/out-of-tree/tools/kernel-factory/output/kernels.toml
+
+## Troubleshooting
+
+If anything happens that you cannot solve -- just remove `$HOME/.out-of-tree`.
+
+But it'll be better if you'll write the bug report.
 
 ## Development
 
@@ -59,6 +95,6 @@ Read [Qemu API](qemu/README.md).
 
 ### Generate images
 
-    $ cd $GOPATH/src/github.com/jollheef/out-of-tree/tools/qemu-debian-img/
+    $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/tools/qemu-debian-img/
     $ docker run --privileged -v $(pwd):/shared -e IMAGE=/shared/ubuntu1404.img -e RELEASE=trusty -t gen-ubuntu1804-image
     $ docker run --privileged -v $(pwd):/shared -e IMAGE=/shared/ubuntu1604.img -e RELEASE=xenial -t gen-ubuntu1804-image

bootstrap.config.go

@@ -1,7 +0,0 @@
-// Copyright 2018 Mikhail Klementev. All rights reserved.
-// Use of this source code is governed by a AGPLv3 license
-// (or later) that can be found in the LICENSE file.
-
-package main
-
-const imagesURL = "https://github.com/jollheef/out-of-tree/releases/download/v0.2/images.tar.gz"

bootstrap.go

@@ -1,76 +0,0 @@
-// Copyright 2018 Mikhail Klementev. All rights reserved.
-// Use of this source code is governed by a AGPLv3 license
-// (or later) that can be found in the LICENSE file.
-
-package main
-
-import (
-	"io"
-	"io/ioutil"
-	"log"
-	"net/http"
-	"os"
-	"os/exec"
-	"os/user"
-)
-
-// inspired by Edd Turtle code
-func downloadFile(filepath string, url string) (err error) {
-	out, err := os.Create(filepath)
-	if err != nil {
-		return
-	}
-	defer out.Close()
-
-	resp, err := http.Get(url)
-	if err != nil {
-		return
-	}
-	defer resp.Body.Close()
-
-	_, err = io.Copy(out, resp.Body)
-	return
-}
-
-func unpackTar(archive, destination string) (err error) {
-	cmd := exec.Command("tar", "xf", archive)
-	cmd.Dir = destination + "/"
-
-	rawOutput, err := cmd.CombinedOutput()
-	if err != nil {
-		// I don't like when some errors printed inside
-		// So if you know way to do it better - FIXME please
-		log.Println("Unpack images error:", string(rawOutput), err)
-		return
-	}
-
-	return
-}
-
-func bootstrapHandler() (err error) {
-	usr, err := user.Current()
-	if err != nil {
-		return
-	}
-
-	imagesPath := usr.HomeDir + "/.out-of-tree/images/"
-	os.MkdirAll(imagesPath, os.ModePerm)
-
-	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_")
-	if err != nil {
-		log.Println("Temporary directory creation error:", err)
-		return
-	}
-	defer os.RemoveAll(tmp)
-
-	imagesArchive := tmp + "/images.tar.gz"
-
-	err = downloadFile(imagesArchive, imagesURL)
-	if err != nil {
-		log.Println("Download file error:", err)
-		return
-	}
-
-	err = unpackTar(imagesArchive, imagesPath)
-	return
-}

config/config.go

@@ -10,27 +10,44 @@ import (
 	"io/ioutil"
 	"os"
 	"regexp"
+	"strconv"
 	"strings"
+	"time"
 
 	"github.com/naoina/toml"
 )
 
+type kernel struct {
+	Version []int
+	Major   []int
+	Minor   []int
+	Patch   []int
+}
+
+// KernelMask defines the kernel
 type KernelMask struct {
 	DistroType    DistroType
 	DistroRelease string // 18.04/7.4.1708/9.1
 	ReleaseMask   string
+
+	// Overrides ReleaseMask
+	Kernel kernel
 }
 
+// DockerName is returns stable name for docker container
 func (km KernelMask) DockerName() string {
 	distro := strings.ToLower(km.DistroType.String())
 	release := strings.Replace(km.DistroRelease, ".", "__", -1)
 	return fmt.Sprintf("out_of_tree_%s_%s", distro, release)
 }
 
+// ArtifactType is the kernel module or exploit
 type ArtifactType int
 
 const (
+	// KernelModule is any kind of kernel module
 	KernelModule ArtifactType = iota
+	// KernelExploit is the privilege escalation exploit
 	KernelExploit
 )
 
@@ -38,6 +55,7 @@ func (at ArtifactType) String() string {
 	return [...]string{"module", "exploit"}[at]
 }
 
+// UnmarshalTOML is for support github.com/naoina/toml
 func (at *ArtifactType) UnmarshalTOML(data []byte) (err error) {
 	stype := strings.Trim(string(data), `"`)
 	stypelower := strings.ToLower(stype)
@@ -46,11 +64,12 @@ func (at *ArtifactType) UnmarshalTOML(data []byte) (err error) {
 	} else if strings.Contains(stypelower, "exploit") {
 		*at = KernelExploit
 	} else {
-		err = errors.New(fmt.Sprintf("Type %s is unsupported", stype))
+		err = fmt.Errorf("Type %s is unsupported", stype)
 	}
 	return
 }
 
+// MarshalTOML is for support github.com/naoina/toml
 func (at ArtifactType) MarshalTOML() (data []byte, err error) {
 	s := ""
 	switch at {
@@ -59,17 +78,49 @@ func (at ArtifactType) MarshalTOML() (data []byte, err error) {
 	case KernelExploit:
 		s = "exploit"
 	default:
-		err = errors.New(fmt.Sprintf("Cannot marshal %d", at))
+		err = fmt.Errorf("Cannot marshal %d", at)
 	}
 	data = []byte(`"` + s + `"`)
 	return
 }
 
+// Duration type with toml unmarshalling support
+type Duration struct {
+	time.Duration
+}
+
+// UnmarshalTOML for Duration
+func (d *Duration) UnmarshalTOML(data []byte) (err error) {
+	duration := strings.Replace(string(data), "\"", "", -1)
+	d.Duration, err = time.ParseDuration(duration)
+	return
+}
+
+// MarshalTOML for Duration
+func (d Duration) MarshalTOML() (data []byte, err error) {
+	data = []byte(`"` + d.Duration.String() + `"`)
+	return
+}
+
+// Artifact is for .out-of-tree.toml
 type Artifact struct {
 	Name             string
 	Type             ArtifactType
 	SourcePath       string
 	SupportedKernels []KernelMask
+
+	Qemu struct {
+		Cpus    int
+		Memory  int
+		Timeout Duration
+	}
+
+	Mitigations struct {
+		DisableSmep  bool
+		DisableSmap  bool
+		DisableKaslr bool
+		DisableKpti  bool
+	}
 }
 
 func (ka Artifact) checkSupport(ki KernelInfo, km KernelMask) (
@@ -90,6 +141,7 @@ func (ka Artifact) checkSupport(ki KernelInfo, km KernelMask) (
 	return
 }
 
+// Supported returns true if given kernel is supported by artifact
 func (ka Artifact) Supported(ki KernelInfo) (supported bool, err error) {
 	for _, km := range ka.SupportedKernels {
 		supported, err = ka.checkSupport(ki, km)
@@ -101,16 +153,22 @@ func (ka Artifact) Supported(ki KernelInfo) (supported bool, err error) {
 	return
 }
 
+// DistroType is enum with all supported distros
 type DistroType int
 
 const (
+	// Ubuntu https://ubuntu.com/
 	Ubuntu DistroType = iota
+	// CentOS https://www.centos.org/
 	CentOS
+	// Debian https://www.debian.org/
 	Debian
 )
 
+// DistroTypeStrings is the string version of enum DistroType
 var DistroTypeStrings = [...]string{"Ubuntu", "CentOS", "Debian"}
 
+// NewDistroType is create new Distro object
 func NewDistroType(dType string) (dt DistroType, err error) {
 	err = dt.UnmarshalTOML([]byte(dType))
 	return
@@ -120,6 +178,7 @@ func (dt DistroType) String() string {
 	return DistroTypeStrings[dt]
 }
 
+// UnmarshalTOML is for support github.com/naoina/toml
 func (dt *DistroType) UnmarshalTOML(data []byte) (err error) {
 	sDistro := strings.Trim(string(data), `"`)
 	if strings.EqualFold(sDistro, "Ubuntu") {
@@ -129,11 +188,12 @@ func (dt *DistroType) UnmarshalTOML(data []byte) (err error) {
 	} else if strings.EqualFold(sDistro, "Debian") {
 		*dt = Debian
 	} else {
-		err = errors.New(fmt.Sprintf("Distro %s is unsupported", sDistro))
+		err = fmt.Errorf("Distro %s is unsupported", sDistro)
 	}
 	return
 }
 
+// MarshalTOML is for support github.com/naoina/toml
 func (dt DistroType) MarshalTOML() (data []byte, err error) {
 	s := ""
 	switch dt {
@@ -144,12 +204,20 @@ func (dt DistroType) MarshalTOML() (data []byte, err error) {
 	case Debian:
 		s = "Debian"
 	default:
-		err = errors.New(fmt.Sprintf("Cannot marshal %d", dt))
+		err = fmt.Errorf("Cannot marshal %d", dt)
 	}
 	data = []byte(`"` + s + `"`)
 	return
 }
 
+// ByRootFS is sorting by .RootFS lexicographically
+type ByRootFS []KernelInfo
+
+func (a ByRootFS) Len() int           { return len(a) }
+func (a ByRootFS) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
+func (a ByRootFS) Less(i, j int) bool { return a[i].RootFS < a[j].RootFS }
+
+// KernelInfo defines kernels.toml entries
 type KernelInfo struct {
 	DistroType    DistroType
 	DistroRelease string // 18.04/7.4.1708/9.1
@@ -158,14 +226,19 @@ type KernelInfo struct {
 	KernelRelease string
 
 	// Build-time information
+	KernelSource  string // module/exploit will be build on host
 	ContainerName string
 
 	// Runtime information
 	KernelPath string
 	InitrdPath string
 	RootFS     string
+
+	// Debug symbols
+	VmlinuxPath string
 }
 
+// KernelConfig is the ~/.out-of-tree/kernels.toml configuration description
 type KernelConfig struct {
 	Kernels []KernelInfo
 }
@@ -181,6 +254,7 @@ func readFileAll(path string) (buf []byte, err error) {
 	return
 }
 
+// ReadKernelConfig is for read kernels.toml
 func ReadKernelConfig(path string) (kernelCfg KernelConfig, err error) {
 	buf, err := readFileAll(path)
 	if err != nil {
@@ -195,16 +269,93 @@ func ReadKernelConfig(path string) (kernelCfg KernelConfig, err error) {
 	return
 }
 
-func ReadArtifactConfig(path string) (artifactCfg Artifact, err error) {
+func rangeRegexp(start, end int) (s string) {
+	s += "("
+	for i := start; i <= end; i++ {
+		s += strconv.Itoa(i)
+		if i != end {
+			s += "|"
+		}
+	}
+	s += ")"
+	return
+}
+
+func versionRegexp(l []int) (s string, err error) {
+	switch len(l) {
+	case 1:
+		s += strconv.Itoa(l[0])
+	case 2:
+		s += rangeRegexp(l[0], l[1])
+	default:
+		err = errors.New("version must contain one value or range")
+		return
+	}
+	return
+}
+
+func genReleaseMask(km kernel) (mask string, err error) {
+	s, err := versionRegexp(km.Version)
+	if err != nil {
+		return
+	}
+	mask += s + "[.]"
+
+	s, err = versionRegexp(km.Major)
+	if err != nil {
+		return
+	}
+	mask += s + "[.]"
+
+	s, err = versionRegexp(km.Minor)
+	if err != nil {
+		return
+	}
+	mask += s
+
+	switch len(km.Patch) {
+	case 0:
+		// ok
+	case 1:
+		mask += "-" + strconv.Itoa(km.Patch[0]) + "-"
+	case 2:
+		mask += "-" + rangeRegexp(km.Patch[0], km.Patch[1]) + "-"
+	default:
+		err = errors.New("version must contain one value or range")
+		return
+	}
+
+	mask += ".*"
+	return
+}
+
+// ReadArtifactConfig is for read .out-of-tree.toml
+func ReadArtifactConfig(path string) (ka Artifact, err error) {
 	buf, err := readFileAll(path)
 	if err != nil {
 		return
 	}
 
-	err = toml.Unmarshal(buf, &artifactCfg)
+	err = toml.Unmarshal(buf, &ka)
 	if err != nil {
 		return
 	}
 
+	for i, _ := range ka.SupportedKernels {
+		km := &ka.SupportedKernels[i]
+		if len(km.Kernel.Version) != 0 && km.ReleaseMask != "" {
+			s := "Only one way to define kernel version is allowed"
+			err = errors.New(s)
+			return
+		}
+
+		if km.ReleaseMask == "" {
+			km.ReleaseMask, err = genReleaseMask(km.Kernel)
+			if err != nil {
+				return
+			}
+		}
+	}
+
 	return
 }

config/config_test.go

@@ -16,7 +16,7 @@ func TestMarshalUnmarshal(t *testing.T) {
 		Type: KernelModule,
 	}
 	artifactCfg.SupportedKernels = append(artifactCfg.SupportedKernels,
-		KernelMask{Ubuntu, "18.04", ".*"})
+		KernelMask{Ubuntu, "18.04", ".*", kernel{}})
 	buf, err := toml.Marshal(&artifactCfg)
 	if err != nil {
 		t.Fatal(err)
@@ -28,3 +28,38 @@ func TestMarshalUnmarshal(t *testing.T) {
 		t.Fatal(err)
 	}
 }
+
+func TestKernelRegex(t *testing.T) {
+	mask := "4[.]4[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99|100|101|102|103|104|105|106|107|108|109|110|111|112|113|114|115|116)-.*"
+	k := kernel{
+		Version: []int{4},
+		Major:   []int{4},
+		Minor:   []int{0},
+		Patch:   []int{1, 116},
+	}
+
+	gmask, err := genReleaseMask(k)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if mask != gmask {
+		t.Fatal("Got", gmask, "instead of", mask)
+	}
+
+	mask = "4[.]4[.]0.*"
+	k = kernel{
+		Version: []int{4},
+		Major:   []int{4},
+		Minor:   []int{0},
+	}
+
+	gmask, err = genReleaseMask(k)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if mask != gmask {
+		t.Fatal("Got", gmask, "instead of", mask)
+	}
+}

config/out-of-tree.go

@@ -0,0 +1,77 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package config
+
+import (
+	"os/user"
+
+	"github.com/naoina/toml"
+)
+
+type DockerCommand struct {
+	DistroType DistroType
+	Command    string
+}
+
+type OutOfTree struct {
+	Kernels     string
+	UserKernels string
+
+	Database string
+
+	Qemu struct {
+		Timeout string
+	}
+
+	Docker struct {
+		Timeout  string
+		Registry string
+
+		// Commands that will be executed before
+		// the base layer of Dockerfile
+		Commands []DockerCommand
+	}
+}
+
+func ReadOutOfTreeConf(path string) (c OutOfTree, err error) {
+	buf, err := readFileAll(path)
+	if err == nil {
+		err = toml.Unmarshal(buf, &c)
+		if err != nil {
+			return
+		}
+	} else {
+		// It's ok if there's no configuration
+		// then we'll just set default values
+		err = nil
+	}
+
+	usr, err := user.Current()
+	if err != nil {
+		return
+	}
+
+	if c.Kernels == "" {
+		c.Kernels = usr.HomeDir + "/.out-of-tree/kernels.toml"
+	}
+
+	if c.UserKernels == "" {
+		c.UserKernels = usr.HomeDir + "/.out-of-tree/kernels.user.toml"
+	}
+
+	if c.Database == "" {
+		c.Database = usr.HomeDir + "/.out-of-tree/db.sqlite"
+	}
+
+	if c.Qemu.Timeout == "" {
+		c.Qemu.Timeout = "1m"
+	}
+
+	if c.Docker.Timeout == "" {
+		c.Docker.Timeout = "1m"
+	}
+
+	return
+}

db.go

@@ -0,0 +1,317 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"strconv"
+	"time"
+
+	_ "github.com/mattn/go-sqlite3"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/qemu"
+)
+
+// Change on ANY database update
+const currentDatabaseVersion = 2
+
+const versionField = "db_version"
+
+type logEntry struct {
+	ID        int
+	Tag       string
+	Timestamp time.Time
+
+	qemu.System
+	config.Artifact
+	config.KernelInfo
+	phasesResult
+}
+
+func createLogTable(db *sql.DB) (err error) {
+	_, err = db.Exec(`
+	CREATE TABLE IF NOT EXISTS log (
+		id		INTEGER PRIMARY KEY,
+		time		DATETIME DEFAULT CURRENT_TIMESTAMP,
+		tag		TEXT,
+
+		name		TEXT,
+		type		TEXT,
+
+		distro_type	TEXT,
+		distro_release	TEXT,
+		kernel_release	TEXT,
+
+		build_output	TEXT,
+		build_ok	BOOLEAN,
+
+		run_output	TEXT,
+		run_ok		BOOLEAN,
+
+		test_output	TEXT,
+		test_ok		BOOLEAN,
+
+		qemu_stdout	TEXT,
+		qemu_stderr	TEXT,
+
+		kernel_panic	BOOLEAN,
+		timeout_kill	BOOLEAN
+	)`)
+	return
+}
+
+func createMetadataTable(db *sql.DB) (err error) {
+	_, err = db.Exec(`
+	CREATE TABLE IF NOT EXISTS metadata (
+		id	INTEGER PRIMARY KEY,
+		key	TEXT UNIQUE,
+		value	TEXT
+	)`)
+	return
+}
+
+func metaChkValue(db *sql.DB, key string) (exist bool, err error) {
+	sql := "SELECT EXISTS(SELECT id FROM metadata WHERE key = $1)"
+	stmt, err := db.Prepare(sql)
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	err = stmt.QueryRow(key).Scan(&exist)
+	return
+}
+
+func metaGetValue(db *sql.DB, key string) (value string, err error) {
+	stmt, err := db.Prepare("SELECT value FROM metadata " +
+		"WHERE key = $1")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	err = stmt.QueryRow(key).Scan(&value)
+	return
+}
+
+func metaSetValue(db *sql.DB, key, value string) (err error) {
+	stmt, err := db.Prepare("INSERT OR REPLACE INTO metadata " +
+		"(key, value) VALUES ($1, $2)")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	_, err = stmt.Exec(key, value)
+	return
+}
+
+func getVersion(db *sql.DB) (version int, err error) {
+	s, err := metaGetValue(db, versionField)
+	if err != nil {
+		return
+	}
+
+	version, err = strconv.Atoi(s)
+	return
+}
+
+func addToLog(db *sql.DB, q *qemu.System, ka config.Artifact,
+	ki config.KernelInfo, res *phasesResult, tag string) (err error) {
+
+	stmt, err := db.Prepare("INSERT INTO log (name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_output, build_ok, " +
+		"run_output, run_ok, " +
+		"test_output, test_ok, " +
+		"qemu_stdout, qemu_stderr, " +
+		"kernel_panic, timeout_kill) " +
+		"VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, " +
+		"$10, $11, $12, $13, $14, $15, $16);")
+	if err != nil {
+		return
+	}
+
+	defer stmt.Close()
+
+	_, err = stmt.Exec(
+		ka.Name, ka.Type, tag,
+		ki.DistroType, ki.DistroRelease, ki.KernelRelease,
+		res.Build.Output, res.Build.Ok,
+		res.Run.Output, res.Run.Ok,
+		res.Test.Output, res.Test.Ok,
+		q.Stdout, q.Stderr,
+		q.KernelPanic, q.KilledByTimeout,
+	)
+	if err != nil {
+		return
+	}
+
+	return
+}
+
+func getAllLogs(db *sql.DB, tag string, num int) (les []logEntry, err error) {
+	stmt, err := db.Prepare("SELECT id, time, name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_ok, run_ok, test_ok, kernel_panic, " +
+		"timeout_kill FROM log ORDER BY datetime(time) DESC " +
+		"LIMIT $1")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	rows, err := stmt.Query(num)
+	if err != nil {
+		return
+	}
+	defer rows.Close()
+
+	for rows.Next() {
+		le := logEntry{}
+		err = rows.Scan(&le.ID, &le.Timestamp,
+			&le.Name, &le.Type, &le.Tag,
+			&le.DistroType, &le.DistroRelease, &le.KernelRelease,
+			&le.Build.Ok, &le.Run.Ok, &le.Test.Ok,
+			&le.KernelPanic, &le.KilledByTimeout,
+		)
+		if err != nil {
+			return
+		}
+
+		if tag == "" || tag == le.Tag {
+			les = append(les, le)
+		}
+	}
+
+	return
+}
+
+func getAllArtifactLogs(db *sql.DB, tag string, num int, ka config.Artifact) (
+	les []logEntry, err error) {
+
+	stmt, err := db.Prepare("SELECT id, time, name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_ok, run_ok, test_ok, kernel_panic, " +
+		"timeout_kill FROM log WHERE name=$1 AND type=$2 " +
+		"ORDER BY datetime(time) DESC LIMIT $3")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	rows, err := stmt.Query(ka.Name, ka.Type, num)
+	if err != nil {
+		return
+	}
+	defer rows.Close()
+
+	for rows.Next() {
+		le := logEntry{}
+		err = rows.Scan(&le.ID, &le.Timestamp,
+			&le.Name, &le.Type, &le.Tag,
+			&le.DistroType, &le.DistroRelease, &le.KernelRelease,
+			&le.Build.Ok, &le.Run.Ok, &le.Test.Ok,
+			&le.KernelPanic, &le.KilledByTimeout,
+		)
+		if err != nil {
+			return
+		}
+
+		if tag == "" || tag == le.Tag {
+			les = append(les, le)
+		}
+	}
+
+	return
+}
+
+func getLogByID(db *sql.DB, id int) (le logEntry, err error) {
+	stmt, err := db.Prepare("SELECT id, time, name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_ok, run_ok, test_ok, " +
+		"build_output, run_output, test_output, " +
+		"qemu_stdout, qemu_stderr, " +
+		"kernel_panic, timeout_kill " +
+		"FROM log WHERE id=$1")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	err = stmt.QueryRow(id).Scan(&le.ID, &le.Timestamp,
+		&le.Name, &le.Type, &le.Tag,
+		&le.DistroType, &le.DistroRelease, &le.KernelRelease,
+		&le.Build.Ok, &le.Run.Ok, &le.Test.Ok,
+		&le.Build.Output, &le.Run.Output, &le.Test.Output,
+		&le.Stdout, &le.Stderr,
+		&le.KernelPanic, &le.KilledByTimeout,
+	)
+	return
+}
+
+func createSchema(db *sql.DB) (err error) {
+	err = createMetadataTable(db)
+	if err != nil {
+		return
+	}
+
+	err = createLogTable(db)
+	if err != nil {
+		return
+	}
+
+	return
+}
+
+func openDatabase(path string) (db *sql.DB, err error) {
+	db, err = sql.Open("sqlite3", path)
+	if err != nil {
+		return
+	}
+
+	db.SetMaxOpenConns(1)
+
+	exists, _ := metaChkValue(db, versionField)
+	if !exists {
+		err = createSchema(db)
+		if err != nil {
+			return
+		}
+
+		err = metaSetValue(db, versionField,
+			strconv.Itoa(currentDatabaseVersion))
+		return
+	}
+
+	version, err := getVersion(db)
+	if err != nil {
+		return
+	}
+
+	if version == 1 {
+		_, err = db.Exec(`ALTER TABLE log ADD tag TEXT`)
+		if err != nil {
+			return
+		}
+
+		err = metaSetValue(db, versionField, "2")
+		if err != nil {
+			return
+		}
+
+		version = 2
+	}
+
+	if version != currentDatabaseVersion {
+		err = fmt.Errorf("Database is not supported (%d instead of %d)",
+			version, currentDatabaseVersion)
+		return
+	}
+
+	return
+}

debug.go

@@ -13,9 +13,10 @@ import (
 	"strings"
 	"time"
 
-	"github.com/jollheef/out-of-tree/config"
-	qemu "github.com/jollheef/out-of-tree/qemu"
-	"github.com/logrusorgru/aurora"
+	"gopkg.in/logrusorgru/aurora.v1"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/qemu"
 )
 
 func firstSupported(kcfg config.KernelConfig, ka config.Artifact,
@@ -40,7 +41,7 @@ func firstSupported(kcfg config.KernelConfig, ka config.Artifact,
 	return
 }
 
-func handleLine(q *qemu.QemuSystem) (err error) {
+func handleLine(q *qemu.System) (err error) {
 	fmt.Print("out-of-tree> ")
 	rawLine := "help"
 	fmt.Scanf("%s", &rawLine)
@@ -63,7 +64,7 @@ func handleLine(q *qemu.QemuSystem) (err error) {
 	case "c", "cleanup":
 		q.Stdout = []byte{}
 	case "s", "ssh":
-		fmt.Println(q.GetSshCommand())
+		fmt.Println(q.GetSSHCommand())
 	case "q", "quit":
 		return errors.New("end of session")
 	default:
@@ -72,7 +73,7 @@ func handleLine(q *qemu.QemuSystem) (err error) {
 	return
 }
 
-func interactive(q *qemu.QemuSystem) (err error) {
+func interactive(q *qemu.System) (err error) {
 	for {
 		err = handleLine(q)
 		if err != nil {
@@ -82,7 +83,8 @@ func interactive(q *qemu.QemuSystem) (err error) {
 }
 
 func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
-	dockerTimeout time.Duration) (err error) {
+	dockerTimeout time.Duration, yekaslr, yesmep, yesmap, yekpti,
+	nokaslr, nosmep, nosmap, nokpti bool) (err error) {
 
 	ka, err := config.ReadArtifactConfig(workPath + "/.out-of-tree.toml")
 	if err != nil {
@@ -99,13 +101,67 @@ func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
 	}
 
 	kernel := qemu.Kernel{KernelPath: ki.KernelPath, InitrdPath: ki.InitrdPath}
-	q, err := qemu.NewQemuSystem(qemu.X86_64, kernel, ki.RootFS)
+	q, err := qemu.NewSystem(qemu.X86x64, kernel, ki.RootFS)
 	if err != nil {
 		return
 	}
+
+	if ka.Qemu.Cpus != 0 {
+		q.Cpus = ka.Qemu.Cpus
+	}
+	if ka.Qemu.Memory != 0 {
+		q.Memory = ka.Qemu.Memory
+	}
+
+	q.SetKASLR(false) // set KASLR to false by default because of gdb
+	q.SetSMEP(!ka.Mitigations.DisableSmep)
+	q.SetSMAP(!ka.Mitigations.DisableSmap)
+	q.SetKPTI(!ka.Mitigations.DisableKpti)
+
+	if yekaslr {
+		q.SetKASLR(true)
+	} else if nokaslr {
+		q.SetKASLR(false)
+	}
+
+	if yesmep {
+		q.SetSMEP(true)
+	} else if nosmep {
+		q.SetSMEP(false)
+	}
+
+	if yesmap {
+		q.SetSMAP(true)
+	} else if nosmap {
+		q.SetSMAP(false)
+	}
+
+	if yekpti {
+		q.SetKPTI(true)
+	} else if nokpti {
+		q.SetKPTI(false)
+	}
+
+	redgreen := func(name string, enabled bool) aurora.Value {
+		if enabled {
+			return aurora.BgGreen(aurora.Black(name))
+		}
+
+		return aurora.BgRed(aurora.Gray(name))
+	}
+
+	fmt.Printf("[*] %s %s %s %s\n",
+		redgreen("KASLR", q.GetKASLR()),
+		redgreen("SMEP", q.GetSMEP()),
+		redgreen("SMAP", q.GetSMAP()),
+		redgreen("KPTI", q.GetKPTI()))
+
+	fmt.Printf("[*] SMP: %d CPUs\n", q.Cpus)
+	fmt.Printf("[*] Memory: %d MB\n", q.Memory)
+
 	q.Debug(gdb)
 	coloredGdbAddress := aurora.BgGreen(aurora.Black(gdb))
-	fmt.Printf("[*] gdb runned on %s\n", coloredGdbAddress)
+	fmt.Printf("[*] gdb is listening on %s\n", coloredGdbAddress)
 
 	err = q.Start()
 	if err != nil {
@@ -125,9 +181,9 @@ func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
 		return
 	}
 
-	remoteFile := "/tmp/artifact"
+	remoteFile := "/tmp/exploit"
 	if ka.Type == config.KernelModule {
-		remoteFile += ".ko"
+		remoteFile = "/tmp/module.ko"
 	}
 
 	err = q.CopyFile("user", outFile, remoteFile)
@@ -138,6 +194,11 @@ func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
 	coloredRemoteFile := aurora.BgGreen(aurora.Black(remoteFile))
 	fmt.Printf("[*] build result copied to %s\n", coloredRemoteFile)
 
+	fmt.Printf("\n%s\n", q.GetSSHCommand())
+	fmt.Printf("gdb %s -ex 'target remote %s'\n\n", ki.VmlinuxPath, gdb)
+
+	// TODO set substitute-path /build/.../linux-... /path/to/linux-source
+
 	err = interactive(q)
 	return
 }

docs/index.rst

@@ -0,0 +1,30 @@
+out-of-tree
+===========
+
+*out-of-tree* is the kernel {module, exploit} development tool.
+
+*out-of-tree* was created on the purpose of decreasing complexity of
+environment for developing, testing and debugging Linux kernel
+exploits and out-of-tree kernel modules (that's why tool got a name
+"out-of-tree").
+
+While I'm trying to keep that documentation up-to-date, there may be
+some missing information. Use ``out-of-tree --help-long`` for checking
+all features.
+
+If you found anything missed here, please make a pull request or send
+patches to patch@dumpstack.io.
+
+If you need personal support, your company is interested in the
+project or you just want to share some thoughts -- feel free to write
+to root@dumpstack.io.
+
+Contents
+========
+
+:ref:`Keyword Index <genindex>`
+
+.. toctree::
+
+   introduction.rst
+   installation.rst

docs/installation.rst

@@ -0,0 +1,64 @@
+Installation
+============
+
+OS/Distro-specific
+==================
+
+Ubuntu
+------
+
+Install dependencies::
+
+  $ sudo snap install go --classic
+  $ sudo snap install docker
+  $ sudo apt install qemu-system-x86 build-essential gdb
+
+macOS
+-----
+
+Install dependencies::
+
+  $ brew install go qemu
+  $ brew cask install docker
+
+NixOS
+-----
+
+There's a minimal configuration that you need to apply::
+
+  #!nix
+  { config, pkgs, ... }:
+  {
+    virtualisation.docker.enable = true;
+    virtualisation.libvirtd.enable = true;
+    environment.systemPackages = with pkgs; [
+      go git
+    ];
+  }
+
+Common
+======
+
+Setup Go environment::
+
+  $ echo 'export GOPATH=$HOME' >> ~/.bashrc
+  $ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
+  $ source ~/.bashrc
+
+Build *out-of-tree*::
+
+  $ go get -u code.dumpstack.io/tools/out-of-tree
+
+.. note::
+  On a GNU/Linux you need to add your user to docker group if you want
+  to use *out-of-tree* without sudo. Note that this has a **serious**
+  security implications. Check *Docker* documentation for more
+  information.
+
+Test that everything works::
+
+  $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/examples/kernel-exploit
+  $ out-of-tree kernel autogen --max=1
+  $ out-of-tree pew --max=1
+
+Enjoy!

docs/introduction.rst

@@ -0,0 +1,109 @@
+Introduction
+============
+
+*out-of-tree* is written in *Go*, it uses *Docker* for generating
+kernel/filesystem images and *Qemu* for virtualization.
+
+Also it possible to generate kernels from the host system and use the
+custom one.
+
+*out-of-tree* supports *GNU/Linux* (usually it's tested on NixOS and
+latest Ubuntu LTS) and *macOS*. Technically all systems that supported
+by Go, Docker, and Qemu must work well. Create the issue if you'll
+notice any issue in integration for your operating system.
+
+All *Qemu* interaction is stateless.
+
+*out-of-tree* is allow and require metadata (``.out-of-tree.toml``)
+for work. TOML (Tom's Obvious, Minimal Language) is used for kernel
+module/exploit description.
+
+``.out-of-tree.toml`` is mandatory, you need to have in the current
+directory (usually, it's a project of kernel module/exploit) or use
+the ``--path`` flag.
+
+Files
+-----
+
+All data is stored in ``~/.out-of-tree/``.
+
+- *db.sqlite* contains logs related to run with ``out-of-tree pew``,
+  debug mode (``out-of-tree debug``) is not store any data.
+
+- *images* used for filesystem images (rootfs images that used for
+  ``qemu -hda ...``) that can be generated with the
+  ``tools/qemu-*-img/...``.
+
+- *kernels* stores all kernel ``vmlinuz/initrd/config/...`` files that
+  generated previously with a some *Docker magic*.
+
+- *kernels.toml* contains metadata for generated kernels. It's not
+  supposed to be edited by hands.
+
+- *kernels.user.toml* is default path for custom kernels definition.
+
+- *Ubuntu* (or *Centos*/*Debian*/...) is the Dockerfiles tree
+  (DistroName/DistroVersion/Dockerfile). Each Dockerfile contains a
+  base layer and incrementally updated list of kernels that must be
+  installed.
+
+Overview
+---------
+
+*out-of-tree* creating debugging environment based on **defined** kernels::
+
+    $ out-of-tree debug --kernel 'Ubuntu:4.15.0-58-generic'
+    [*] KASLR SMEP SMAP
+    [*] gdb is listening on tcp::1234
+    [*] build result copied to /tmp/exploit
+
+    ssh -o StrictHostKeyChecking=no -p 29308 root@127.133.45.236
+    gdb /usr/lib/debug/boot/vmlinux-4.15.0-58-generic -ex 'target remote tcp::1234'
+
+    out-of-tree> help
+    help    : print this help message
+    log     : print qemu log
+    clog    : print qemu log and cleanup buffer
+    cleanup : cleanup qemu log buffer
+    ssh     : print arguments to ssh command
+    quit    : quit
+    out-of-tree>
+
+*out-of-tree* uses three stages for automated runs:
+
+- Build
+
+  - Inside the docker container (default).
+  - Binary version (de facto skip stage).
+  - On host.
+
+- Run
+
+  - Insmod for the kernel module.
+  - This step is skipped for exploits.
+
+- Test
+
+  - Run the test.sh script on the target machine.
+  - Test script is run from *root* for the kernel module.
+  - Test script is run from *user* for the kernel exploit.
+  - Test script for the kernel module is fully custom (only return
+    value is checked).
+  - Test script for the kernel exploit receives two parameters:
+
+    - Path to exploit
+    - Path to file that must be created with root privileges.
+
+  - If there's no test.sh script then default
+    (``echo touch FILE | exploit``) one is used.
+
+Security
+--------
+
+*out-of-tree* is not supposed to be used on multi-user systems or with
+an untrusted input.
+
+Meanwhile, all modern hypervisors are supporting nested
+virtualization, which means you can use it for isolating *out-of-tree*
+if you want to work with an untrusted input (e.g. with a mass-scale
+testing public proofs-of-concept).

examples/kernel-exploit/.out-of-tree.toml

@@ -6,12 +6,12 @@ type = "exploit"
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.4.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99|100|101|102|103|104|105|106|107|108|109|110|111|112|113|114|115|116)-.*"
+release_mask = "4[.]4[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99|100|101|102|103|104|105|106|107|108|109|110|111|112|113|114|115|116)-.*"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.8.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58)-.*"
+release_mask = "4[.]8[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58)-.*"
 
 [[supported_kernels]]
 # Can be Ubuntu/CentOS/Debian/etc.
@@ -20,14 +20,19 @@ distro_release = "16.04"
 # regex for `uname -r`
 # See also: regex-golang.appspot.com
 # stupid way to generate: $ echo '4.4.0-('$(seq 44 | xargs echo | sed 's/ /|/g')')-.*'
-release_mask = "4.10.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42)-.*"
+release_mask = "4[.]10[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42)-.*"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.11.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14)-.*"
+release_mask = "4[.]11[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14)-.*"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.13.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21)-.*"
+# equivalent for "4[.]13[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21)-.*"
+[supported_kernels.kernel]
+version = [ 4 ]
+major = [ 13 ]
+minor = [ 0 ]
+patch = [ 1, 21 ]

examples/kernel-exploit/CVE-2017-16995.c

@@ -317,6 +317,7 @@ void redact(const char *fmt, ...) {
 	va_start(args, fmt);
 	if(doredact) {
 		fprintf(stdout, "[!] ( ( R E D A C T E D ) )\n");
+		va_end(args);
 		return;
 	}
 	fprintf(stdout, "[*] ");

examples/kernel-exploit/Makefile

@@ -2,7 +2,7 @@
 # - KERNEL: kernel headers path
 # - TARGET: name of exploit binary that MUST be produced by makefile.
 # - $(TARGET)_test: name of test binary that MUST be produced by makefile
-# and it's will be runned on a LPE stage. TARGET_TEST MUST accept two argument:
+# and it's will be executed on a LPE stage. TARGET_TEST MUST accept two argument:
 #   - Path to exploit binary
 #   - File that MUST be created with exploit. It uses for test that exploit works
 #     correctly.

examples/kernel-module/.out-of-tree.toml

@@ -9,17 +9,22 @@ distro_type = "Ubuntu"
 distro_release = "16.04"
 # regex for `uname -r`
 # See also: regex-golang.appspot.com
-release_mask = "4.4.0-70-.*"
+release_mask = "4[.]4[.]0-70-.*"
 
 # [[supported_kernels]] may be defined unlimited number of times
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "18.04"
 # Also you can use only one kernel
-release_mask = "4.15.0-(24|29)-generic"
+release_mask = "4[.]15[.]0-(24|29)-generic"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "18.04"
 # Also you can use only one kernel
-release_mask = "4.15.0-23-generic"
+release_mask = "4[.]15[.]0-23-generic"
+
+[[supported_kernels]]
+distro_type = "CentOS"
+distro_release = "7"
+release_mask = "3[.]10[.]0-862.el7.x86_64"

gen.go

@@ -7,8 +7,9 @@ package main
 import (
 	"fmt"
 
-	"github.com/jollheef/out-of-tree/config"
 	"github.com/naoina/toml"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
 )
 
 func genConfig(at config.ArtifactType) (err error) {
@@ -17,7 +18,9 @@ func genConfig(at config.ArtifactType) (err error) {
 		Type: at,
 	}
 	a.SupportedKernels = append(a.SupportedKernels, config.KernelMask{
-		config.Ubuntu, "18.04", ".*",
+		DistroType:    config.Ubuntu,
+		DistroRelease: "18.04",
+		ReleaseMask:   ".*",
 	})
 
 	buf, err := toml.Marshal(&a)

go.mod

@@ -0,0 +1,21 @@
+module code.dumpstack.io/tools/out-of-tree
+
+replace code.dumpstack.io/tools/out-of-tree/qemu => ./qemu
+
+replace code.dumpstack.io/tools/out-of-tree/config => ./config
+
+require (
+	github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc // indirect
+	github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf // indirect
+	github.com/mattn/go-runewidth v0.0.4 // indirect
+	github.com/mattn/go-sqlite3 v1.11.0
+	github.com/naoina/go-stringutil v0.1.0 // indirect
+	github.com/naoina/toml v0.1.1
+	github.com/olekukonko/tablewriter v0.0.1
+	github.com/otiai10/copy v1.0.1
+	github.com/remeh/sizedwaitgroup v0.0.0-20180822144253-5e7302b12cce
+	github.com/zcalusic/sysinfo v0.0.0-20190429151633-fbadb57345c2
+	golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5
+	gopkg.in/alecthomas/kingpin.v2 v2.2.6
+	gopkg.in/logrusorgru/aurora.v1 v1.0.0-20181002194514-a7b3b318ed4e
+)

go.sum

@@ -0,0 +1,33 @@
+bou.ke/monkey v1.0.1/go.mod h1:FgHuK96Rv2Nlf+0u1OOVDpCMdsWyOFmeeketDHE7LIg=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc h1:cAKDfWh5VpdgMhJosfJnn5/FoN2SRZ4p7fJNX58YPaU=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZqLG4oE62mJzwPIB8+Tee4RNCL9ulrY=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/mattn/go-runewidth v0.0.4 h1:2BvfKmzob6Bmd4YsL0zygOqfdFnK7GR4QL06Do4/p7Y=
+github.com/mattn/go-runewidth v0.0.4/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
+github.com/mattn/go-sqlite3 v1.11.0 h1:LDdKkqtYlom37fkvqs8rMPFKAMe8+SgjbwZ6ex1/A/Q=
+github.com/mattn/go-sqlite3 v1.11.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
+github.com/naoina/go-stringutil v0.1.0 h1:rCUeRUHjBjGTSHl0VC00jUPLz8/F9dDzYI70Hzifhks=
+github.com/naoina/go-stringutil v0.1.0/go.mod h1:XJ2SJL9jCtBh+P9q5btrd/Ylo8XwT/h1USek5+NqSA0=
+github.com/naoina/toml v0.1.1 h1:PT/lllxVVN0gzzSqSlHEmP8MJB4MY2U7STGxiouV4X8=
+github.com/naoina/toml v0.1.1/go.mod h1:NBIhNtsFMo3G2szEBne+bO4gS192HuIYRqfvOWb4i1E=
+github.com/olekukonko/tablewriter v0.0.1 h1:b3iUnf1v+ppJiOfNX4yxxqfWKMQPZR5yoh8urCTFX88=
+github.com/olekukonko/tablewriter v0.0.1/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo=
+github.com/otiai10/copy v1.0.1 h1:gtBjD8aq4nychvRZ2CyJvFWAw0aja+VHazDdruZKGZA=
+github.com/otiai10/copy v1.0.1/go.mod h1:8bMCJrAqOtN/d9oyh5HR7HhLQMvcGMpGdwRDYsfOCHc=
+github.com/otiai10/mint v1.2.3/go.mod h1:YnfyPNhBvnY8bW4SGQHCs/aAFhkgySlMZbrF5U0bOVw=
+github.com/remeh/sizedwaitgroup v0.0.0-20180822144253-5e7302b12cce h1:aP+C+YbHZfOQlutA4p4soHi7rVUqHQdWEVMSkHfDTqY=
+github.com/remeh/sizedwaitgroup v0.0.0-20180822144253-5e7302b12cce/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
+github.com/zcalusic/sysinfo v0.0.0-20190429151633-fbadb57345c2 h1:uMiaKNX5zFLOa6nNtun+d/lpV5bOBh7BvE4q9jfZacQ=
+github.com/zcalusic/sysinfo v0.0.0-20190429151633-fbadb57345c2/go.mod h1:zAn3FAIbgZPYnutDND49Ivf8sb/mXYk8UjZdqMswgHg=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5 h1:58fnuSXlxZmFdJyvtTFVmVhcMLU6v5fEb/ok4wyqtNU=
+golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/logrusorgru/aurora.v1 v1.0.0-20181002194514-a7b3b318ed4e h1:uKdf1KQDFZDYqNzSDhxB5hFxj5Fq4e3/C/ejtRJxlY0=
+gopkg.in/logrusorgru/aurora.v1 v1.0.0-20181002194514-a7b3b318ed4e/go.mod h1:DGR33jeYG1jxERD2W4hGjuW94Pxf3mkUf/Ddhf5BskA=

images.config.go

@@ -0,0 +1,7 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+const imagesBaseURL = "https://out-of-tree.fra1.digitaloceanspaces.com/1.0.0/"

images.go

@@ -0,0 +1,82 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"os"
+	"os/exec"
+)
+
+// inspired by Edd Turtle code
+func downloadFile(filepath string, url string) (err error) {
+	out, err := os.Create(filepath)
+	if err != nil {
+		return
+	}
+	defer out.Close()
+
+	resp, err := http.Get(url)
+	if err != nil {
+		return
+	}
+	defer resp.Body.Close()
+
+	switch resp.StatusCode {
+	case http.StatusOK:
+		break
+	case http.StatusForbidden, http.StatusNotFound:
+		err = fmt.Errorf("Cannot download %s. It looks like you need "+
+			"to generate it manually and place it "+
+			"to ~/.out-of-tree/images/. "+
+			"Check documentation for additional information.", url)
+		return
+	default:
+		err = fmt.Errorf("Something weird happens while "+
+			"download file: %d", resp.StatusCode)
+		return
+	}
+
+	_, err = io.Copy(out, resp.Body)
+	return
+}
+
+func unpackTar(archive, destination string) (err error) {
+	// NOTE: If you're change anything in tar command please check also
+	// BSD tar (or if you're using macOS, do not forget to check GNU Tar)
+	// Also make sure that sparse files are extracting correctly
+	cmd := exec.Command("tar", "-Sxf", archive)
+	cmd.Dir = destination + "/"
+
+	rawOutput, err := cmd.CombinedOutput()
+	if err != nil {
+		err = fmt.Errorf("%v: %s", err, rawOutput)
+		return
+	}
+
+	return
+}
+
+func downloadImage(path, file string) (err error) {
+	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_")
+	if err != nil {
+		return
+	}
+	defer os.RemoveAll(tmp)
+
+	archive := tmp + "/" + file + ".tar.gz"
+	url := imagesBaseURL + file + ".tar.gz"
+
+	err = downloadFile(archive, url)
+	if err != nil {
+		return
+	}
+
+	err = unpackTar(archive, path)
+	return
+}

kernel.go

@@ -9,16 +9,23 @@ import (
 	"fmt"
 	"io/ioutil"
 	"log"
+	"math"
+	"math/rand"
 	"os"
 	"os/exec"
 	"os/user"
 	"regexp"
 	"strings"
+	"time"
 
-	"github.com/jollheef/out-of-tree/config"
 	"github.com/naoina/toml"
+	"github.com/zcalusic/sysinfo"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
 )
 
+const kernelsAll int64 = math.MaxInt64
+
 func kernelListHandler(kcfg config.KernelConfig) (err error) {
 	if len(kcfg.Kernels) == 0 {
 		return errors.New("No kernels found")
@@ -29,34 +36,58 @@ func kernelListHandler(kcfg config.KernelConfig) (err error) {
 	return
 }
 
-func matchDebianKernelPkg(container, mask string, generic bool) (pkgs []string,
-	err error) {
+func matchDebianHeadersPkg(container, mask string, generic bool) (
+	pkgs []string, err error) {
 
-	cmd := "apt-cache search linux-image | cut -d ' ' -f 1"
-	c := dockerCommand(container, "/tmp", "1m", cmd)
-	rawOutput, err := c.CombinedOutput()
+	cmd := "apt-cache search linux-headers | cut -d ' ' -f 1"
+	output, err := dockerRun(time.Minute, container, "/tmp", cmd)
 	if err != nil {
 		return
 	}
 
-	r, err := regexp.Compile("linux-image-" + mask)
+	r, err := regexp.Compile("linux-headers-" + mask)
 	if err != nil {
 		return
 	}
 
-	kernels := r.FindAll(rawOutput, -1)
+	kernels := r.FindAll([]byte(output), -1)
 
 	for _, k := range kernels {
 		pkg := string(k)
 		if generic && !strings.HasSuffix(pkg, "generic") {
 			continue
 		}
+		if pkg == "linux-headers-generic" {
+			continue
+		}
 		pkgs = append(pkgs, pkg)
 	}
 
 	return
 }
 
+func matchCentOSDevelPkg(container, mask string, generic bool) (
+	pkgs []string, err error) {
+
+	cmd := "yum search kernel-devel --show-duplicates | " +
+		"grep '^kernel-devel' | cut -d ' ' -f 1"
+	output, err := dockerRun(time.Minute, container, "/tmp", cmd)
+	if err != nil {
+		return
+	}
+
+	r, err := regexp.Compile("kernel-devel-" + mask)
+	if err != nil {
+		return
+	}
+
+	for _, k := range r.FindAll([]byte(output), -1) {
+		pkgs = append(pkgs, string(k))
+	}
+
+	return
+}
+
 func dockerImagePath(sk config.KernelMask) (path string, err error) {
 	usr, err := user.Current()
 	if err != nil {
@@ -68,7 +99,19 @@ func dockerImagePath(sk config.KernelMask) (path string, err error) {
 	return
 }
 
-func generateBaseDockerImage(sk config.KernelMask) (err error) {
+func vsyscallAvailable() (available bool, err error) {
+	buf, err := ioutil.ReadFile("/proc/self/maps")
+	if err != nil {
+		return
+	}
+
+	available = strings.Contains(string(buf), "[vsyscall]")
+	return
+}
+
+func generateBaseDockerImage(registry string, commands []config.DockerCommand,
+	sk config.KernelMask) (err error) {
+
 	imagePath, err := dockerImagePath(sk)
 	if err != nil {
 		return
@@ -81,26 +124,72 @@ func generateBaseDockerImage(sk config.KernelMask) (err error) {
 		log.Printf("Base image for %s:%s found",
 			sk.DistroType.String(), sk.DistroRelease)
 		return
-	} else {
-		log.Printf("Base image for %s:%s not found, start generating",
-			sk.DistroType.String(), sk.DistroRelease)
-		os.MkdirAll(imagePath, os.ModePerm)
 	}
 
-	d += fmt.Sprintf("FROM %s:%s\n",
+	log.Printf("Base image for %s:%s not found, start generating",
+		sk.DistroType.String(), sk.DistroRelease)
+	os.MkdirAll(imagePath, os.ModePerm)
+
+	d += "FROM "
+	if registry != "" {
+		d += registry + "/"
+	}
+
+	d += fmt.Sprintf("%s:%s\n",
 		strings.ToLower(sk.DistroType.String()),
 		sk.DistroRelease,
 	)
 
+	vsyscall, err := vsyscallAvailable()
+	if err != nil {
+		return
+	}
+
+	for _, c := range commands {
+		switch c.DistroType {
+		case config.Ubuntu:
+			d += "RUN " + c.Command + "\n"
+		case config.CentOS:
+			d += "RUN " + c.Command + "\n"
+		case config.Debian:
+			d += "RUN " + c.Command + "\n"
+		default:
+			err = fmt.Errorf("%s not yet supported",
+				sk.DistroType.String())
+			return
+		}
+	}
+
 	switch sk.DistroType {
 	case config.Ubuntu:
 		d += "ENV DEBIAN_FRONTEND=noninteractive\n"
 		d += "RUN apt-get update\n"
 		d += "RUN apt-get install -y build-essential libelf-dev\n"
 		d += "RUN apt-get install -y wget git\n"
+		if sk.DistroRelease >= "14.04" {
+			d += "RUN apt-get install -y libseccomp-dev\n"
+		}
+		d += "RUN mkdir -p /lib/modules\n"
+	case config.CentOS:
+		if sk.DistroRelease < "7" && !vsyscall {
+			log.Println("Old CentOS requires `vsyscall=emulate` " +
+				"on the latest kernels")
+			log.Println("Check out `A note about vsyscall` " +
+				"at https://hub.docker.com/_/centos")
+			log.Println("See also https://lwn.net/Articles/446528/")
+			err = fmt.Errorf("vsyscall is not available")
+			return
+		}
+
+		// enable rpms from old minor releases
+		d += "RUN sed -i 's/enabled=0/enabled=1/' /etc/yum.repos.d/CentOS-Vault.repo\n"
+		// do not remove old kernels
+		d += "RUN sed -i 's;installonly_limit=;installonly_limit=100500;' /etc/yum.conf\n"
+		d += "RUN yum -y update\n"
+		d += "RUN yum -y groupinstall 'Development Tools'\n"
+		d += "RUN yum -y install deltarpm\n"
 	default:
-		s := fmt.Sprintf("%s not yet supported", sk.DistroType.String())
-		err = errors.New(s)
+		err = fmt.Errorf("%s not yet supported", sk.DistroType.String())
 		return
 	}
 
@@ -144,11 +233,32 @@ func dockerImageAppend(sk config.KernelMask, pkgname string) (err error) {
 		return
 	}
 
-	log.Printf("Start adding kernel %s for %s:%s",
-		pkgname, sk.DistroType.String(), sk.DistroRelease)
+	var s string
 
-	s := fmt.Sprintf("RUN apt-get install -y %s %s\n", pkgname,
-		strings.Replace(pkgname, "image", "headers", -1))
+	switch sk.DistroType {
+	case config.Ubuntu:
+		imagepkg := strings.Replace(pkgname, "headers", "image", -1)
+
+		log.Printf("Start adding kernel %s for %s:%s",
+			imagepkg, sk.DistroType.String(), sk.DistroRelease)
+
+		s = fmt.Sprintf("RUN apt-get install -y %s %s\n", imagepkg,
+			pkgname)
+	case config.CentOS:
+		imagepkg := strings.Replace(pkgname, "-devel", "", -1)
+
+		log.Printf("Start adding kernel %s for %s:%s",
+			imagepkg, sk.DistroType.String(), sk.DistroRelease)
+
+		version := strings.Replace(pkgname, "kernel-devel-", "", -1)
+
+		s = fmt.Sprintf("RUN yum -y install %s %s\n", imagepkg,
+			pkgname)
+		s += fmt.Sprintf("RUN dracut --add-drivers 'e1000 ext4' -f "+
+			"/boot/initramfs-%s.img %s\n", version, version)
+	default:
+		err = fmt.Errorf("%s not yet supported", sk.DistroType.String())
+	}
 
 	err = ioutil.WriteFile(imagePath+"/Dockerfile",
 		append(raw, []byte(s)...), 0644)
@@ -226,7 +336,7 @@ func copyKernels(name string) (err error) {
 
 func genKernelPath(files []os.FileInfo, kname string) string {
 	for _, file := range files {
-		if strings.Contains(file.Name(), "vmlinuz") {
+		if strings.HasPrefix(file.Name(), "vmlinuz") {
 			if strings.Contains(file.Name(), kname) {
 				return file.Name()
 			}
@@ -237,7 +347,9 @@ func genKernelPath(files []os.FileInfo, kname string) string {
 
 func genInitrdPath(files []os.FileInfo, kname string) string {
 	for _, file := range files {
-		if strings.Contains(file.Name(), "initrd") {
+		if strings.HasPrefix(file.Name(), "initrd") ||
+			strings.HasPrefix(file.Name(), "initramfs") {
+
 			if strings.Contains(file.Name(), kname) {
 				return file.Name()
 			}
@@ -246,13 +358,24 @@ func genInitrdPath(files []os.FileInfo, kname string) string {
 	return "unknown"
 }
 
-func genRootfsImage(d dockerImageInfo) string {
+func genRootfsImage(d dockerImageInfo, download bool) (rootfs string, err error) {
 	usr, err := user.Current()
 	if err != nil {
-		return fmt.Sprintln(err)
+		return
 	}
 	imageFile := d.ContainerName + ".img"
-	return usr.HomeDir + "/.out-of-tree/images/" + imageFile
+
+	imagesPath := usr.HomeDir + "/.out-of-tree/images/"
+	os.MkdirAll(imagesPath, os.ModePerm)
+
+	rootfs = imagesPath + imageFile
+	if !exists(rootfs) {
+		if download {
+			log.Println(imageFile, "not exists, start downloading...")
+			err = downloadImage(imagesPath, imageFile)
+		}
+	}
+	return
 }
 
 type dockerImageInfo struct {
@@ -296,16 +419,85 @@ func listDockerImages() (diis []dockerImageInfo, err error) {
 	return
 }
 
-func updateKernelsCfg() (err error) {
+func genHostKernels(download bool) (kcfg config.KernelConfig, err error) {
+	si := sysinfo.SysInfo{}
+	si.GetSysInfo()
+
+	distroType, err := config.NewDistroType(si.OS.Vendor)
+	if err != nil {
+		return
+	}
+
+	cmd := exec.Command("ls", "/lib/modules")
+	rawOutput, err := cmd.CombinedOutput()
+	if err != nil {
+		log.Println(string(rawOutput), err)
+		return
+	}
+
+	kernelsBase := "/boot/"
+	files, err := ioutil.ReadDir(kernelsBase)
+	if err != nil {
+		return
+	}
+
+	// only for compatibility, docker is not really used
+	dii := dockerImageInfo{
+		ContainerName: config.KernelMask{
+			DistroType:    distroType,
+			DistroRelease: si.OS.Version,
+		}.DockerName(),
+	}
+
+	rootfs, err := genRootfsImage(dii, download)
+	if err != nil {
+		return
+	}
+
+	for _, k := range strings.Fields(string(rawOutput)) {
+		ki := config.KernelInfo{
+			DistroType:    distroType,
+			DistroRelease: si.OS.Version,
+			KernelRelease: k,
+
+			KernelSource: "/lib/modules/" + k + "/build",
+
+			KernelPath: kernelsBase + genKernelPath(files, k),
+			InitrdPath: kernelsBase + genInitrdPath(files, k),
+			RootFS:     rootfs,
+		}
+
+		vmlinux := "/usr/lib/debug/boot/vmlinux-" + k
+		log.Println("vmlinux", vmlinux)
+		if exists(vmlinux) {
+			ki.VmlinuxPath = vmlinux
+		}
+
+		kcfg.Kernels = append(kcfg.Kernels, ki)
+	}
+
+	return
+}
+
+func updateKernelsCfg(host, download bool) (err error) {
+	newkcfg := config.KernelConfig{}
+
+	if host {
+		// Get host kernels
+		newkcfg, err = genHostKernels(download)
+		if err != nil {
+			return
+		}
+	}
+
+	// Get docker kernels
 	dockerImages, err := listDockerImages()
 	if err != nil {
 		return
 	}
 
-	newkcfg := config.KernelConfig{}
-
 	for _, d := range dockerImages {
-		err = genKernels(d, &newkcfg)
+		err = genDockerKernels(d, &newkcfg, download)
 		if err != nil {
 			log.Println("gen kernels", d.ContainerName, ":", err)
 			continue
@@ -342,8 +534,8 @@ func updateKernelsCfg() (err error) {
 	return
 }
 
-func genKernels(dii dockerImageInfo, newkcfg *config.KernelConfig) (
-	err error) {
+func genDockerKernels(dii dockerImageInfo, newkcfg *config.KernelConfig,
+	download bool) (err error) {
 
 	name := dii.ContainerName
 	cmd := exec.Command("docker", "run", name, "ls", "/lib/modules")
@@ -363,6 +555,11 @@ func genKernels(dii dockerImageInfo, newkcfg *config.KernelConfig) (
 		return
 	}
 
+	rootfs, err := genRootfsImage(dii, download)
+	if err != nil {
+		return
+	}
+
 	for _, k := range strings.Fields(string(rawOutput)) {
 		ki := config.KernelInfo{
 			DistroType:    dii.DistroType,
@@ -372,7 +569,7 @@ func genKernels(dii dockerImageInfo, newkcfg *config.KernelConfig) (
 
 			KernelPath: kernelsBase + genKernelPath(files, k),
 			InitrdPath: kernelsBase + genInitrdPath(files, k),
-			RootFS:     genRootfsImage(dii),
+			RootFS:     rootfs,
 		}
 		newkcfg.Kernels = append(newkcfg.Kernels, ki)
 	}
@@ -389,7 +586,81 @@ func hasKernel(ki config.KernelInfo, kcfg config.KernelConfig) bool {
 	return false
 }
 
-func kernelAutogenHandler(workPath string) (err error) {
+func shuffle(a []string) []string {
+	// Fisher–Yates shuffle
+	for i := len(a) - 1; i > 0; i-- {
+		j := rand.Intn(i + 1)
+		a[i], a[j] = a[j], a[i]
+	}
+	return a
+}
+
+func generateKernels(km config.KernelMask, registry string,
+	commands []config.DockerCommand, max int64,
+	download bool) (err error) {
+
+	log.Println("Generating for kernel mask", km)
+
+	_, err = genRootfsImage(dockerImageInfo{ContainerName: km.DockerName()},
+		download)
+	if err != nil {
+		return
+	}
+
+	err = generateBaseDockerImage(registry, commands, km)
+	if err != nil {
+		return
+	}
+
+	var pkgs []string
+	switch km.DistroType {
+	case config.Ubuntu:
+		pkgs, err = matchDebianHeadersPkg(km.DockerName(),
+			km.ReleaseMask, true)
+	case config.CentOS:
+		pkgs, err = matchCentOSDevelPkg(km.DockerName(),
+			km.ReleaseMask, true)
+	default:
+		err = fmt.Errorf("%s not yet supported", km.DistroType.String())
+	}
+	if err != nil {
+		return
+	}
+
+	for i, pkg := range shuffle(pkgs) {
+		if max <= 0 {
+			log.Println("Max is reached")
+			break
+		}
+
+		log.Println(i, "/", len(pkgs), pkg)
+
+		err = dockerImageAppend(km, pkg)
+		if err == nil {
+			max--
+		} else {
+			log.Println("dockerImageAppend", err)
+		}
+	}
+
+	err = kickImage(km.DockerName())
+	if err != nil {
+		log.Println("kick image", km.DockerName(), ":", err)
+		return
+	}
+
+	err = copyKernels(km.DockerName())
+	if err != nil {
+		log.Println("copy kernels", km.DockerName(), ":", err)
+		return
+	}
+	return
+}
+
+func kernelAutogenHandler(workPath, registry string,
+	commands []config.DockerCommand,
+	max int64, host, download bool) (err error) {
+
 	ka, err := config.ReadArtifactConfig(workPath + "/.out-of-tree.toml")
 	if err != nil {
 		return
@@ -401,40 +672,17 @@ func kernelAutogenHandler(workPath string) (err error) {
 			return
 		}
 
-		err = generateBaseDockerImage(sk)
+		err = generateKernels(sk, registry, commands, max, download)
 		if err != nil {
 			return
 		}
-
-		var pkgs []string
-		pkgs, err = matchDebianKernelPkg(sk.DockerName(),
-			sk.ReleaseMask, true)
-		if err != nil {
-			return
-		}
-
-		for _, pkg := range pkgs {
-			dockerImageAppend(sk, pkg)
-		}
-
-		err = kickImage(sk.DockerName())
-		if err != nil {
-			log.Println("kick image", sk.DockerName(), ":", err)
-			continue
-		}
-
-		err = copyKernels(sk.DockerName())
-		if err != nil {
-			log.Println("copy kernels", sk.DockerName(), ":", err)
-			continue
-		}
 	}
 
-	err = updateKernelsCfg()
+	err = updateKernelsCfg(host, download)
 	return
 }
 
-func kernelDockerRegenHandler() (err error) {
+func kernelDockerRegenHandler(host, download bool) (err error) {
 	dockerImages, err := listDockerImages()
 	if err != nil {
 		return
@@ -472,5 +720,26 @@ func kernelDockerRegenHandler() (err error) {
 		}
 	}
 
-	return updateKernelsCfg()
+	return updateKernelsCfg(host, download)
+}
+
+func kernelGenallHandler(distro, version, registry string,
+	commands []config.DockerCommand, host, download bool) (err error) {
+
+	distroType, err := config.NewDistroType(distro)
+	if err != nil {
+		return
+	}
+
+	km := config.KernelMask{
+		DistroType:    distroType,
+		DistroRelease: version,
+		ReleaseMask:   ".*",
+	}
+	err = generateKernels(km, registry, commands, kernelsAll, download)
+	if err != nil {
+		return
+	}
+
+	return updateKernelsCfg(host, download)
 }

log.go

@@ -0,0 +1,245 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"database/sql"
+	"encoding/json"
+	"fmt"
+	"log"
+	"math"
+	"os"
+
+	"github.com/olekukonko/tablewriter"
+	"gopkg.in/logrusorgru/aurora.v1"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
+)
+
+func logLogEntry(l logEntry) {
+	distroInfo := fmt.Sprintf("%s-%s {%s}", l.DistroType,
+		l.DistroRelease, l.KernelRelease)
+
+	artifactInfo := fmt.Sprintf("{[%s] %s}", l.Type, l.Name)
+
+	colored := ""
+	if l.Type == config.KernelExploit {
+		colored = aurora.Sprintf("[%4d %4s] [%s] %40s %40s: %s %s",
+			l.ID, l.Tag, l.Timestamp, artifactInfo, distroInfo,
+			genOkFail("BUILD", l.Build.Ok),
+			genOkFail("LPE", l.Test.Ok))
+	} else {
+		colored = aurora.Sprintf("[%4d %4s] [%s] %40s %40s: %s %s %s",
+			l.ID, l.Tag, l.Timestamp, artifactInfo, distroInfo,
+			genOkFail("BUILD", l.Build.Ok),
+			genOkFail("INSMOD", l.Run.Ok),
+			genOkFail("TEST", l.Test.Ok))
+	}
+
+	additional := ""
+	if l.KernelPanic {
+		additional = "(panic)"
+	} else if l.KilledByTimeout {
+		additional = "(timeout)"
+	}
+
+	if additional != "" {
+		fmt.Println(colored, additional)
+	} else {
+		fmt.Println(colored)
+	}
+}
+
+func logHandler(db *sql.DB, path, tag string, num int, rate bool) (err error) {
+	var les []logEntry
+
+	ka, kaErr := config.ReadArtifactConfig(path + "/.out-of-tree.toml")
+	if kaErr == nil {
+		log.Println(".out-of-tree.toml found, filter by artifact name")
+		les, err = getAllArtifactLogs(db, tag, num, ka)
+	} else {
+		les, err = getAllLogs(db, tag, num)
+	}
+	if err != nil {
+		return
+	}
+
+	s := "\nS"
+	if rate {
+		if kaErr != nil {
+			err = kaErr
+			return
+		}
+
+		s = fmt.Sprintf("{[%s] %s} Overall s", ka.Type, ka.Name)
+
+		les, err = getAllArtifactLogs(db, tag, math.MaxInt64, ka)
+		if err != nil {
+			return
+		}
+	} else {
+		for _, l := range les {
+			logLogEntry(l)
+		}
+	}
+
+	success := 0
+	for _, l := range les {
+		if l.Test.Ok {
+			success++
+		}
+	}
+
+	overall := float64(success) / float64(len(les))
+	fmt.Printf("%success rate: %.04f (~%.0f%%)\n",
+		s, overall, overall*100)
+
+	return
+}
+
+func logDumpHandler(db *sql.DB, id int) (err error) {
+	l, err := getLogByID(db, id)
+	if err != nil {
+		return
+	}
+
+	fmt.Println("ID:", l.ID)
+	fmt.Println("Date:", l.Timestamp)
+	fmt.Println("Tag:", l.Tag)
+	fmt.Println()
+
+	fmt.Println("Type:", l.Type.String())
+	fmt.Println("Name:", l.Name)
+	fmt.Println()
+
+	fmt.Println("Distro:", l.DistroType.String(), l.DistroRelease)
+	fmt.Println("Kernel:", l.KernelRelease)
+	fmt.Println()
+
+	fmt.Println("Build ok:", l.Build.Ok)
+	if l.Type == config.KernelModule {
+		fmt.Println("Insmod ok:", l.Run.Ok)
+	}
+	fmt.Println("Test ok:", l.Test.Ok)
+	fmt.Println()
+
+	fmt.Printf("Build output:\n%s\n", l.Build.Output)
+	fmt.Println()
+
+	if l.Type == config.KernelModule {
+		fmt.Printf("Insmod output:\n%s\n", l.Run.Output)
+		fmt.Println()
+	}
+
+	fmt.Printf("Test output:\n%s\n", l.Test.Output)
+	fmt.Println()
+
+	fmt.Printf("Qemu stdout:\n%s\n", l.Stdout)
+	fmt.Println()
+
+	fmt.Printf("Qemu stderr:\n%s\n", l.Stderr)
+	fmt.Println()
+
+	return
+}
+
+type runstat struct {
+	All, BuildOK, RunOK, TestOK, Timeout, Panic int
+}
+
+func getStats(db *sql.DB, path, tag string) (
+	distros map[string]map[string]map[string]runstat, err error) {
+
+	var les []logEntry
+
+	ka, kaErr := config.ReadArtifactConfig(path + "/.out-of-tree.toml")
+	if kaErr == nil {
+		les, err = getAllArtifactLogs(db, tag, -1, ka)
+	} else {
+		les, err = getAllLogs(db, tag, -1)
+	}
+	if err != nil {
+		return
+	}
+
+	distros = make(map[string]map[string]map[string]runstat)
+
+	for _, l := range les {
+		_, ok := distros[l.DistroType.String()]
+		if !ok {
+			distros[l.DistroType.String()] = make(map[string]map[string]runstat)
+		}
+
+		_, ok = distros[l.DistroType.String()][l.DistroRelease]
+		if !ok {
+			distros[l.DistroType.String()][l.DistroRelease] = make(map[string]runstat)
+		}
+
+		rs := distros[l.DistroType.String()][l.DistroRelease][l.KernelRelease]
+
+		rs.All++
+		if l.Build.Ok {
+			rs.BuildOK++
+		}
+		if l.Run.Ok {
+			rs.RunOK++
+		}
+		if l.Test.Ok {
+			rs.TestOK++
+		}
+		if l.KernelPanic {
+			rs.Panic++
+		}
+		if l.KilledByTimeout {
+			rs.Timeout++
+		}
+
+		distros[l.DistroType.String()][l.DistroRelease][l.KernelRelease] = rs
+	}
+
+	return
+}
+
+func logJSONHandler(db *sql.DB, path, tag string) (err error) {
+	distros, err := getStats(db, path, tag)
+	if err != nil {
+		return
+	}
+
+	bytes, err := json.Marshal(&distros)
+	if err != nil {
+		return
+	}
+
+	fmt.Println(string(bytes))
+	return
+}
+
+func logMarkdownHandler(db *sql.DB, path, tag string) (err error) {
+	distros, err := getStats(db, path, tag)
+	if err != nil {
+		return
+	}
+
+	table := tablewriter.NewWriter(os.Stdout)
+	table.SetHeader([]string{"Distro", "Release", "Kernel", "Reliability"})
+	table.SetBorders(tablewriter.Border{
+		Left: true, Top: false, Right: true, Bottom: false})
+	table.SetCenterSeparator("|")
+
+	for distro, releases := range distros {
+		for release, kernels := range releases {
+			for kernel, stats := range kernels {
+				all := float64(stats.All)
+				ok := float64(stats.TestOK)
+				r := fmt.Sprintf("%6.02f%%", (ok/all)*100)
+				table.Append([]string{distro, release, kernel, r})
+			}
+		}
+	}
+
+	table.Render()
+	return
+}

main.go

@@ -5,24 +5,86 @@
 package main
 
 import (
+	"fmt"
 	"log"
+	"math/rand"
 	"os"
 	"os/exec"
 	"os/user"
+	"runtime"
+	"sort"
+	"strconv"
+	"time"
 
 	kingpin "gopkg.in/alecthomas/kingpin.v2"
 
-	"github.com/jollheef/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/config"
 )
 
+func findFallback(kcfg config.KernelConfig, ki config.KernelInfo) (rootfs string) {
+	for _, k := range kcfg.Kernels {
+		if !exists(k.RootFS) || k.DistroType != ki.DistroType {
+			continue
+		}
+		if k.RootFS < ki.RootFS {
+			rootfs = k.RootFS
+			return
+		}
+	}
+	return
+}
+
+func handleFallbacks(kcfg config.KernelConfig) {
+	sort.Sort(sort.Reverse(config.ByRootFS(kcfg.Kernels)))
+
+	for i, k := range kcfg.Kernels {
+		if !exists(k.RootFS) {
+			newRootFS := findFallback(kcfg, k)
+
+			s := k.RootFS + " does not exists "
+			if newRootFS != "" {
+				s += "(fallback to " + newRootFS + ")"
+			} else {
+				s += "(no fallback found)"
+			}
+
+			kcfg.Kernels[i].RootFS = newRootFS
+			log.Println(s)
+		}
+	}
+}
+
+func checkRequiredUtils() (err error) {
+	// Check for required commands
+	for _, cmd := range []string{"docker", "qemu-system-x86_64"} {
+		_, err := exec.Command("which", cmd).CombinedOutput()
+		if err != nil {
+			return fmt.Errorf("Command not found: %s", cmd)
+		}
+	}
+	return
+}
+
+func checkDockerPermissions() (err error) {
+	output, err := exec.Command("docker", "ps").CombinedOutput()
+	if err != nil {
+		err = fmt.Errorf("%s", output)
+	}
+	return
+}
+
 func main() {
+	log.SetFlags(log.Lshortfile)
+
+	rand.Seed(time.Now().UnixNano())
+
 	app := kingpin.New(
 		"out-of-tree",
 		"kernel {module, exploit} development tool",
 	)
 
-	app.Author("Mikhail Klementev <jollheef@riseup.net>")
-	app.Version("0.1.0")
+	app.Author("Mikhail Klementev <root@dumpstack.io>")
+	app.Version("1.1.0")
 
 	pathFlag := app.Flag("path", "Path to work directory")
 	path := pathFlag.Default(".").ExistingDir()
@@ -31,22 +93,47 @@ func main() {
 	if err != nil {
 		return
 	}
-	defaultKcfgPath := usr.HomeDir + "/.out-of-tree/kernels.toml"
+	os.MkdirAll(usr.HomeDir+"/.out-of-tree", os.ModePerm)
+
+	confPath := usr.HomeDir + "/.out-of-tree/out-of-tree.toml"
+	conf, err := config.ReadOutOfTreeConf(confPath)
+	if err != nil {
+		return
+	}
 
 	kcfgPathFlag := app.Flag("kernels", "Path to main kernels config")
-	kcfgPath := kcfgPathFlag.Default(defaultKcfgPath).ExistingFile()
+	kcfgPath := kcfgPathFlag.Default(conf.Kernels).String()
+
+	dbPathFlag := app.Flag("db", "Path to database")
+	dbPath := dbPathFlag.Default(conf.Database).String()
 
-	defaultUserKcfgPath := usr.HomeDir + "/.out-of-tree/kernels.user.toml"
 	userKcfgPathFlag := app.Flag("user-kernels", "User kernels config")
 	userKcfgPathEnv := userKcfgPathFlag.Envar("OUT_OF_TREE_KCFG")
-	userKcfgPath := userKcfgPathEnv.Default(defaultUserKcfgPath).String()
+	userKcfgPath := userKcfgPathEnv.Default(conf.UserKernels).String()
+
+	timeoutFlag := app.Flag("timeout", "Timeout after tool will not spawn new tests")
+	timeout := timeoutFlag.Duration()
 
 	qemuTimeoutFlag := app.Flag("qemu-timeout", "Timeout for qemu")
-	qemuTimeout := qemuTimeoutFlag.Default("1m").Duration()
+	qemuTimeout := qemuTimeoutFlag.Default(conf.Qemu.Timeout).Duration()
 
 	dockerTimeoutFlag := app.Flag("docker-timeout", "Timeout for docker")
-	dockerTimeout := dockerTimeoutFlag.Default("1m").Duration()
+	dockerTimeout := dockerTimeoutFlag.Default(conf.Docker.Timeout).Duration()
+
+	dockerRegistryFlag := app.Flag("docker-registry", "Registry for docker")
+	dockerRegistry := dockerRegistryFlag.Default(conf.Docker.Registry).String()
+
+	thresholdFlag := app.Flag("threshold", "Reliablity threshold for exit code")
+	threshold := thresholdFlag.Default("1.00").Float64()
+
 	pewCommand := app.Command("pew", "Build, run and test module/exploit")
+
+	pewMax := pewCommand.Flag("max", "Test no more than X kernels").
+		PlaceHolder("X").Default(fmt.Sprint(kernelsAll)).Int64()
+
+	pewRuns := pewCommand.Flag("runs", "Runs per each kernel").
+		Default("1").Int64()
+
 	pewKernelFlag := pewCommand.Flag("kernel", "Override kernel regex")
 	pewKernel := pewKernelFlag.String()
 
@@ -59,12 +146,37 @@ func main() {
 	pewTestFlag := pewCommand.Flag("test", "Override path test")
 	pewTest := pewTestFlag.String()
 
+	pewDistFlag := pewCommand.Flag("dist", "Build result path")
+	pewDist := pewDistFlag.Default(pathDevNull).String()
+
+	pewThreadsFlag := pewCommand.Flag("threads", "Build result path")
+	pewThreads := pewThreadsFlag.Default(strconv.Itoa(runtime.NumCPU())).Int()
+
+	pewTagFlag := pewCommand.Flag("tag", "Log tagging")
+	pewTag := pewTagFlag.String()
+
 	kernelCommand := app.Command("kernel", "Manipulate kernels")
+	kernelNoDownload := kernelCommand.Flag("no-download",
+		"Do not download qemu image while kernel generation").Bool()
+	kernelUseHost := kernelCommand.Flag("host", "Use also host kernels").Bool()
 	kernelListCommand := kernelCommand.Command("list", "List kernels")
 	kernelAutogenCommand := kernelCommand.Command("autogen",
 		"Generate kernels based on a current config")
+	kernelAutogenMax := kernelAutogenCommand.Flag("max",
+		"Download random kernels from set defined by regex in "+
+			"release_mask, but no more than X for each of "+
+			"release_mask").PlaceHolder("X").Default(
+		fmt.Sprint(kernelsAll)).Int64()
 	kernelDockerRegenCommand := kernelCommand.Command("docker-regen",
 		"Regenerate kernels config from out_of_tree_* docker images")
+	kernelGenallCommand := kernelCommand.Command("genall",
+		"Generate all kernels for distro")
+
+	genallDistroFlag := kernelGenallCommand.Flag("distro", "Distributive")
+	distro := genallDistroFlag.Required().String()
+
+	genallVerFlag := kernelGenallCommand.Flag("ver", "Distro version")
+	version := genallVerFlag.Required().String()
 
 	genCommand := app.Command("gen", "Generate .out-of-tree.toml skeleton")
 	genModuleCommand := genCommand.Command("module",
@@ -78,22 +190,83 @@ func main() {
 	debugFlagGDB := debugCommand.Flag("gdb", "Set gdb listen address")
 	debugGDB := debugFlagGDB.Default("tcp::1234").String()
 
-	bootstrapCommand := app.Command("bootstrap",
-		"Create directories && download images")
+	yekaslr := debugCommand.Flag("enable-kaslr", "Enable KASLR").Bool()
+	yesmep := debugCommand.Flag("enable-smep", "Enable SMEP").Bool()
+	yesmap := debugCommand.Flag("enable-smap", "Enable SMAP").Bool()
+	yekpti := debugCommand.Flag("enable-kpti", "Enable KPTI").Bool()
 
-	// Check for required commands
-	for _, cmd := range []string{"timeout", "docker", "qemu"} {
-		_, err := exec.Command("which", cmd).CombinedOutput()
-		if err != nil {
-			log.Fatalln("Command not found:", cmd)
-		}
+	nokaslr := debugCommand.Flag("disable-kaslr", "Disable KASLR").Bool()
+	nosmep := debugCommand.Flag("disable-smep", "Disable SMEP").Bool()
+	nosmap := debugCommand.Flag("disable-smap", "Disable SMAP").Bool()
+	nokpti := debugCommand.Flag("disable-kpti", "Disable KPTI").Bool()
+
+	bootstrapCommand := app.Command("bootstrap", "Apparently nothing")
+
+	logCommand := app.Command("log", "Logs")
+
+	logQueryCommand := logCommand.Command("query", "Query logs")
+	logNum := logQueryCommand.Flag("num", "How much lines").Default("50").Int()
+	logRate := logQueryCommand.Flag("rate", "Show artifact success rate").Bool()
+	logTag := logQueryCommand.Flag("tag", "Filter tag").String()
+
+	logDumpCommand := logCommand.Command("dump",
+		"Show all info for log entry with ID")
+	logDumpID := logDumpCommand.Arg("ID", "").Required().Int()
+
+	logJSONCommand := logCommand.Command("json", "Generate json statistics")
+	logJSONTag := logJSONCommand.Flag("tag", "Filter tag").Required().String()
+
+	logMarkdownCommand := logCommand.Command("markdown", "Generate markdown statistics")
+	logMarkdownTag := logMarkdownCommand.Flag("tag", "Filter tag").Required().String()
+
+	packCommand := app.Command("pack", "Exploit pack test")
+	packAutogen := packCommand.Flag("autogen", "Kernel autogeneration").Bool()
+	packNoDownload := packCommand.Flag("no-download",
+		"Do not download qemu image while kernel generation").Bool()
+	packExploitRuns := packCommand.Flag("exploit-runs",
+		"Amount of runs of each exploit").Default("4").Int64()
+	packKernelRuns := packCommand.Flag("kernel-runs",
+		"Amount of runs of each kernel").Default("1").Int64()
+
+	err = checkRequiredUtils()
+	if err != nil {
+		log.Fatalln(err)
+	}
+
+	err = checkDockerPermissions()
+	if err != nil {
+		log.Println(err)
+		log.Println("You have two options:")
+		log.Println("\t1. Add user to group docker;")
+		log.Println("\t2. Run out-of-tree with sudo.")
+		os.Exit(1)
+	}
+
+	if !exists(usr.HomeDir + "/.out-of-tree/kernels.toml") {
+		log.Println("No ~/.out-of-tree/kernels.toml: Probably you " +
+			"need to run `out-of-tree kernel autogen` in " +
+			"directory that contains .out-of-tree.toml " +
+			"with defined kernel masks " +
+			"(see docs at https://out-of-tree.io)")
 	}
 
 	kingpin.MustParse(app.Parse(os.Args[1:]))
 
+	if *yekaslr && *nokaslr {
+		log.Fatalln("Only one of disable/enable can be used at once")
+	}
+
+	if *yesmep && *nosmep {
+		log.Fatalln("Only one of disable/enable can be used at once")
+	}
+
+	if *yesmap && *nosmap {
+		log.Fatalln("Only one of disable/enable can be used at once")
+	}
+
 	kcfg, err := config.ReadKernelConfig(*kcfgPath)
 	if err != nil {
-		log.Fatalln(err)
+		log.Println(err)
 	}
 
 	if exists(*userKcfgPath) {
@@ -109,28 +282,68 @@ func main() {
 		}
 	}
 
+	handleFallbacks(kcfg)
+
+	db, err := openDatabase(*dbPath)
+	if err != nil {
+		log.Fatalln(err)
+	}
+	defer db.Close()
+
+	stop := time.Time{} // never stop
+	if *timeout != 0 {
+		stop = time.Now().Add(*timeout)
+	}
+
 	switch kingpin.MustParse(app.Parse(os.Args[1:])) {
 	case pewCommand.FullCommand():
 		err = pewHandler(kcfg, *path, *pewKernel, *pewBinary,
-			*pewTest, *pewGuess, *qemuTimeout, *dockerTimeout)
+			*pewTest, *pewGuess, stop, *qemuTimeout, *dockerTimeout,
+			*pewMax, *pewRuns, *pewDist, *pewTag, *pewThreads, db)
 	case kernelListCommand.FullCommand():
 		err = kernelListHandler(kcfg)
 	case kernelAutogenCommand.FullCommand():
-		err = kernelAutogenHandler(*path)
+		err = kernelAutogenHandler(*path, *dockerRegistry,
+			conf.Docker.Commands, *kernelAutogenMax,
+			*kernelUseHost, !*kernelNoDownload)
 	case kernelDockerRegenCommand.FullCommand():
-		err = kernelDockerRegenHandler()
+		err = kernelDockerRegenHandler(*kernelUseHost, !*kernelNoDownload)
+	case kernelGenallCommand.FullCommand():
+		err = kernelGenallHandler(*distro, *version,
+			*dockerRegistry, conf.Docker.Commands,
+			*kernelUseHost, !*kernelNoDownload)
 	case genModuleCommand.FullCommand():
 		err = genConfig(config.KernelModule)
 	case genExploitCommand.FullCommand():
 		err = genConfig(config.KernelExploit)
 	case debugCommand.FullCommand():
 		err = debugHandler(kcfg, *path, *debugKernel, *debugGDB,
-			*dockerTimeout)
+			*dockerTimeout, *yekaslr, *yesmep, *yesmap, *yekpti,
+			*nokaslr, *nosmep, *nosmap, *nokpti)
 	case bootstrapCommand.FullCommand():
-		err = bootstrapHandler()
+		fmt.Println("bootstrap is no more required, " +
+			"now images downloading on-demand")
+		fmt.Println("please, remove it from any automation scripts, " +
+			"because it'll be removed in the next release")
+	case logQueryCommand.FullCommand():
+		err = logHandler(db, *path, *logTag, *logNum, *logRate)
+	case logDumpCommand.FullCommand():
+		err = logDumpHandler(db, *logDumpID)
+	case logJSONCommand.FullCommand():
+		err = logJSONHandler(db, *path, *logJSONTag)
+	case logMarkdownCommand.FullCommand():
+		err = logMarkdownHandler(db, *path, *logMarkdownTag)
+	case packCommand.FullCommand():
+		err = packHandler(db, *path, *dockerRegistry, stop,
+			conf.Docker.Commands, kcfg, *packAutogen,
+			!*packNoDownload, *packExploitRuns, *packKernelRuns)
 	}
 
 	if err != nil {
 		log.Fatalln(err)
 	}
+
+	if successRate(state) < *threshold {
+		os.Exit(1)
+	}
 }

pack.go

@@ -0,0 +1,58 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"runtime"
+	"time"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
+)
+
+func packHandler(db *sql.DB, path, registry string, stop time.Time,
+	commands []config.DockerCommand, kcfg config.KernelConfig,
+	autogen, download bool, exploitRuns, kernelRuns int64) (err error) {
+
+	dockerTimeout := time.Minute
+	qemuTimeout := time.Minute
+	threads := runtime.NumCPU()
+
+	tag := fmt.Sprintf("pack_run_%d", time.Now().Unix())
+	log.Println("Tag:", tag)
+
+	files, err := ioutil.ReadDir(path)
+	if err != nil {
+		return
+	}
+
+	for _, f := range files {
+		workPath := path + "/" + f.Name()
+
+		if !exists(workPath + "/.out-of-tree.toml") {
+			continue
+		}
+
+		if autogen {
+			var perRegex int64 = 1
+			err = kernelAutogenHandler(workPath, registry,
+				commands, perRegex, false, download)
+			if err != nil {
+				return
+			}
+		}
+
+		log.Println(f.Name())
+
+		pewHandler(kcfg, workPath, "", "", "", false,
+			stop, dockerTimeout, qemuTimeout,
+			kernelRuns, exploitRuns, pathDevNull, tag, threads, db)
+	}
+
+	return
+}

pew.go

@@ -5,29 +5,62 @@
 package main
 
 import (
+	"database/sql"
 	"errors"
 	"fmt"
+	"io"
 	"io/ioutil"
 	"log"
 	"math/rand"
 	"os"
 	"os/exec"
-	"runtime"
+	"os/user"
 	"strings"
 	"time"
 
-	"github.com/logrusorgru/aurora"
 	"github.com/otiai10/copy"
 	"github.com/remeh/sizedwaitgroup"
+	"gopkg.in/logrusorgru/aurora.v1"
 
-	"github.com/jollheef/out-of-tree/config"
-	qemu "github.com/jollheef/out-of-tree/qemu"
+	"code.dumpstack.io/tools/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/qemu"
 )
 
-func dockerCommand(container, workdir, timeout, command string) *exec.Cmd {
-	return exec.Command("timeout", "-k", timeout, timeout, "docker", "run",
-		"-v", workdir+":/work", container,
-		"bash", "-c", "cd /work && "+command)
+type runstate struct {
+	Overall, Success float64
+}
+
+var (
+	state runstate
+)
+
+func successRate(state runstate) float64 {
+	return state.Success / state.Overall
+}
+
+const pathDevNull = "/dev/null"
+
+func dockerRun(timeout time.Duration, container, workdir, command string) (
+	output string, err error) {
+
+	cmd := exec.Command("docker", "run", "-v", workdir+":/work",
+		container, "bash", "-c", "cd /work && "+command)
+
+	timer := time.AfterFunc(timeout, func() {
+		cmd.Process.Kill()
+	})
+	defer timer.Stop()
+
+	raw, err := cmd.CombinedOutput()
+	if err != nil {
+		e := fmt.Sprintf("error `%v` for cmd `%v` with output `%v`",
+			err, command, string(raw))
+		err = errors.New(e)
+		return
+	}
+
+	output = string(raw)
+	return
 }
 
 func build(tmp string, ka config.Artifact, ki config.KernelInfo,
@@ -48,38 +81,37 @@ func build(tmp string, ka config.Artifact, ki config.KernelInfo,
 	}
 
 	kernel := "/lib/modules/" + ki.KernelRelease + "/build"
-
-	seconds := fmt.Sprintf("%ds", dockerTimeout/time.Second)
-	cmd := dockerCommand(ki.ContainerName, tmpSourcePath, seconds,
-		"make KERNEL="+kernel+" TARGET="+target)
-	rawOutput, err := cmd.CombinedOutput()
-	output = string(rawOutput)
-	if err != nil {
-		err = errors.New("make execution error")
-		return
+	if ki.KernelSource != "" {
+		kernel = ki.KernelSource
 	}
 
-	return
-}
+	if ki.ContainerName != "" {
+		output, err = dockerRun(dockerTimeout, ki.ContainerName,
+			tmpSourcePath, "make KERNEL="+kernel+" TARGET="+target+
+				" && chmod -R 777 /work")
+	} else {
+		command := "make KERNEL=" + kernel + " TARGET=" + target
+		cmd := exec.Command("bash", "-c", "cd "+tmpSourcePath+" && "+command)
+		timer := time.AfterFunc(dockerTimeout, func() {
+			cmd.Process.Kill()
+		})
+		defer timer.Stop()
 
-func cleanDmesg(q *qemu.QemuSystem) (err error) {
-	start := time.Now()
-	for {
-		_, err = q.Command("root", "dmesg -c")
-		if err == nil {
-			break
+		var raw []byte
+		raw, err = cmd.CombinedOutput()
+		if err != nil {
+			e := fmt.Sprintf("error `%v` for cmd `%v` with output `%v`",
+				err, command, string(raw))
+			err = errors.New(e)
+			return
 		}
-		time.Sleep(time.Second)
 
-		if time.Now().After(start.Add(time.Minute)) {
-			err = errors.New("Can't connect to qemu")
-			break
-		}
+		output = string(raw)
 	}
 	return
 }
 
-func testKernelModule(q *qemu.QemuSystem, ka config.Artifact,
+func testKernelModule(q *qemu.System, ka config.Artifact,
 	test string) (output string, err error) {
 
 	output, err = q.Command("root", test)
@@ -87,7 +119,7 @@ func testKernelModule(q *qemu.QemuSystem, ka config.Artifact,
 	return
 }
 
-func testKernelExploit(q *qemu.QemuSystem, ka config.Artifact,
+func testKernelExploit(q *qemu.System, ka config.Artifact,
 	test, exploit string) (output string, err error) {
 
 	output, err = q.Command("user", "chmod +x "+exploit)
@@ -111,18 +143,49 @@ func testKernelExploit(q *qemu.QemuSystem, ka config.Artifact,
 	return
 }
 
-func genOkFail(name string, ok bool) aurora.Value {
+func genOkFail(name string, ok bool) (aurv aurora.Value) {
+	state.Overall += 1
 	if ok {
+		state.Success += 1
 		s := " " + name + " SUCCESS "
-		return aurora.BgGreen(aurora.Black(s))
+		aurv = aurora.BgGreen(aurora.Black(s))
 	} else {
 		s := " " + name + " FAILURE "
-		return aurora.BgRed(aurora.Gray(aurora.Bold(s)))
+		aurv = aurora.BgRed(aurora.Gray(aurora.Bold(s)))
+	}
+	return
+}
+
+type phasesResult struct {
+	BuildArtifact    string
+	Build, Run, Test struct {
+		Output string
+		Ok     bool
 	}
 }
 
-func dumpResult(q *qemu.QemuSystem, ka config.Artifact, ki config.KernelInfo,
-	build_ok, run_ok, test_ok *bool) {
+func copyFile(sourcePath, destinationPath string) (err error) {
+	sourceFile, err := os.Open(sourcePath)
+	if err != nil {
+		return
+	}
+	defer sourceFile.Close()
+
+	destinationFile, err := os.Create(destinationPath)
+	if err != nil {
+		return err
+	}
+	if _, err := io.Copy(destinationFile, sourceFile); err != nil {
+		destinationFile.Close()
+		return err
+	}
+	return destinationFile.Close()
+}
+
+func dumpResult(q *qemu.System, ka config.Artifact, ki config.KernelInfo,
+	res *phasesResult, dist, tag, binary string, db *sql.DB) {
+
+	// TODO merge (problem is it's not 100% same) with log.go:logLogEntry
 
 	distroInfo := fmt.Sprintf("%s-%s {%s}", ki.DistroType,
 		ki.DistroRelease, ki.KernelRelease)
@@ -130,13 +193,13 @@ func dumpResult(q *qemu.QemuSystem, ka config.Artifact, ki config.KernelInfo,
 	colored := ""
 	if ka.Type == config.KernelExploit {
 		colored = aurora.Sprintf("[*] %40s: %s %s", distroInfo,
-			genOkFail("BUILD", *build_ok),
-			genOkFail("LPE", *test_ok))
+			genOkFail("BUILD", res.Build.Ok),
+			genOkFail("LPE", res.Test.Ok))
 	} else {
 		colored = aurora.Sprintf("[*] %40s: %s %s %s", distroInfo,
-			genOkFail("BUILD", *build_ok),
-			genOkFail("INSMOD", *run_ok),
-			genOkFail("TEST", *test_ok))
+			genOkFail("BUILD", res.Build.Ok),
+			genOkFail("INSMOD", res.Run.Ok),
+			genOkFail("TEST", res.Test.Ok))
 	}
 
 	additional := ""
@@ -151,22 +214,121 @@ func dumpResult(q *qemu.QemuSystem, ka config.Artifact, ki config.KernelInfo,
 	} else {
 		fmt.Println(colored)
 	}
+
+	err := addToLog(db, q, ka, ki, res, tag)
+	if err != nil {
+		log.Println("[db] addToLog (", ka, ") error:", err)
+	}
+
+	if binary == "" && dist != pathDevNull {
+		err = os.MkdirAll(dist, os.ModePerm)
+		if err != nil {
+			log.Println("os.MkdirAll (", ka, ") error:", err)
+		}
+
+		path := fmt.Sprintf("%s/%s-%s-%s", dist, ki.DistroType,
+			ki.DistroRelease, ki.KernelRelease)
+		if ka.Type != config.KernelExploit {
+			path += ".ko"
+		}
+
+		err = copyFile(res.BuildArtifact, path)
+		if err != nil {
+			log.Println("copyFile (", ka, ") error:", err)
+		}
+	}
+}
+
+func copyArtifactAndTest(q *qemu.System, ka config.Artifact,
+	res *phasesResult, remoteTest string) (err error) {
+
+	switch ka.Type {
+	case config.KernelModule:
+		res.Run.Output, err = q.CopyAndInsmod(res.BuildArtifact)
+		if err != nil {
+			log.Println(res.Run.Output, err)
+			return
+		}
+		res.Run.Ok = true
+
+		res.Test.Output, err = testKernelModule(q, ka, remoteTest)
+		if err != nil {
+			log.Println(res.Test.Output, err)
+			return
+		}
+		res.Test.Ok = true
+	case config.KernelExploit:
+		remoteExploit := fmt.Sprintf("/tmp/exploit_%d", rand.Int())
+		err = q.CopyFile("user", res.BuildArtifact, remoteExploit)
+		if err != nil {
+			return
+		}
+
+		res.Test.Output, err = testKernelExploit(q, ka, remoteTest,
+			remoteExploit)
+		if err != nil {
+			log.Println(res.Test.Output)
+			return
+		}
+		res.Run.Ok = true // does not really used
+		res.Test.Ok = true
+	default:
+		log.Println("Unsupported artifact type")
+	}
+
+	return
+}
+
+func copyTest(q *qemu.System, testPath string, ka config.Artifact) (
+	remoteTest string, err error) {
+
+	remoteTest = fmt.Sprintf("/tmp/test_%d", rand.Int())
+	err = q.CopyFile("user", testPath, remoteTest)
+	if err != nil {
+		if ka.Type == config.KernelExploit {
+			q.Command("user",
+				"echo -e '#!/bin/sh\necho touch $2 | $1' "+
+					"> "+remoteTest+
+					" && chmod +x "+remoteTest)
+		} else {
+			q.Command("user", "echo '#!/bin/sh' "+
+				"> "+remoteTest+" && chmod +x "+remoteTest)
+		}
+	}
+
+	_, err = q.Command("root", "chmod +x "+remoteTest)
+	return
 }
 
 func whatever(swg *sizedwaitgroup.SizedWaitGroup, ka config.Artifact,
 	ki config.KernelInfo, binaryPath, testPath string,
-	qemuTimeout, dockerTimeout time.Duration) {
+	qemuTimeout, dockerTimeout time.Duration, dist, tag string,
+	db *sql.DB) {
 
 	defer swg.Done()
 
 	kernel := qemu.Kernel{KernelPath: ki.KernelPath, InitrdPath: ki.InitrdPath}
-	q, err := qemu.NewQemuSystem(qemu.X86_64, kernel, ki.RootFS)
+	q, err := qemu.NewSystem(qemu.X86x64, kernel, ki.RootFS)
 	if err != nil {
 		log.Println("Qemu creation error:", err)
 		return
 	}
 	q.Timeout = qemuTimeout
 
+	if ka.Qemu.Timeout.Duration != 0 {
+		q.Timeout = ka.Qemu.Timeout.Duration
+	}
+	if ka.Qemu.Cpus != 0 {
+		q.Cpus = ka.Qemu.Cpus
+	}
+	if ka.Qemu.Memory != 0 {
+		q.Memory = ka.Qemu.Memory
+	}
+
+	q.SetKASLR(!ka.Mitigations.DisableKaslr)
+	q.SetSMEP(!ka.Mitigations.DisableSmep)
+	q.SetSMAP(!ka.Mitigations.DisableSmap)
+
 	err = q.Start()
 	if err != nil {
 		log.Println("Qemu start error:", err)
@@ -174,106 +336,74 @@ func whatever(swg *sizedwaitgroup.SizedWaitGroup, ka config.Artifact,
 	}
 	defer q.Stop()
 
-	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_")
+	usr, err := user.Current()
+	if err != nil {
+		return
+	}
+	tmpdir := usr.HomeDir + "/.out-of-tree/tmp"
+	os.MkdirAll(tmpdir, os.ModePerm)
+
+	tmp, err := ioutil.TempDir(tmpdir, "out-of-tree_")
 	if err != nil {
 		log.Println("Temporary directory creation error:", err)
 		return
 	}
 	defer os.RemoveAll(tmp)
 
-	build_ok := false
-	run_ok := false
-	test_ok := false
-	defer dumpResult(q, ka, ki, &build_ok, &run_ok, &test_ok)
+	result := phasesResult{}
+	defer dumpResult(q, ka, ki, &result, dist, tag, binaryPath, db)
 
-	var outFile, output string
 	if binaryPath == "" {
-		// TODO Write build log to file or database
-		outFile, output, err = build(tmp, ka, ki, dockerTimeout)
+		result.BuildArtifact, result.Build.Output, err = build(tmp, ka,
+			ki, dockerTimeout)
 		if err != nil {
-			log.Println(output)
+			log.Println(err)
 			return
 		}
-		build_ok = true
+		result.Build.Ok = true
 	} else {
-		outFile = binaryPath
-		build_ok = true
-	}
-
-	err = cleanDmesg(q)
-	if err != nil {
-		return
+		result.BuildArtifact = binaryPath
+		result.Build.Ok = true
 	}
 
 	if testPath == "" {
-		testPath = outFile + "_test"
-	}
-
-	remoteTest := fmt.Sprintf("/tmp/test_%d", rand.Int())
-	err = q.CopyFile("user", testPath, remoteTest)
-	if err != nil {
-		if ka.Type == config.KernelExploit {
-			log.Println("Use `echo touch FILE | exploit` for test")
-			q.Command("user",
-				"echo -e '#!/bin/sh\necho touch $2 | $1' "+
-					"> "+remoteTest+
-					" && chmod +x "+remoteTest)
-		} else {
-			log.Println("copy file err", err)
-			// we should not exit because of testing 'insmod' part
-			// for kernel module
+		testPath = result.BuildArtifact + "_test"
+		if !exists(testPath) {
+			testPath = tmp + "/" + "test.sh"
 		}
 	}
 
-	_, err = q.Command("root", "chmod +x "+remoteTest)
+	remoteTest, err := copyTest(q, testPath, ka)
 	if err != nil {
 		return
 	}
 
-	if ka.Type == config.KernelModule {
-		// TODO Write insmod log to file or database
-		output, err := q.CopyAndInsmod(outFile)
-		if err != nil {
-			log.Println(output, err)
-			return
-		}
-		run_ok = true
+	copyArtifactAndTest(q, ka, &result, remoteTest)
+}
 
-		// TODO Write test results to file or database
-		output, err = testKernelModule(q, ka, remoteTest)
-		if err != nil {
-			log.Println(output, err)
-			return
-		}
-		test_ok = true
-	} else if ka.Type == config.KernelExploit {
-		remoteExploit := fmt.Sprintf("/tmp/exploit_%d", rand.Int())
-		err = q.CopyFile("user", outFile, remoteExploit)
-		if err != nil {
-			return
-		}
-
-		// TODO Write test results to file or database
-		output, err = testKernelExploit(q, ka, remoteTest, remoteExploit)
-		if err != nil {
-			log.Println(output)
-			return
-		}
-		run_ok = true // does not really used
-		test_ok = true
-	} else {
-		err = errors.New("Unsupported artifact type")
+func shuffleKernels(a []config.KernelInfo) []config.KernelInfo {
+	// Fisher–Yates shuffle
+	for i := len(a) - 1; i > 0; i-- {
+		j := rand.Intn(i + 1)
+		a[i], a[j] = a[j], a[i]
 	}
-	return
+	return a
 }
 
 func performCI(ka config.Artifact, kcfg config.KernelConfig, binaryPath,
-	testPath string, qemuTimeout, dockerTimeout time.Duration) (err error) {
+	testPath string, stop time.Time,
+	qemuTimeout, dockerTimeout time.Duration,
+	max, runs int64, dist, tag string, threads int,
+	db *sql.DB) (err error) {
 
 	found := false
 
-	swg := sizedwaitgroup.New(runtime.NumCPU())
-	for _, kernel := range kcfg.Kernels {
+	swg := sizedwaitgroup.New(threads)
+	for _, kernel := range shuffleKernels(kcfg.Kernels) {
+		if max <= 0 {
+			break
+		}
+
 		var supported bool
 		supported, err = ka.Supported(kernel)
 		if err != nil {
@@ -282,9 +412,16 @@ func performCI(ka config.Artifact, kcfg config.KernelConfig, binaryPath,
 
 		if supported {
 			found = true
-			swg.Add()
-			go whatever(&swg, ka, kernel, binaryPath, testPath,
-				qemuTimeout, dockerTimeout)
+			max--
+			for i := int64(0); i < runs; i++ {
+				if !stop.IsZero() && time.Now().After(stop) {
+					break
+				}
+				swg.Add()
+				go whatever(&swg, ka, kernel, binaryPath,
+					testPath, qemuTimeout, dockerTimeout,
+					dist, tag, db)
+			}
 		}
 	}
 	swg.Wait()
@@ -319,9 +456,28 @@ func kernelMask(kernel string) (km config.KernelMask, err error) {
 	return
 }
 
+func genAllKernels() (sk []config.KernelMask, err error) {
+	for _, dType := range config.DistroTypeStrings {
+		var dt config.DistroType
+		dt, err = config.NewDistroType(dType)
+		if err != nil {
+			return
+		}
+
+		sk = append(sk, config.KernelMask{
+			DistroType:  dt,
+			ReleaseMask: ".*",
+		})
+	}
+	return
+}
+
+// TODO: Now too many parameters, move all of them to some structure
 func pewHandler(kcfg config.KernelConfig,
 	workPath, ovrrdKrnl, binary, test string, guess bool,
-	qemuTimeout, dockerTimeout time.Duration) (err error) {
+	stop time.Time, qemuTimeout, dockerTimeout time.Duration,
+	max, runs int64, dist, tag string, threads int,
+	db *sql.DB) (err error) {
 
 	ka, err := config.ReadArtifactConfig(workPath + "/.out-of-tree.toml")
 	if err != nil {
@@ -343,20 +499,15 @@ func pewHandler(kcfg config.KernelConfig,
 	}
 
 	if guess {
-		ka.SupportedKernels = []config.KernelMask{}
-		for _, dType := range config.DistroTypeStrings {
-			var dt config.DistroType
-			dt, err = config.NewDistroType(dType)
-			if err != nil {
-				return
-			}
-
-			km := config.KernelMask{DistroType: dt, ReleaseMask: ".*"}
-			ka.SupportedKernels = append(ka.SupportedKernels, km)
+		ka.SupportedKernels, err = genAllKernels()
+		if err != nil {
+			return
 		}
 	}
 
-	err = performCI(ka, kcfg, binary, test, qemuTimeout, dockerTimeout)
+	err = performCI(ka, kcfg, binary, test,
+		stop, qemuTimeout, dockerTimeout,
+		max, runs, dist, tag, threads, db)
 	if err != nil {
 		return
 	}

pew_test.go

@@ -0,0 +1,44 @@
+// Copyright 2018 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestDockerRun(t *testing.T) {
+	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_test_")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(tmp)
+
+	start := time.Now()
+
+	timeout := time.Second
+
+	_, err = dockerRun(timeout, "ubuntu", tmp, "sleep 5s")
+	if err == nil {
+		t.Fatal("docker is not killed by timeout")
+	}
+	if time.Since(start) > 3*time.Second {
+		t.Fatal(fmt.Sprintf("timeout failed (%v instead of %v)",
+			time.Since(start), time.Second))
+	}
+
+	output, err := dockerRun(time.Minute, "ubuntu", tmp, "echo hello")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if !strings.Contains(output, "hello") {
+		t.Fatal("wrong output (" + output + ")")
+	}
+}

qemu/README.md

@@ -9,7 +9,7 @@ Features:
 
 ## Installation
 
-    $ go get github.com/jollheef/out-of-tree/qemu
+    $ go get code.dumpstack.io/tools/out-of-tree/qemu
 
 ### Generate root image
 
@@ -30,12 +30,12 @@ Note: qemu on macOS since v2.12 (24 April 2018) supports Hypervisor.framework.
 
 #### Generate image
 
-    $ cd $GOPATH/src/github.com/jollheef/out-of-tree/tools/qemu-debian-img
+    $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/tools/qemu-debian-img
     $ ./bootstrap.sh
 
 ### Fill configuration file
 
-    $ $EDITOR $GOPATH/src/github.com/jollheef/out-of-tree/qemu/test.config.go
+    $ $EDITOR $GOPATH/src/code.dumpstack.io/tools/out-of-tree/qemu/test.config.go
 
 ### Run tests
 
@@ -43,7 +43,7 @@ Note: qemu on macOS since v2.12 (24 April 2018) supports Hypervisor.framework.
 
 ## Usage
 
-    $ go get github.com/jollheef/out-of-tree/qemu
+    $ go get code.dumpstack.io/tools/out-of-tree/qemu
 
 Minimal example:
 
@@ -52,7 +52,7 @@ Minimal example:
 		KernelPath: "/path/to/vmlinuz",
 		InitrdPath: "/path/to/initrd", // if required
 	}
-	q, err := qemu.NewQemuSystem(qemu.X86_64, kernel, "/path/to/qcow2")
+	q, err := qemu.NewSystem(qemu.X86_64, kernel, "/path/to/qcow2")
 	if err != nil {
 		log.Fatal(err)
 	}
@@ -71,4 +71,4 @@ Minimal example:
 
 More information and list of all functions see at go documentation project, or just run locally:
 
-    $ godoc github.com/jollheef/out-of-tree/qemu
+    $ godoc code.dumpstack.io/tools/out-of-tree/qemu

qemu/qemu-kernel.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a AGPLv3 license
 // (or later) that can be found in the LICENSE file.
 
-package qemukernel
+package qemu
 
 import (
 	"bytes"
@@ -44,8 +44,10 @@ func readUntilEOF(pipe io.ReadCloser, buf *[]byte) (err error) {
 type arch string
 
 const (
-	X86_64 arch = "x86_64"
-	I386        = "i386"
+	// X86x64 is the qemu-system-x86_64
+	X86x64 arch = "x86_64"
+	// X86x32 is the qemu-system-i386
+	X86x32 = "i386"
 	// TODO add other
 
 	unsupported = "unsupported" // for test purposes
@@ -58,8 +60,8 @@ type Kernel struct {
 	InitrdPath string
 }
 
-// QemuSystem describe qemu parameters and runned process
-type QemuSystem struct {
+// System describe qemu parameters and executed process
+type System struct {
 	arch      arch
 	kernel    Kernel
 	drivePath string
@@ -70,6 +72,11 @@ type QemuSystem struct {
 	debug bool
 	gdb   string // tcp::1234
 
+	noKASLR bool
+	noSMEP  bool
+	noSMAP  bool
+	noKPTI  bool
+
 	// Timeout works after Start invocation
 	Timeout         time.Duration
 	KilledByTimeout bool
@@ -79,7 +86,7 @@ type QemuSystem struct {
 	Died        bool
 	sshAddrPort string
 
-	// accessible while qemu is runned
+	// accessible while qemu is running
 	cmd  *exec.Cmd
 	pipe struct {
 		stdin  io.WriteCloser
@@ -93,12 +100,12 @@ type QemuSystem struct {
 	exitErr error
 }
 
-// NewQemuSystem constructor
-func NewQemuSystem(arch arch, kernel Kernel, drivePath string) (q *QemuSystem, err error) {
+// NewSystem constructor
+func NewSystem(arch arch, kernel Kernel, drivePath string) (q *System, err error) {
 	if _, err = exec.LookPath("qemu-system-" + string(arch)); err != nil {
 		return
 	}
-	q = &QemuSystem{}
+	q = &System{}
 	q.arch = arch
 
 	if _, err = os.Stat(kernel.KernelPath); err != nil {
@@ -156,10 +163,19 @@ func kvmExists() bool {
 	if _, err := os.Stat("/dev/kvm"); err != nil {
 		return false
 	}
+
+	file, err := os.OpenFile("/dev/kvm", os.O_WRONLY, 0666)
+	if err != nil {
+		if os.IsPermission(err) {
+			return false
+		}
+	}
+	file.Close()
+
 	return true
 }
 
-func (q *QemuSystem) panicWatcher() {
+func (q *System) panicWatcher() {
 	for {
 		time.Sleep(time.Second)
 		if bytes.Contains(q.Stdout, []byte("Kernel panic")) {
@@ -172,15 +188,36 @@ func (q *QemuSystem) panicWatcher() {
 	}
 }
 
+func (q System) cmdline() (s string) {
+	s = "root=/dev/sda ignore_loglevel console=ttyS0 rw"
+
+	if q.noKASLR {
+		s += " nokaslr"
+	}
+
+	if q.noSMEP {
+		s += " nosmep"
+	}
+
+	if q.noSMAP {
+		s += " nosmap"
+	}
+
+	if q.noKPTI {
+		s += " nokpti"
+	}
+
+	return
+}
+
 // Start qemu process
-func (q *QemuSystem) Start() (err error) {
+func (q *System) Start() (err error) {
 	rand.Seed(time.Now().UnixNano()) // Are you sure?
 	q.sshAddrPort = getFreeAddrPort()
 	hostfwd := fmt.Sprintf("hostfwd=tcp:%s-:22", q.sshAddrPort)
 	qemuArgs := []string{"-snapshot", "-nographic",
 		"-hda", q.drivePath,
 		"-kernel", q.kernel.KernelPath,
-		"-append", "root=/dev/sda ignore_loglevel console=ttyS0 rw",
 		"-smp", fmt.Sprintf("%d", q.Cpus),
 		"-m", fmt.Sprintf("%d", q.Memory),
 		"-device", "e1000,netdev=n1",
@@ -195,14 +232,16 @@ func (q *QemuSystem) Start() (err error) {
 		qemuArgs = append(qemuArgs, "-initrd", q.kernel.InitrdPath)
 	}
 
-	if (q.arch == X86_64 || q.arch == I386) && kvmExists() {
-		qemuArgs = append(qemuArgs, "-enable-kvm")
+	if (q.arch == X86x64 || q.arch == X86x32) && kvmExists() {
+		qemuArgs = append(qemuArgs, "-enable-kvm", "-cpu", "host")
 	}
 
-	if q.arch == X86_64 && runtime.GOOS == "darwin" {
+	if q.arch == X86x64 && runtime.GOOS == "darwin" {
 		qemuArgs = append(qemuArgs, "-accel", "hvf", "-cpu", "host")
 	}
 
+	qemuArgs = append(qemuArgs, "-append", q.cmdline())
+
 	q.cmd = exec.Command("qemu-system-"+string(q.arch), qemuArgs...)
 
 	if q.pipe.stdin, err = q.cmd.StdinPipe(); err != nil {
@@ -250,7 +289,7 @@ func (q *QemuSystem) Start() (err error) {
 }
 
 // Stop qemu process
-func (q *QemuSystem) Stop() {
+func (q *System) Stop() {
 	// 1  00/01   01  01  SOH  (Ctrl-A)  START OF HEADING
 	fmt.Fprintf(q.pipe.stdin, "%cx", 1)
 	// wait for die
@@ -262,7 +301,7 @@ func (q *QemuSystem) Stop() {
 	}
 }
 
-func (q QemuSystem) ssh(user string) (client *ssh.Client, err error) {
+func (q System) ssh(user string) (client *ssh.Client, err error) {
 	cfg := &ssh.ClientConfig{
 		User:            user,
 		HostKeyCallback: ssh.InsecureIgnoreHostKey(),
@@ -273,7 +312,7 @@ func (q QemuSystem) ssh(user string) (client *ssh.Client, err error) {
 }
 
 // Command executes shell commands on qemu system
-func (q QemuSystem) Command(user, cmd string) (output string, err error) {
+func (q System) Command(user, cmd string) (output string, err error) {
 	client, err := q.ssh(user)
 	if err != nil {
 		return
@@ -291,7 +330,7 @@ func (q QemuSystem) Command(user, cmd string) (output string, err error) {
 }
 
 // AsyncCommand executes command on qemu system but does not wait for exit
-func (q QemuSystem) AsyncCommand(user, cmd string) (err error) {
+func (q System) AsyncCommand(user, cmd string) (err error) {
 	client, err := q.ssh(user)
 	if err != nil {
 		return
@@ -308,7 +347,7 @@ func (q QemuSystem) AsyncCommand(user, cmd string) (err error) {
 }
 
 // CopyFile is copy file from local machine to remote through ssh/scp
-func (q *QemuSystem) CopyFile(user, localPath, remotePath string) (err error) {
+func (q *System) CopyFile(user, localPath, remotePath string) (err error) {
 	addrPort := strings.Split(q.sshAddrPort, ":")
 	addr := addrPort[0]
 	port := addrPort[1]
@@ -325,7 +364,8 @@ func (q *QemuSystem) CopyFile(user, localPath, remotePath string) (err error) {
 	return
 }
 
-func (q *QemuSystem) CopyAndInsmod(localKoPath string) (output string, err error) {
+// CopyAndInsmod copy kernel module to temporary file on qemu then insmod it
+func (q *System) CopyAndInsmod(localKoPath string) (output string, err error) {
 	remoteKoPath := fmt.Sprintf("/tmp/module_%d.ko", rand.Int())
 	err = q.CopyFile("root", localKoPath, remoteKoPath)
 	if err != nil {
@@ -336,7 +376,7 @@ func (q *QemuSystem) CopyAndInsmod(localKoPath string) (output string, err error
 }
 
 // CopyAndRun is copy local file to qemu vm then run it
-func (q *QemuSystem) CopyAndRun(user, path string) (output string, err error) {
+func (q *System) CopyAndRun(user, path string) (output string, err error) {
 	remotePath := fmt.Sprintf("/tmp/executable_%d", rand.Int())
 	err = q.CopyFile(user, path, remotePath)
 	if err != nil {
@@ -346,12 +386,54 @@ func (q *QemuSystem) CopyAndRun(user, path string) (output string, err error) {
 	return q.Command(user, "chmod +x "+remotePath+" && "+remotePath)
 }
 
-func (q *QemuSystem) Debug(conn string) {
+// Debug is for enable qemu debug and set hostname and port for listen
+func (q *System) Debug(conn string) {
 	q.debug = true
 	q.gdb = conn
 }
 
-func (q QemuSystem) GetSshCommand() (cmd string) {
+// SetKASLR is changing KASLR state through kernel boot args
+func (q *System) SetKASLR(state bool) {
+	q.noKASLR = !state
+}
+
+// SetSMEP is changing SMEP state through kernel boot args
+func (q *System) SetSMEP(state bool) {
+	q.noSMEP = !state
+}
+
+// SetSMAP is changing SMAP state through kernel boot args
+func (q *System) SetSMAP(state bool) {
+	q.noSMAP = !state
+}
+
+// SetKPTI is changing KPTI state through kernel boot args
+func (q *System) SetKPTI(state bool) {
+	q.noKPTI = !state
+}
+
+// GetKASLR is retrieve KASLR settings
+func (q *System) GetKASLR() bool {
+	return !q.noKASLR
+}
+
+// GetSMEP is retrieve SMEP settings
+func (q *System) GetSMEP() bool {
+	return !q.noSMEP
+}
+
+// GetSMAP is retrieve SMAP settings
+func (q *System) GetSMAP() bool {
+	return !q.noSMAP
+}
+
+// GetKPTI is retrieve KPTI settings
+func (q *System) GetKPTI() bool {
+	return !q.noKPTI
+}
+
+// GetSSHCommand returns command for connect to qemu machine over ssh
+func (q System) GetSSHCommand() (cmd string) {
 	addrPort := strings.Split(q.sshAddrPort, ":")
 	addr := addrPort[0]
 	port := addrPort[1]

qemu/qemu-kernel_test.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a AGPLv3 license
 // (or later) that can be found in the LICENSE file.
 
-package qemukernel
+package qemu
 
 import (
 	"crypto/sha512"
@@ -20,46 +20,46 @@ func init() {
 	rand.Seed(time.Now().UnixNano())
 }
 
-func TestQemuSystemNew_InvalidKernelPath(t *testing.T) {
+func TestSystemNew_InvalidKernelPath(t *testing.T) {
 	kernel := Kernel{Name: "Invalid", KernelPath: "/invalid/path"}
-	if _, err := NewQemuSystem(X86_64, kernel, "/bin/sh"); err == nil {
+	if _, err := NewSystem(X86x64, kernel, "/bin/sh"); err == nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemNew_InvalidQemuArch(t *testing.T) {
+func TestSystemNew_InvalidQemuArch(t *testing.T) {
 	kernel := Kernel{Name: "Valid path", KernelPath: testConfigVmlinuz}
-	if _, err := NewQemuSystem(unsupported, kernel, "/bin/sh"); err == nil {
+	if _, err := NewSystem(unsupported, kernel, "/bin/sh"); err == nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemNew_InvalidQemuDrivePath(t *testing.T) {
+func TestSystemNew_InvalidQemuDrivePath(t *testing.T) {
 	kernel := Kernel{Name: "Valid path", KernelPath: testConfigVmlinuz}
-	if _, err := NewQemuSystem(X86_64, kernel, "/invalid/path"); err == nil {
+	if _, err := NewSystem(X86x64, kernel, "/invalid/path"); err == nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemNew(t *testing.T) {
+func TestSystemNew(t *testing.T) {
 	kernel := Kernel{Name: "Valid path", KernelPath: testConfigVmlinuz}
-	if _, err := NewQemuSystem(X86_64, kernel, "/bin/sh"); err != nil {
+	if _, err := NewSystem(X86x64, kernel, "/bin/sh"); err != nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemStart(t *testing.T) {
+func TestSystemStart(t *testing.T) {
 	kernel := Kernel{Name: "Test kernel", KernelPath: testConfigVmlinuz}
-	qemu, err := NewQemuSystem(X86_64, kernel, "/bin/sh")
+	q, err := NewSystem(X86x64, kernel, "/bin/sh")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	if err = qemu.Start(); err != nil {
+	if err = q.Start(); err != nil {
 		t.Fatal(err)
 	}
 
-	qemu.Stop()
+	q.Stop()
 }
 
 func TestGetFreeAddrPort(t *testing.T) {
@@ -71,39 +71,39 @@ func TestGetFreeAddrPort(t *testing.T) {
 	ln.Close()
 }
 
-func TestQemuSystemStart_Timeout(t *testing.T) {
+func TestSystemStart_Timeout(t *testing.T) {
 	t.Parallel()
 	kernel := Kernel{Name: "Test kernel", KernelPath: testConfigVmlinuz}
-	qemu, err := NewQemuSystem(X86_64, kernel, "/bin/sh")
+	q, err := NewSystem(X86x64, kernel, "/bin/sh")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	qemu.Timeout = time.Second
+	q.Timeout = time.Second
 
-	if err = qemu.Start(); err != nil {
+	if err = q.Start(); err != nil {
 		t.Fatal(err)
 	}
 
 	time.Sleep(2 * time.Second)
 
-	if !qemu.Died {
+	if !q.Died {
 		t.Fatal("qemu does not died :c")
 	}
 
-	if !qemu.KilledByTimeout {
+	if !q.KilledByTimeout {
 		t.Fatal("qemu died not because of timeout O_o")
 	}
 }
 
-func startTestQemu(t *testing.T, timeout time.Duration) (q *QemuSystem, err error) {
+func startTestQemu(t *testing.T, timeout time.Duration) (q *System, err error) {
 	t.Parallel()
 	kernel := Kernel{
 		Name:       "Test kernel",
 		KernelPath: testConfigVmlinuz,
 		InitrdPath: testConfigInitrd,
 	}
-	q, err = NewQemuSystem(X86_64, kernel, testConfigRootfs)
+	q, err = NewSystem(X86x64, kernel, testConfigRootfs)
 	if err != nil {
 		return
 	}
@@ -119,14 +119,14 @@ func startTestQemu(t *testing.T, timeout time.Duration) (q *QemuSystem, err erro
 	return
 }
 
-func TestQemuSystemCommand(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+func TestSystemCommand(t *testing.T) {
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
-	output, err := qemu.Command("root", "cat /etc/shadow")
+	output, err := q.Command("root", "cat /etc/shadow")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -134,7 +134,7 @@ func TestQemuSystemCommand(t *testing.T) {
 		t.Fatal("Wrong output from `cat /etc/shadow` by root")
 	}
 
-	output, err = qemu.Command("user", "cat /etc/passwd")
+	output, err = q.Command("user", "cat /etc/passwd")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -142,18 +142,19 @@ func TestQemuSystemCommand(t *testing.T) {
 		t.Fatal("Wrong output from `cat /etc/passwd` by user")
 	}
 
-	output, err = qemu.Command("user", "cat /etc/shadow")
-	if err == nil { // unsucessful is good because user must not read /etc/shadow
+	_, err = q.Command("user", "cat /etc/shadow")
+	// unsuccessful is good because user must not read /etc/shadow
+	if err == nil {
 		t.Fatal("User have rights for /etc/shadow. WAT?!")
 	}
 }
 
-func TestQemuSystemCopyFile(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+func TestSystemCopyFile(t *testing.T) {
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	localPath := "/bin/sh"
 
@@ -162,30 +163,31 @@ func TestQemuSystemCopyFile(t *testing.T) {
 		return
 	}
 
-	sha_local := fmt.Sprintf("%x", sha512.Sum512(content))
+	shaLocal := fmt.Sprintf("%x", sha512.Sum512(content))
 
-	err = qemu.CopyFile("user", localPath, "/tmp/test")
+	err = q.CopyFile("user", localPath, "/tmp/test")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	sha_remote, err := qemu.Command("user", "sha512sum /tmp/test")
+	shaRemote, err := q.Command("user", "sha512sum /tmp/test")
 	if err != nil {
 		t.Fatal(err)
 	}
-	sha_remote = strings.Split(sha_remote, " ")[0]
+	shaRemote = strings.Split(shaRemote, " ")[0]
 
-	if sha_local != sha_remote {
-		t.Fatal(fmt.Sprintf("Broken file (%s instead of %s)", sha_remote, sha_local))
+	if shaLocal != shaRemote {
+		t.Fatal(fmt.Sprintf("Broken file (%s instead of %s)",
+			shaRemote, shaLocal))
 	}
 }
 
-func TestQemuSystemCopyAndRun(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+func TestSystemCopyAndRun(t *testing.T) {
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	randStr := fmt.Sprintf("%d", rand.Int())
 	content := []byte("#!/bin/sh\n echo -n " + randStr + "\n")
@@ -203,34 +205,35 @@ func TestQemuSystemCopyAndRun(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	output, err := qemu.CopyAndRun("user", tmpfile.Name())
+	output, err := q.CopyAndRun("user", tmpfile.Name())
 	if err != nil {
 		t.Fatal(err)
 	}
 
 	if output != randStr {
-		t.Fatal("Wrong output from copyied executable (" + output + "," + randStr + ")")
+		t.Fatal("Wrong output from copyied executable (" +
+			output + "," + randStr + ")")
 	}
 }
 
-func TestQemuSystemCopyAndInsmod(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+func TestSystemCopyAndInsmod(t *testing.T) {
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
-	lsmodBefore, err := qemu.Command("root", "lsmod | wc -l")
+	lsmodBefore, err := q.Command("root", "lsmod | wc -l")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	_, err = qemu.CopyAndInsmod(testConfigSampleKo)
+	_, err = q.CopyAndInsmod(testConfigSampleKo)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	lsmodAfter, err := qemu.Command("root", "lsmod | wc -l")
+	lsmodAfter, err := q.Command("root", "lsmod | wc -l")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -240,21 +243,21 @@ func TestQemuSystemCopyAndInsmod(t *testing.T) {
 	}
 }
 
-func TestQemuSystemKernelPanic(t *testing.T) {
-	qemu, err := startTestQemu(t, time.Minute)
+func TestSystemKernelPanic(t *testing.T) {
+	q, err := startTestQemu(t, 5*time.Minute)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	// Enable sysrq
-	_, err = qemu.Command("root", "echo 1 > /proc/sys/kernel/sysrq")
+	_, err = q.Command("root", "echo 1 > /proc/sys/kernel/sysrq")
 	if err != nil {
 		t.Fatal(err)
 	}
 
 	// Trigger kernel panic
-	err = qemu.AsyncCommand("root", "sleep 1s && echo c > /proc/sysrq-trigger")
+	err = q.AsyncCommand("root", "sleep 1s && echo c > /proc/sysrq-trigger")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -262,40 +265,41 @@ func TestQemuSystemKernelPanic(t *testing.T) {
 	// Wait for panic watcher timeout
 	time.Sleep(5 * time.Second)
 
-	if qemu.KilledByTimeout {
+	if q.KilledByTimeout {
 		t.Fatal("qemu is killed by timeout, not because of panic")
 	}
 
-	if !qemu.Died {
+	if !q.Died {
 		t.Fatal("qemu is not killed after kernel panic")
 	}
 
-	if !qemu.KernelPanic {
+	if !q.KernelPanic {
 		t.Fatal("qemu is died but there's no information about panic")
 	}
 }
 
-func TestQemuSystemRun(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+func TestSystemRun(t *testing.T) {
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	for {
-		_, err := qemu.Command("root", "echo")
+		_, err := q.Command("root", "echo")
 		if err == nil {
 			break
 		}
 	}
 
 	start := time.Now()
-	err = qemu.AsyncCommand("root", "sleep 10s")
+	err = q.AsyncCommand("root", "sleep 1m")
 	if err != nil {
 		t.Fatal(err)
 	}
-	if time.Since(start) > time.Second {
-		t.Fatalf("qemu.Run does not async (waited %s)", +time.Since(start))
+	if time.Since(start) > 10*time.Second {
+		t.Fatalf("q.AsyncCommand does not async (waited %s)",
+			time.Since(start))
 	}
 
 }
@@ -309,13 +313,13 @@ func openedPort(port int) bool {
 	return true
 }
 
-func TestQemuSystemDebug(t *testing.T) {
+func TestSystemDebug(t *testing.T) {
 	t.Parallel()
 	kernel := Kernel{
 		KernelPath: testConfigVmlinuz,
 		InitrdPath: testConfigInitrd,
 	}
-	q, err := NewQemuSystem(X86_64, kernel, testConfigRootfs)
+	q, err := NewSystem(X86x64, kernel, testConfigRootfs)
 	if err != nil {
 		return
 	}

qemu/test.config.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a AGPLv3 license
 // (or later) that can be found in the LICENSE file.
 
-package qemukernel
+package qemu
 
 const testConfigVmlinuz = "../tools/qemu-debian-img/ubuntu1804.vmlinuz"
 const testConfigInitrd = "../tools/qemu-debian-img/ubuntu1804.initrd"

tools/kernel-factory/.gitignore

@@ -1 +0,0 @@
-output

tools/kernel-factory/Ubuntu/14.04/Dockerfile

@@ -1,9 +0,0 @@
-FROM ubuntu:14.04
-
-RUN apt-get update
-
-# for linux-image-3*-generic, so... CRUTCHES!
-# E: Unable to locate package linux-image-3*-generic
-ENV DEBIAN_FRONTEND=noninteractive
-RUN apt search linux-image | grep -e linux-image-3 -e linux-image-4 | grep generic | cut -d '/' -f 1 | xargs apt install -y
-RUN apt search linux-headers | grep -e linux-headers-3 -e linux-headers-4 | grep generic | cut -d '/' -f 1 | xargs apt install -y

tools/kernel-factory/Ubuntu/14.04/Image

@@ -1 +0,0 @@
-../../../qemu-debian-img/ubuntu1404.img

tools/kernel-factory/Ubuntu/16.04/Dockerfile

@@ -1,4 +0,0 @@
-FROM ubuntu:16.04
-
-RUN apt-get update
-RUN DEBIAN_FRONTEND=noninteractive apt-get install -y linux-image-4*-generic linux-headers-*-generic build-essential wget git

tools/kernel-factory/Ubuntu/16.04/Image

@@ -1 +0,0 @@
-../../../qemu-debian-img/ubuntu1604.img

tools/kernel-factory/Ubuntu/18.04/Dockerfile

@@ -1,4 +0,0 @@
-FROM ubuntu:18.04
-
-RUN apt-get update
-RUN DEBIAN_FRONTEND=noninteractive apt-get install -y linux-image-4*-generic linux-headers-*-generic build-essential wget git libelf-dev

tools/kernel-factory/Ubuntu/18.04/Image

@@ -1 +0,0 @@
-../../../qemu-debian-img/ubuntu1804.img

tools/kernel-factory/bootstrap.sh

@@ -1,29 +0,0 @@
-#!/bin/sh -eux
-mkdir -p output
-echo > output/kernels.toml
-find | grep Docker | sed 's/Dockerfile//' | while read DOCKER; do
-    CONTAINER_NAME=$(echo $DOCKER | sed -e 's;/;;g' -e 's;\.;;g' -e 's;\(.*\);\L\1;')
-    docker build -t ${CONTAINER_NAME} ${DOCKER}
-    docker run ${CONTAINER_NAME} bash -c 'ls /boot'
-    CONTAINER_ID=$(docker ps -a | grep ${CONTAINER_NAME} | awk '{print $1}' | head -n 1)
-    docker cp ${CONTAINER_ID}:/boot/. output/
-    DISTRO_NAME=$(echo $DOCKER | cut -d '/' -f 2)
-    DISTRO_VER=$(echo $DOCKER | cut -d '/' -f 3)
-
-    BOOT_FILES="$(docker run $CONTAINER_NAME ls /boot)"
-    for KERNEL_RELEASE in $(docker run $CONTAINER_NAME ls /lib/modules); do
-	echo '[[Kernels]]' >> output/kernels.toml
-	echo 'distro_type =' \"$DISTRO_NAME\" >> output/kernels.toml
-	echo 'distro_release =' \"$DISTRO_VER\" >> output/kernels.toml
-	echo 'kernel_release =' \"$KERNEL_RELEASE\" >> output/kernels.toml
-	echo 'container_name =' \"$CONTAINER_NAME\" >> output/kernels.toml
-	KERNEL_PATH=$(echo $BOOT_FILES | sed  's/ /\n/g' | grep $KERNEL_RELEASE | grep vmlinuz)
-	echo 'kernel_path =' \"$(realpath output/$KERNEL_PATH)\" >> output/kernels.toml
-	INITRD_PATH=$(echo $BOOT_FILES | sed  's/ /\n/g' | grep $KERNEL_RELEASE | grep init)
-	echo 'initrd_path =' \"$(realpath output/$INITRD_PATH)\" >> output/kernels.toml
-	ROOTFS_PATH=$(realpath $DOCKER/Image)
-	echo 'root_f_s =' \"$ROOTFS_PATH\" >> output/kernels.toml
-	echo >> output/kernels.toml
-    done
-done
-rm -rf output/grub

tools/qemu-centos-img/7/Dockerfile

@@ -1,4 +1,4 @@
-# Copyright 2018 Mikhail Klementev. All rights reserved.
+# Copyright 2019 Mikhail Klementev. All rights reserved.
 # Use of this source code is governed by a AGPLv3 license
 # (or later) that can be found in the LICENSE file.
 #
@@ -7,11 +7,13 @@
 #     $ docker build -t gen-centos7-image .
 #     $ docker run --privileged -v $(pwd):/shared -t gen-centos7-image
 #
-# centos7.img will be created in current directory. You can change $(pwd) to
-# different directory to use different destination for image.
+# out_of_tree_centos_7.img will be created in current directory.
+# You can change $(pwd) to different directory to use different destination
+# for image.
 #
 FROM centos:7
 
+RUN yum -y update
 RUN yum -y groupinstall "Development Tools"
 RUN yum -y install qemu-img e2fsprogs
 
@@ -26,7 +28,7 @@ RUN yum --installroot=$TMPDIR \
         --releasever=7 \
         --disablerepo='*' \
         --enablerepo=base \
-        -y install openssh-server
+        -y install openssh-server openssh-clients
 
 RUN chroot $TMPDIR /bin/sh -c 'useradd -m user'
 RUN sed -i 's/root:\*:/root::/' $TMPDIR/etc/shadow
@@ -37,17 +39,15 @@ RUN sed -i '/PermitRootLogin/d' $TMPDIR/etc/ssh/sshd_config
 RUN echo PermitRootLogin yes >> $TMPDIR/etc/ssh/sshd_config
 
 # network workaround
-# FIXME kernel module compatibility issues
 RUN chmod +x $TMPDIR/etc/rc.local
-RUN echo 'find /lib/modules | grep e1000.ko | xargs insmod -f' >> $TMPDIR/etc/rc.local
 RUN echo 'dhclient' >> $TMPDIR/etc/rc.local
 
 ENV IMAGEDIR=/tmp/image
-ENV IMAGE=/shared/centos7.img
+ENV IMAGE=/shared/out_of_tree_centos_7.img
 
 RUN mkdir $IMAGEDIR
 
-# Must be runned with --privileged because of /dev/loop
+# Must be executed with --privileged because of /dev/loop
 CMD qemu-img create $IMAGE 2G && \
 	mkfs.ext4 -F $IMAGE && \
 	mount -o loop $IMAGE $IMAGEDIR && \

tools/qemu-debian-img/14.04/Dockerfile

@@ -0,0 +1,34 @@
+# Copyright 2018 Mikhail Klementev. All rights reserved.
+# Use of this source code is governed by a AGPLv3 license
+# (or later) that can be found in the LICENSE file.
+#
+# Usage:
+#
+#     $ docker build -t gen-ubuntu1404-image .
+#     $ docker run --privileged -v $(pwd):/shared -t gen-ubuntu1404-image
+#
+# ubuntu1404.img will be created in current directory. You can change $(pwd) to
+# different directory to use different destination for image.
+#
+FROM ubuntu:14.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update
+RUN apt-get install -y debootstrap qemu
+
+ENV TMPDIR=/tmp/ubuntu
+ENV IMAGEDIR=/tmp/image
+ENV IMAGE=/shared/out_of_tree_ubuntu_14__04.img
+ENV REPOSITORY=http://archive.ubuntu.com/ubuntu
+ENV RELEASE=trusty
+
+RUN mkdir $IMAGEDIR
+
+# Must be executed with --privileged because of /dev/loop
+CMD debootstrap --include=openssh-server $RELEASE $TMPDIR $REPOSITORY && \
+	/shared/setup.sh $TMPDIR && \
+	qemu-img create $IMAGE 2G && \
+	mkfs.ext4 -F $IMAGE && \
+	mount -o loop $IMAGE $IMAGEDIR && \
+	cp -a $TMPDIR/* $IMAGEDIR/ && \
+	umount $IMAGEDIR

tools/qemu-debian-img/14.04/setup.sh

@@ -0,0 +1,17 @@
+#!/bin/sh -eux
+# Copyright 2018 Mikhail Klementev. All rights reserved.
+# Use of this source code is governed by a AGPLv3 license
+# (or later) that can be found in the LICENSE file.
+TMPDIR=$1
+chroot $TMPDIR /bin/sh -c 'useradd -m user'
+sed -i 's/root:\*:/root::/' $TMPDIR/etc/shadow
+sed -i 's/user:!!:/user::/' $TMPDIR/etc/shadow
+echo auth sufficient pam_permit.so > $TMPDIR/etc/pam.d/sshd
+sed -i '/PermitEmptyPasswords/d' $TMPDIR/etc/ssh/sshd_config
+echo PermitEmptyPasswords yes >> $TMPDIR/etc/ssh/sshd_config
+sed -i '/PermitRootLogin/d' $TMPDIR/etc/ssh/sshd_config
+echo PermitRootLogin yes >> $TMPDIR/etc/ssh/sshd_config
+
+echo '#!/bin/sh' > $TMPDIR/etc/rc.local
+echo 'dhclient eth0' >> $TMPDIR/etc/rc.local
+chmod +x $TMPDIR/etc/rc.local

tools/qemu-debian-img/Dockerfile

@@ -7,7 +7,7 @@
 #     $ docker build -t gen-ubuntu1804-image .
 #     $ docker run --privileged -v $(pwd):/shared -t gen-ubuntu1804-image
 #
-# centos7.img will be created in current directory. You can change $(pwd) to
+# ubuntu1804.img will be created in current directory. You can change $(pwd) to
 # different directory to use different destination for image.
 #
 FROM ubuntu:18.04
@@ -25,7 +25,7 @@ ENV RELEASE=bionic
 
 RUN mkdir $IMAGEDIR
 
-# Must be runned with --privileged because of /dev/loop
+# Must be executed with --privileged because of /dev/loop
 CMD debootstrap --include=openssh-server $RELEASE $TMPDIR $REPOSITORY && \
 	/shared/setup.sh $TMPDIR && \
 	qemu-img create $IMAGE 2G && \

tools/qemu-debian-img/bootstrap.sh

@@ -1,4 +1,6 @@
 #!/bin/sh -eux
+cd $(dirname $(realpath $0))
+
 docker build -t gen-ubuntu1804-image .
 docker run --privileged -v $(pwd):/shared -t gen-ubuntu1804-image
 RUN="docker run -v $(pwd):/shared -t gen-ubuntu1804-image"