Troubleshooting

Resolves #8

.travis.yml

@@ -0,0 +1,30 @@
+language: go
+
+go:
+  - 1.x
+  - master
+
+os:
+  - linux
+
+dist:
+  - bionic
+
+addons:
+  apt:
+    packages:
+    - qemu
+
+services:
+  - docker
+
+env:
+  - GO111MODULE=on
+
+install: true
+
+before_script:
+  - ./tools/qemu-debian-img/bootstrap.sh
+
+script:
+  - go test -parallel 1 -v ./...

CHANGELOG.md

@@ -0,0 +1,127 @@
+# Changelog
+
+[ISO 8601](https://xkcd.com/1179/).
+
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+- New parameter `--max=X` is added for `autogen` (generate kernels
+  base on `.out-of-tree.toml` definitions) and `pew` (automated
+  runs) and allows to specify a maximum number of runs per each
+  supported kernel in module/exploit definition.
+
+- New command `genall` -- generate all kernels for specified
+  distro/version.
+
+- All logs stores in sqlite3 database. Implemented specific commands
+  for making simple queries and export data to markdown and json.
+
+- Implemented success rate calculation for previous runs.
+
+- Save of build results supported by parameter `--dist` for `pew`.
+
+- Support for generating kernels info from host system.
+
+- Support for build on host.
+
+- Support for custom kernels.
+
+- Now debugging environment is automatically looking for debug
+  kernel on the host system.
+
+- Added ability to enable/disable kaslr/smep/smap/kpti for debugging
+  by command line flags.
+
+- New parameter `--threads=N` is added for `pew` and allows to
+  specify maximum number of threads that will be used for parallel
+  build/run/test.
+
+- Tagging for runs. Tags write to log and can be used for
+  statistics.
+
+- Added non-regex way to set kernel version in .out-of-tree.toml (see
+  examples).
+
+- New command `pack` that perform tests in subdirectories.
+
+- Added ability to disable kaslr/smep/smap/kpti for in artifact
+  definition.
+
+- Added ability to change amount of memory/CPUs and set qemu timeout
+  in artifact definition (`.out-of-tree.toml`).
+
+- Now images downloading while `kernel autogen`, bootstrap is not
+  required anymore.
+
+- Support CentOS kernels.
+
+### Changed
+
+- Now if there's no base image found — out-of-tree will try to use
+  an image from closest previous version, e.g. image from Ubuntu
+  18.04 for Ubuntu 18.10.
+
+- Kernel modules tests will not be failed if there are no tests
+  exists.
+
+- Now *out-of-tree* will return negative error code if at least one
+  of the stage was failed.
+
+- Project is switch to use Go modules.
+
+- Now test.sh is used by default if copying is not implemented in
+  Makefile.
+
+- dmesg is not cleaned before the start of module/exploit anymore.
+
+- qemu/kvm will use all host cpu features.
+
+### Removed
+
+- *Kernel factory* is removed completely in favor of incremental
+  Dockerfiles.
+
+- `bootstrap` is not doing anything anymore. It'll be removed in next
+  release.
+
+### Fixed
+
+- Command `timeout` is not required anymore.
+
+- Errors is more meaningful.
+
+- Temporary files is moved to `~/.out-of-tree/tmp/` to avoid docker
+  mounting issues on some systems.
+
+## [0.2.0] - 2019-12-01
+
+The main purpose of the release is to simplify installation.
+
+### Changes
+
+- All configuration moved to `~/.out-of-tree`.
+
+- Now prebuilt images can be downloaded with bootstrap.
+
+- Ability to generate kernels specific to .out-of-tree.toml in
+  current directory. So now there's no need to wait for several
+  hours for start work on specific kernel with module/exploit.
+
+- Now there's no need to keep source tree and _out-of-tree_ can be
+  distributed in binary form.
+
+- New command: **debug**. Creates interactive environment for kernel
+  module/exploit development. Still work-in-progress.
+
+- No warning anymore if test.sh is not exists.
+
+## [0.1.0] - 2019-11-20
+
+Initial release that was never tagged.
+
+Refer to state after first public release on ZeroNights 2018
+([video](https://youtu.be/2tL7bbCdIio),
+[slides](https://2018.zeronights.ru/wp-content/uploads/materials/07-Ways-to-automate-testing-Linux-kernel-exploits.pdf)).

README.md

@@ -1,17 +1,48 @@
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/aba4aad2046b4d1a9a99cf98e22c018b)](https://app.codacy.com/app/jollheef/out-of-tree?utm_source=github.com&utm_medium=referral&utm_content=jollheef/out-of-tree&utm_campaign=Badge_Grade_Dashboard)
+[![Build Status](https://travis-ci.org/jollheef/out-of-tree.svg?branch=master)](https://travis-ci.org/jollheef/out-of-tree)
+[![Go Report Card](https://goreportcard.com/badge/code.dumpstack.io/tools/out-of-tree)](https://goreportcard.com/report/code.dumpstack.io/tools/out-of-tree)
+[![Documentation Status](https://readthedocs.org/projects/out-of-tree/badge/?version=latest)](https://out-of-tree.readthedocs.io/en/latest/?badge=latest)
+[![Donate](https://img.shields.io/badge/donate-paypal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=R8W2UQPZ5X5JE&source=url)
+[![Donate](https://img.shields.io/badge/donate-bitcoin-green.svg)](https://blockchair.com/bitcoin/address/bc1q23fyuq7kmngrgqgp6yq9hk8a5q460f39m8nv87)
+
 # [out-of-tree](https://out-of-tree.io)
 
 out-of-tree kernel {module, exploit} development tool
 
 ![Screenshot](https://cloudflare-ipfs.com/ipfs/Qmb88fgdDjbWkxz91sWsgmoZZNfVThnCtj37u3mF2s3T3T)
 
-## Installation
+## Requirements
 
-    $ go get github.com/jollheef/out-of-tree
+[Qemu](https://www.qemu.org), [docker](https://docker.com) and [golang](https://golang.org) is required.
-    $ out-of-tree bootstrap
+
+Also do not forget to set GOPATH and PATH e.g.:
+
+    $ echo 'export GOPATH=$HOME' >> ~/.bashrc
+    $ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
+    $ source ~/.bashrc
+
+### Gentoo
+
+    # emerge app-emulation/qemu app-emulation/docker dev-lang/go
+
+### macOS
+
+    $ brew install go qemu
+    $ brew cask install docker
+
+### Fedora
+
+    $ sudo dnf install go qemu moby-engine
+
+Also check out [docker post-installation steps](https://docs.docker.com/install/linux/linux-postinstall/).
+
+## Build from source
+
+    $ go get -u code.dumpstack.io/tools/out-of-tree
 
 Then you can check it on kernel module example:
 
-    $ cd $GOPATH/github.com/jollheef/out-of-tree/examples/kernel-module
+    $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/examples/kernel-module
     $ out-of-tree kernel autogen # generate kernels based on .out-of-tree.toml
     $ out-of-tree pew
 
@@ -45,13 +76,16 @@ Use custom kernels config
 
     $ out-of-tree --kernels /path/to/kernels.toml pew
 
-## Generate all kernels
+Generate all kernels
 
-Does not required if you dont need to use `--guess`.
+    $ out-of-tree kernel genall --distro Ubuntu --ver 16.04
 
-    $ cd $GOPATH/src/github.com/jollheef/out-of-tree/tools/kernel-factory
+
-    $ ./bootstrap.sh # more than 6-8 hours for all kernels
+## Troubleshooting
-    $ export OUT_OF_TREE_KCFG=$GOPATH/src/github.com/jollheef/out-of-tree/tools/kernel-factory/output/kernels.toml
+
+If anything happens that you cannot solve -- just remove `$HOME/.out-of-tree`.
+
+But it'll be better if you'll write the bug report.
 
 ## Development
 
@@ -59,6 +93,6 @@ Read [Qemu API](qemu/README.md).
 
 ### Generate images
 
-    $ cd $GOPATH/src/github.com/jollheef/out-of-tree/tools/qemu-debian-img/
+    $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/tools/qemu-debian-img/
     $ docker run --privileged -v $(pwd):/shared -e IMAGE=/shared/ubuntu1404.img -e RELEASE=trusty -t gen-ubuntu1804-image
     $ docker run --privileged -v $(pwd):/shared -e IMAGE=/shared/ubuntu1604.img -e RELEASE=xenial -t gen-ubuntu1804-image

bootstrap.config.go

@@ -1,7 +0,0 @@
-// Copyright 2018 Mikhail Klementev. All rights reserved.
-// Use of this source code is governed by a AGPLv3 license
-// (or later) that can be found in the LICENSE file.
-
-package main
-
-const imagesURL = "https://github.com/jollheef/out-of-tree/releases/download/v0.2/images.tar.gz"

bootstrap.go

@@ -1,76 +0,0 @@
-// Copyright 2018 Mikhail Klementev. All rights reserved.
-// Use of this source code is governed by a AGPLv3 license
-// (or later) that can be found in the LICENSE file.
-
-package main
-
-import (
-	"io"
-	"io/ioutil"
-	"log"
-	"net/http"
-	"os"
-	"os/exec"
-	"os/user"
-)
-
-// inspired by Edd Turtle code
-func downloadFile(filepath string, url string) (err error) {
-	out, err := os.Create(filepath)
-	if err != nil {
-		return
-	}
-	defer out.Close()
-
-	resp, err := http.Get(url)
-	if err != nil {
-		return
-	}
-	defer resp.Body.Close()
-
-	_, err = io.Copy(out, resp.Body)
-	return
-}
-
-func unpackTar(archive, destination string) (err error) {
-	cmd := exec.Command("tar", "xf", archive)
-	cmd.Dir = destination + "/"
-
-	rawOutput, err := cmd.CombinedOutput()
-	if err != nil {
-		// I don't like when some errors printed inside
-		// So if you know way to do it better - FIXME please
-		log.Println("Unpack images error:", string(rawOutput), err)
-		return
-	}
-
-	return
-}
-
-func bootstrapHandler() (err error) {
-	usr, err := user.Current()
-	if err != nil {
-		return
-	}
-
-	imagesPath := usr.HomeDir + "/.out-of-tree/images/"
-	os.MkdirAll(imagesPath, os.ModePerm)
-
-	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_")
-	if err != nil {
-		log.Println("Temporary directory creation error:", err)
-		return
-	}
-	defer os.RemoveAll(tmp)
-
-	imagesArchive := tmp + "/images.tar.gz"
-
-	err = downloadFile(imagesArchive, imagesURL)
-	if err != nil {
-		log.Println("Download file error:", err)
-		return
-	}
-
-	err = unpackTar(imagesArchive, imagesPath)
-	return
-}

config/config.go

@@ -10,27 +10,44 @@ import (
 	"io/ioutil"
 	"os"
 	"regexp"
+	"strconv"
 	"strings"
+	"time"
 
 	"github.com/naoina/toml"
 )
 
+type kernel struct {
+	Version []int
+	Major   []int
+	Minor   []int
+	Patch   []int
+}
+
+// KernelMask defines the kernel
 type KernelMask struct {
 	DistroType    DistroType
 	DistroRelease string // 18.04/7.4.1708/9.1
 	ReleaseMask   string
+
+	// Overrides ReleaseMask
+	Kernel kernel
 }
 
+// DockerName is returns stable name for docker container
 func (km KernelMask) DockerName() string {
 	distro := strings.ToLower(km.DistroType.String())
 	release := strings.Replace(km.DistroRelease, ".", "__", -1)
 	return fmt.Sprintf("out_of_tree_%s_%s", distro, release)
 }
 
+// ArtifactType is the kernel module or exploit
 type ArtifactType int
 
 const (
+	// KernelModule is any kind of kernel module
 	KernelModule ArtifactType = iota
+	// KernelExploit is the privilege escalation exploit
 	KernelExploit
 )
 
@@ -38,6 +55,7 @@ func (at ArtifactType) String() string {
 	return [...]string{"module", "exploit"}[at]
 }
 
+// UnmarshalTOML is for support github.com/naoina/toml
 func (at *ArtifactType) UnmarshalTOML(data []byte) (err error) {
 	stype := strings.Trim(string(data), `"`)
 	stypelower := strings.ToLower(stype)
@@ -46,11 +64,12 @@ func (at *ArtifactType) UnmarshalTOML(data []byte) (err error) {
 	} else if strings.Contains(stypelower, "exploit") {
 		*at = KernelExploit
 	} else {
-		err = errors.New(fmt.Sprintf("Type %s is unsupported", stype))
+		err = fmt.Errorf("Type %s is unsupported", stype)
 	}
 	return
 }
 
+// MarshalTOML is for support github.com/naoina/toml
 func (at ArtifactType) MarshalTOML() (data []byte, err error) {
 	s := ""
 	switch at {
@@ -59,17 +78,49 @@ func (at ArtifactType) MarshalTOML() (data []byte, err error) {
 	case KernelExploit:
 		s = "exploit"
 	default:
-		err = errors.New(fmt.Sprintf("Cannot marshal %d", at))
+		err = fmt.Errorf("Cannot marshal %d", at)
 	}
 	data = []byte(`"` + s + `"`)
 	return
 }
 
+// Duration type with toml unmarshalling support
+type Duration struct {
+	time.Duration
+}
+
+// UnmarshalTOML for Duration
+func (d *Duration) UnmarshalTOML(data []byte) (err error) {
+	duration := strings.Replace(string(data), "\"", "", -1)
+	d.Duration, err = time.ParseDuration(duration)
+	return
+}
+
+// MarshalTOML for Duration
+func (d Duration) MarshalTOML() (data []byte, err error) {
+	data = []byte(`"` + d.Duration.String() + `"`)
+	return
+}
+
+// Artifact is for .out-of-tree.toml
 type Artifact struct {
 	Name             string
 	Type             ArtifactType
 	SourcePath       string
 	SupportedKernels []KernelMask
+
+	Qemu struct {
+		Cpus    int
+		Memory  int
+		Timeout Duration
+	}
+
+	Mitigations struct {
+		DisableSmep  bool
+		DisableSmap  bool
+		DisableKaslr bool
+		DisableKpti  bool
+	}
 }
 
 func (ka Artifact) checkSupport(ki KernelInfo, km KernelMask) (
@@ -90,6 +141,7 @@ func (ka Artifact) checkSupport(ki KernelInfo, km KernelMask) (
 	return
 }
 
+// Supported returns true if given kernel is supported by artifact
 func (ka Artifact) Supported(ki KernelInfo) (supported bool, err error) {
 	for _, km := range ka.SupportedKernels {
 		supported, err = ka.checkSupport(ki, km)
@@ -101,16 +153,22 @@ func (ka Artifact) Supported(ki KernelInfo) (supported bool, err error) {
 	return
 }
 
+// DistroType is enum with all supported distros
 type DistroType int
 
 const (
+	// Ubuntu https://ubuntu.com/
 	Ubuntu DistroType = iota
+	// CentOS https://www.centos.org/
 	CentOS
+	// Debian https://www.debian.org/
 	Debian
 )
 
+// DistroTypeStrings is the string version of enum DistroType
 var DistroTypeStrings = [...]string{"Ubuntu", "CentOS", "Debian"}
 
+// NewDistroType is create new Distro object
 func NewDistroType(dType string) (dt DistroType, err error) {
 	err = dt.UnmarshalTOML([]byte(dType))
 	return
@@ -120,6 +178,7 @@ func (dt DistroType) String() string {
 	return DistroTypeStrings[dt]
 }
 
+// UnmarshalTOML is for support github.com/naoina/toml
 func (dt *DistroType) UnmarshalTOML(data []byte) (err error) {
 	sDistro := strings.Trim(string(data), `"`)
 	if strings.EqualFold(sDistro, "Ubuntu") {
@@ -129,11 +188,12 @@ func (dt *DistroType) UnmarshalTOML(data []byte) (err error) {
 	} else if strings.EqualFold(sDistro, "Debian") {
 		*dt = Debian
 	} else {
-		err = errors.New(fmt.Sprintf("Distro %s is unsupported", sDistro))
+		err = fmt.Errorf("Distro %s is unsupported", sDistro)
 	}
 	return
 }
 
+// MarshalTOML is for support github.com/naoina/toml
 func (dt DistroType) MarshalTOML() (data []byte, err error) {
 	s := ""
 	switch dt {
@@ -144,12 +204,20 @@ func (dt DistroType) MarshalTOML() (data []byte, err error) {
 	case Debian:
 		s = "Debian"
 	default:
-		err = errors.New(fmt.Sprintf("Cannot marshal %d", dt))
+		err = fmt.Errorf("Cannot marshal %d", dt)
 	}
 	data = []byte(`"` + s + `"`)
 	return
 }
 
+// ByRootFS is sorting by .RootFS lexicographically
+type ByRootFS []KernelInfo
+
+func (a ByRootFS) Len() int           { return len(a) }
+func (a ByRootFS) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
+func (a ByRootFS) Less(i, j int) bool { return a[i].RootFS < a[j].RootFS }
+
+// KernelInfo defines kernels.toml entries
 type KernelInfo struct {
 	DistroType    DistroType
 	DistroRelease string // 18.04/7.4.1708/9.1
@@ -158,14 +226,19 @@ type KernelInfo struct {
 	KernelRelease string
 
 	// Build-time information
+	KernelSource  string // module/exploit will be build on host
 	ContainerName string
 
 	// Runtime information
 	KernelPath string
 	InitrdPath string
 	RootFS     string
+
+	// Debug symbols
+	VmlinuxPath string
 }
 
+// KernelConfig is the ~/.out-of-tree/kernels.toml configuration description
 type KernelConfig struct {
 	Kernels []KernelInfo
 }
@@ -181,6 +254,7 @@ func readFileAll(path string) (buf []byte, err error) {
 	return
 }
 
+// ReadKernelConfig is for read kernels.toml
 func ReadKernelConfig(path string) (kernelCfg KernelConfig, err error) {
 	buf, err := readFileAll(path)
 	if err != nil {
@@ -195,16 +269,93 @@ func ReadKernelConfig(path string) (kernelCfg KernelConfig, err error) {
 	return
 }
 
-func ReadArtifactConfig(path string) (artifactCfg Artifact, err error) {
+func rangeRegexp(start, end int) (s string) {
+	s += "("
+	for i := start; i <= end; i++ {
+		s += strconv.Itoa(i)
+		if i != end {
+			s += "|"
+		}
+	}
+	s += ")"
+	return
+}
+
+func versionRegexp(l []int) (s string, err error) {
+	switch len(l) {
+	case 1:
+		s += strconv.Itoa(l[0])
+	case 2:
+		s += rangeRegexp(l[0], l[1])
+	default:
+		err = errors.New("version must contain one value or range")
+		return
+	}
+	return
+}
+
+func genReleaseMask(km kernel) (mask string, err error) {
+	s, err := versionRegexp(km.Version)
+	if err != nil {
+		return
+	}
+	mask += s + "[.]"
+
+	s, err = versionRegexp(km.Major)
+	if err != nil {
+		return
+	}
+	mask += s + "[.]"
+
+	s, err = versionRegexp(km.Minor)
+	if err != nil {
+		return
+	}
+	mask += s
+
+	switch len(km.Patch) {
+	case 0:
+		// ok
+	case 1:
+		mask += "-" + strconv.Itoa(km.Patch[0]) + "-"
+	case 2:
+		mask += "-" + rangeRegexp(km.Patch[0], km.Patch[1]) + "-"
+	default:
+		err = errors.New("version must contain one value or range")
+		return
+	}
+
+	mask += ".*"
+	return
+}
+
+// ReadArtifactConfig is for read .out-of-tree.toml
+func ReadArtifactConfig(path string) (ka Artifact, err error) {
 	buf, err := readFileAll(path)
 	if err != nil {
 		return
 	}
 
-	err = toml.Unmarshal(buf, &artifactCfg)
+	err = toml.Unmarshal(buf, &ka)
 	if err != nil {
 		return
 	}
 
+	for i, _ := range ka.SupportedKernels {
+		km := &ka.SupportedKernels[i]
+		if len(km.Kernel.Version) != 0 && km.ReleaseMask != "" {
+			s := "Only one way to define kernel version is allowed"
+			err = errors.New(s)
+			return
+		}
+
+		if km.ReleaseMask == "" {
+			km.ReleaseMask, err = genReleaseMask(km.Kernel)
+			if err != nil {
+				return
+			}
+		}
+	}
+
 	return
 }

config/config_test.go

@@ -16,7 +16,7 @@ func TestMarshalUnmarshal(t *testing.T) {
 		Type: KernelModule,
 	}
 	artifactCfg.SupportedKernels = append(artifactCfg.SupportedKernels,
-		KernelMask{Ubuntu, "18.04", ".*"})
+		KernelMask{Ubuntu, "18.04", ".*", kernel{}})
 	buf, err := toml.Marshal(&artifactCfg)
 	if err != nil {
 		t.Fatal(err)
@@ -28,3 +28,38 @@ func TestMarshalUnmarshal(t *testing.T) {
 		t.Fatal(err)
 	}
 }
+
+func TestKernelRegex(t *testing.T) {
+	mask := "4[.]4[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99|100|101|102|103|104|105|106|107|108|109|110|111|112|113|114|115|116)-.*"
+	k := kernel{
+		Version: []int{4},
+		Major:   []int{4},
+		Minor:   []int{0},
+		Patch:   []int{1, 116},
+	}
+
+	gmask, err := genReleaseMask(k)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if mask != gmask {
+		t.Fatal("Got", gmask, "instead of", mask)
+	}
+
+	mask = "4[.]4[.]0.*"
+	k = kernel{
+		Version: []int{4},
+		Major:   []int{4},
+		Minor:   []int{0},
+	}
+
+	gmask, err = genReleaseMask(k)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if mask != gmask {
+		t.Fatal("Got", gmask, "instead of", mask)
+	}
+}

db.go

@@ -0,0 +1,317 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"strconv"
+	"time"
+
+	_ "github.com/mattn/go-sqlite3"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/qemu"
+)
+
+// Change on ANY database update
+const currentDatabaseVersion = 2
+
+const versionField = "db_version"
+
+type logEntry struct {
+	ID        int
+	Tag       string
+	Timestamp time.Time
+
+	qemu.System
+	config.Artifact
+	config.KernelInfo
+	phasesResult
+}
+
+func createLogTable(db *sql.DB) (err error) {
+	_, err = db.Exec(`
+	CREATE TABLE IF NOT EXISTS log (
+		id		INTEGER PRIMARY KEY,
+		time		DATETIME DEFAULT CURRENT_TIMESTAMP,
+		tag		TEXT,
+
+		name		TEXT,
+		type		TEXT,
+
+		distro_type	TEXT,
+		distro_release	TEXT,
+		kernel_release	TEXT,
+
+		build_output	TEXT,
+		build_ok	BOOLEAN,
+
+		run_output	TEXT,
+		run_ok		BOOLEAN,
+
+		test_output	TEXT,
+		test_ok		BOOLEAN,
+
+		qemu_stdout	TEXT,
+		qemu_stderr	TEXT,
+
+		kernel_panic	BOOLEAN,
+		timeout_kill	BOOLEAN
+	)`)
+	return
+}
+
+func createMetadataTable(db *sql.DB) (err error) {
+	_, err = db.Exec(`
+	CREATE TABLE IF NOT EXISTS metadata (
+		id	INTEGER PRIMARY KEY,
+		key	TEXT UNIQUE,
+		value	TEXT
+	)`)
+	return
+}
+
+func metaChkValue(db *sql.DB, key string) (exist bool, err error) {
+	sql := "SELECT EXISTS(SELECT id FROM metadata WHERE key = $1)"
+	stmt, err := db.Prepare(sql)
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	err = stmt.QueryRow(key).Scan(&exist)
+	return
+}
+
+func metaGetValue(db *sql.DB, key string) (value string, err error) {
+	stmt, err := db.Prepare("SELECT value FROM metadata " +
+		"WHERE key = $1")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	err = stmt.QueryRow(key).Scan(&value)
+	return
+}
+
+func metaSetValue(db *sql.DB, key, value string) (err error) {
+	stmt, err := db.Prepare("INSERT OR REPLACE INTO metadata " +
+		"(key, value) VALUES ($1, $2)")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	_, err = stmt.Exec(key, value)
+	return
+}
+
+func getVersion(db *sql.DB) (version int, err error) {
+	s, err := metaGetValue(db, versionField)
+	if err != nil {
+		return
+	}
+
+	version, err = strconv.Atoi(s)
+	return
+}
+
+func addToLog(db *sql.DB, q *qemu.System, ka config.Artifact,
+	ki config.KernelInfo, res *phasesResult, tag string) (err error) {
+
+	stmt, err := db.Prepare("INSERT INTO log (name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_output, build_ok, " +
+		"run_output, run_ok, " +
+		"test_output, test_ok, " +
+		"qemu_stdout, qemu_stderr, " +
+		"kernel_panic, timeout_kill) " +
+		"VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, " +
+		"$10, $11, $12, $13, $14, $15, $16);")
+	if err != nil {
+		return
+	}
+
+	defer stmt.Close()
+
+	_, err = stmt.Exec(
+		ka.Name, ka.Type, tag,
+		ki.DistroType, ki.DistroRelease, ki.KernelRelease,
+		res.Build.Output, res.Build.Ok,
+		res.Run.Output, res.Run.Ok,
+		res.Test.Output, res.Test.Ok,
+		q.Stdout, q.Stderr,
+		q.KernelPanic, q.KilledByTimeout,
+	)
+	if err != nil {
+		return
+	}
+
+	return
+}
+
+func getAllLogs(db *sql.DB, tag string, num int) (les []logEntry, err error) {
+	stmt, err := db.Prepare("SELECT id, time, name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_ok, run_ok, test_ok, kernel_panic, " +
+		"timeout_kill FROM log ORDER BY datetime(time) DESC " +
+		"LIMIT $1")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	rows, err := stmt.Query(num)
+	if err != nil {
+		return
+	}
+	defer rows.Close()
+
+	for rows.Next() {
+		le := logEntry{}
+		err = rows.Scan(&le.ID, &le.Timestamp,
+			&le.Name, &le.Type, &le.Tag,
+			&le.DistroType, &le.DistroRelease, &le.KernelRelease,
+			&le.Build.Ok, &le.Run.Ok, &le.Test.Ok,
+			&le.KernelPanic, &le.KilledByTimeout,
+		)
+		if err != nil {
+			return
+		}
+
+		if tag == "" || tag == le.Tag {
+			les = append(les, le)
+		}
+	}
+
+	return
+}
+
+func getAllArtifactLogs(db *sql.DB, tag string, num int, ka config.Artifact) (
+	les []logEntry, err error) {
+
+	stmt, err := db.Prepare("SELECT id, time, name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_ok, run_ok, test_ok, kernel_panic, " +
+		"timeout_kill FROM log WHERE name=$1 AND type=$2 " +
+		"ORDER BY datetime(time) DESC LIMIT $3")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	rows, err := stmt.Query(ka.Name, ka.Type, num)
+	if err != nil {
+		return
+	}
+	defer rows.Close()
+
+	for rows.Next() {
+		le := logEntry{}
+		err = rows.Scan(&le.ID, &le.Timestamp,
+			&le.Name, &le.Type, &le.Tag,
+			&le.DistroType, &le.DistroRelease, &le.KernelRelease,
+			&le.Build.Ok, &le.Run.Ok, &le.Test.Ok,
+			&le.KernelPanic, &le.KilledByTimeout,
+		)
+		if err != nil {
+			return
+		}
+
+		if tag == "" || tag == le.Tag {
+			les = append(les, le)
+		}
+	}
+
+	return
+}
+
+func getLogByID(db *sql.DB, id int) (le logEntry, err error) {
+	stmt, err := db.Prepare("SELECT id, time, name, type, tag, " +
+		"distro_type, distro_release, kernel_release, " +
+		"build_ok, run_ok, test_ok, " +
+		"build_output, run_output, test_output, " +
+		"qemu_stdout, qemu_stderr, " +
+		"kernel_panic, timeout_kill " +
+		"FROM log WHERE id=$1")
+	if err != nil {
+		return
+	}
+	defer stmt.Close()
+
+	err = stmt.QueryRow(id).Scan(&le.ID, &le.Timestamp,
+		&le.Name, &le.Type, &le.Tag,
+		&le.DistroType, &le.DistroRelease, &le.KernelRelease,
+		&le.Build.Ok, &le.Run.Ok, &le.Test.Ok,
+		&le.Build.Output, &le.Run.Output, &le.Test.Output,
+		&le.Stdout, &le.Stderr,
+		&le.KernelPanic, &le.KilledByTimeout,
+	)
+	return
+}
+
+func createSchema(db *sql.DB) (err error) {
+	err = createMetadataTable(db)
+	if err != nil {
+		return
+	}
+
+	err = createLogTable(db)
+	if err != nil {
+		return
+	}
+
+	return
+}
+
+func openDatabase(path string) (db *sql.DB, err error) {
+	db, err = sql.Open("sqlite3", path)
+	if err != nil {
+		return
+	}
+
+	db.SetMaxOpenConns(1)
+
+	exists, _ := metaChkValue(db, versionField)
+	if !exists {
+		err = createSchema(db)
+		if err != nil {
+			return
+		}
+
+		err = metaSetValue(db, versionField,
+			strconv.Itoa(currentDatabaseVersion))
+		return
+	}
+
+	version, err := getVersion(db)
+	if err != nil {
+		return
+	}
+
+	if version == 1 {
+		_, err = db.Exec(`ALTER TABLE log ADD tag TEXT`)
+		if err != nil {
+			return
+		}
+
+		err = metaSetValue(db, versionField, "2")
+		if err != nil {
+			return
+		}
+
+		version = 2
+	}
+
+	if version != currentDatabaseVersion {
+		err = fmt.Errorf("Database is not supported (%d instead of %d)",
+			version, currentDatabaseVersion)
+		return
+	}
+
+	return
+}

debug.go

@@ -13,9 +13,10 @@ import (
 	"strings"
 	"time"
 
-	"github.com/jollheef/out-of-tree/config"
+	"gopkg.in/logrusorgru/aurora.v1"
-	qemu "github.com/jollheef/out-of-tree/qemu"
+
-	"github.com/logrusorgru/aurora"
+	"code.dumpstack.io/tools/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/qemu"
 )
 
 func firstSupported(kcfg config.KernelConfig, ka config.Artifact,
@@ -40,7 +41,7 @@ func firstSupported(kcfg config.KernelConfig, ka config.Artifact,
 	return
 }
 
-func handleLine(q *qemu.QemuSystem) (err error) {
+func handleLine(q *qemu.System) (err error) {
 	fmt.Print("out-of-tree> ")
 	rawLine := "help"
 	fmt.Scanf("%s", &rawLine)
@@ -63,7 +64,7 @@ func handleLine(q *qemu.QemuSystem) (err error) {
 	case "c", "cleanup":
 		q.Stdout = []byte{}
 	case "s", "ssh":
-		fmt.Println(q.GetSshCommand())
+		fmt.Println(q.GetSSHCommand())
 	case "q", "quit":
 		return errors.New("end of session")
 	default:
@@ -72,7 +73,7 @@ func handleLine(q *qemu.QemuSystem) (err error) {
 	return
 }
 
-func interactive(q *qemu.QemuSystem) (err error) {
+func interactive(q *qemu.System) (err error) {
 	for {
 		err = handleLine(q)
 		if err != nil {
@@ -82,7 +83,8 @@ func interactive(q *qemu.QemuSystem) (err error) {
 }
 
 func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
-	dockerTimeout time.Duration) (err error) {
+	dockerTimeout time.Duration, yekaslr, yesmep, yesmap, yekpti,
+	nokaslr, nosmep, nosmap, nokpti bool) (err error) {
 
 	ka, err := config.ReadArtifactConfig(workPath + "/.out-of-tree.toml")
 	if err != nil {
@@ -99,10 +101,64 @@ func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
 	}
 
 	kernel := qemu.Kernel{KernelPath: ki.KernelPath, InitrdPath: ki.InitrdPath}
-	q, err := qemu.NewQemuSystem(qemu.X86_64, kernel, ki.RootFS)
+	q, err := qemu.NewSystem(qemu.X86x64, kernel, ki.RootFS)
 	if err != nil {
 		return
 	}
+
+	if ka.Qemu.Cpus != 0 {
+		q.Cpus = ka.Qemu.Cpus
+	}
+	if ka.Qemu.Memory != 0 {
+		q.Memory = ka.Qemu.Memory
+	}
+
+	q.SetKASLR(false) // set KASLR to false by default because of gdb
+	q.SetSMEP(!ka.Mitigations.DisableSmep)
+	q.SetSMAP(!ka.Mitigations.DisableSmap)
+	q.SetKPTI(!ka.Mitigations.DisableKpti)
+
+	if yekaslr {
+		q.SetKASLR(true)
+	} else if nokaslr {
+		q.SetKASLR(false)
+	}
+
+	if yesmep {
+		q.SetSMEP(true)
+	} else if nosmep {
+		q.SetSMEP(false)
+	}
+
+	if yesmap {
+		q.SetSMAP(true)
+	} else if nosmap {
+		q.SetSMAP(false)
+	}
+
+	if yekpti {
+		q.SetKPTI(true)
+	} else if nokpti {
+		q.SetKPTI(false)
+	}
+
+	redgreen := func(name string, enabled bool) aurora.Value {
+		if enabled {
+			return aurora.BgGreen(aurora.Black(name))
+		}
+
+		return aurora.BgRed(aurora.Gray(name))
+	}
+
+	fmt.Printf("[*] %s %s %s %s\n",
+		redgreen("KASLR", q.GetKASLR()),
+		redgreen("SMEP", q.GetSMEP()),
+		redgreen("SMAP", q.GetSMAP()),
+		redgreen("KPTI", q.GetKPTI()))
+
+	fmt.Printf("[*] SMP: %d CPUs\n", q.Cpus)
+	fmt.Printf("[*] Memory: %d MB\n", q.Memory)
+
 	q.Debug(gdb)
 	coloredGdbAddress := aurora.BgGreen(aurora.Black(gdb))
 	fmt.Printf("[*] gdb runned on %s\n", coloredGdbAddress)
@@ -125,9 +181,9 @@ func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
 		return
 	}
 
-	remoteFile := "/tmp/artifact"
+	remoteFile := "/tmp/exploit"
 	if ka.Type == config.KernelModule {
-		remoteFile += ".ko"
+		remoteFile = "/tmp/module.ko"
 	}
 
 	err = q.CopyFile("user", outFile, remoteFile)
@@ -138,6 +194,11 @@ func debugHandler(kcfg config.KernelConfig, workPath, kernRegex, gdb string,
 	coloredRemoteFile := aurora.BgGreen(aurora.Black(remoteFile))
 	fmt.Printf("[*] build result copied to %s\n", coloredRemoteFile)
 
+	fmt.Printf("\n%s\n", q.GetSSHCommand())
+	fmt.Printf("gdb %s -ex 'target remote %s'\n\n", ki.VmlinuxPath, gdb)
+
+	// TODO set substitute-path /build/.../linux-... /path/to/linux-source
+
 	err = interactive(q)
 	return
 }

docs/index.rst

@@ -0,0 +1,30 @@
+out-of-tree
+===========
+
+*out-of-tree* is the kernel {module, exploit} development tool.
+
+*out-of-tree* was created on the purpose of decreasing complexity of
+environment for developing, testing and debugging Linux kernel
+exploits and out-of-tree kernel modules (that's why tool got a name
+"out-of-tree").
+
+While I'm trying to keep that documentation up-to-date, there may be
+some missing information. Use ``out-of-tree --help-long`` for checking
+all features.
+
+If you found anything missed here, please make a pull request or send
+patches to patch@dumpstack.io.
+
+If you need personal support, your company is interested in the
+project or you just want to share some thoughts -- feel free to write
+to root@dumpstack.io.
+
+Contents
+========
+
+:ref:`Keyword Index <genindex>`
+
+.. toctree::
+
+   introduction.rst
+   installation.rst

docs/installation.rst

@@ -0,0 +1,64 @@
+Installation
+============
+
+OS/Distro-specific
+==================
+
+Ubuntu
+------
+
+Install dependencies::
+
+  $ sudo snap install go --classic
+  $ sudo snap install docker
+  $ sudo apt install qemu-system-x86 build-essential gdb
+
+macOS
+-----
+
+Install dependencies::
+
+  $ brew install go qemu
+  $ brew cask install docker
+
+NixOS
+-----
+
+There's a minimal configuration that you need to apply::
+
+  #!nix
+  { config, pkgs, ... }:
+  {
+    virtualisation.docker.enable = true;
+    virtualisation.libvirtd.enable = true;
+    environment.systemPackages = with pkgs; [
+      go git
+    ];
+  }
+
+Common
+======
+
+Setup Go environment::
+
+  $ echo 'export GOPATH=$HOME' >> ~/.bashrc
+  $ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
+  $ source ~/.bashrc
+
+Build *out-of-tree*::
+
+  $ go get -u code.dumpstack.io/tools/out-of-tree
+
+.. note::
+  On a GNU/Linux you need to add your user to docker group if you want
+  to use *out-of-tree* without sudo. Note that this has a **serious**
+  security implications. Check *Docker* documentation for more
+  information.
+
+Test that everything works::
+
+  $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/examples/kernel-exploit
+  $ out-of-tree kernel autogen --max=1
+  $ out-of-tree pew --max=1
+
+Enjoy!

docs/introduction.rst

@@ -0,0 +1,109 @@
+Introduction
+============
+
+*out-of-tree* is written in *Go*, it uses *Docker* for generating
+kernel/filesystem images and *Qemu* for virtualization.
+
+Also it possible to generate kernels from the host system and use the
+custom one.
+
+*out-of-tree* supports *GNU/Linux* (usually it's tested on NixOS and
+latest Ubuntu LTS) and *macOS*. Technically all systems that supported
+by Go, Docker, and Qemu must work well. Create the issue if you'll
+notice any issue in integration for your operating system.
+
+All *Qemu* interaction is stateless.
+
+*out-of-tree* is allow and require metadata (``.out-of-tree.toml``)
+for work. TOML (Tom's Obvious, Minimal Language) is used for kernel
+module/exploit description.
+
+``.out-of-tree.toml`` is mandatory, you need to have in the current
+directory (usually, it's a project of kernel module/exploit) or use
+the ``--path`` flag.
+
+Files
+-----
+
+All data is stored in ``~/.out-of-tree/``.
+
+- *db.sqlite* contains logs related to run with ``out-of-tree pew``,
+  debug mode (``out-of-tree debug``) is not store any data.
+
+- *images* used for filesystem images (rootfs images that used for
+  ``qemu -hda ...``) that can be generated with the
+  ``tools/qemu-*-img/...``.
+
+- *kernels* stores all kernel ``vmlinuz/initrd/config/...`` files that
+  generated previously with a some *Docker magic*.
+
+- *kernels.toml* contains metadata for generated kernels. It's not
+  supposed to be edited by hands.
+
+- *kernels.user.toml* is default path for custom kernels definition.
+
+- *Ubuntu* (or *Centos*/*Debian*/...) is the Dockerfiles tree
+  (DistroName/DistroVersion/Dockerfile). Each Dockerfile contains a
+  base layer and incrementally updated list of kernels that must be
+  installed.
+
+Overview
+---------
+
+*out-of-tree* creating debugging environment based on **defined** kernels::
+
+    $ out-of-tree debug --kernel 'Ubuntu:4.15.0-58-generic'
+    [*] KASLR SMEP SMAP
+    [*] gdb runned on tcp::1234
+    [*] build result copied to /tmp/exploit
+
+    ssh -o StrictHostKeyChecking=no -p 29308 root@127.133.45.236
+    gdb /usr/lib/debug/boot/vmlinux-4.15.0-58-generic -ex 'target remote tcp::1234'
+
+    out-of-tree> help
+    help    : print this help message
+    log     : print qemu log
+    clog    : print qemu log and cleanup buffer
+    cleanup : cleanup qemu log buffer
+    ssh     : print arguments to ssh command
+    quit    : quit
+    out-of-tree>
+
+*out-of-tree* uses three stages for automated runs:
+
+- Build
+
+  - Inside the docker container (default).
+  - Binary version (de facto skip stage).
+  - On host.
+
+- Run
+
+  - Insmod for the kernel module.
+  - This step is skipped for exploits.
+
+- Test
+
+  - Run the test.sh script on the target machine.
+  - Test script is run from *root* for the kernel module.
+  - Test script is run from *user* for the kernel exploit.
+  - Test script for the kernel module is fully custom (only return
+    value is checked).
+  - Test script for the kernel exploit receives two parameters:
+
+    - Path to exploit
+    - Path to file that must be created with root privileges.
+
+  - If there's no test.sh script then default
+    (``echo touch FILE | exploit``) one is used.
+
+Security
+--------
+
+*out-of-tree* is not supposed to be used on multi-user systems or with
+an untrusted input.
+
+Meanwhile, all modern hypervisors are supporting nested
+virtualization, which means you can use it for isolating *out-of-tree*
+if you want to work with an untrusted input (e.g. with a mass-scale
+testing public proofs-of-concept).

examples/kernel-exploit/.out-of-tree.toml

@@ -6,12 +6,12 @@ type = "exploit"
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.4.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99|100|101|102|103|104|105|106|107|108|109|110|111|112|113|114|115|116)-.*"
+release_mask = "4[.]4[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99|100|101|102|103|104|105|106|107|108|109|110|111|112|113|114|115|116)-.*"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.8.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58)-.*"
+release_mask = "4[.]8[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58)-.*"
 
 [[supported_kernels]]
 # Can be Ubuntu/CentOS/Debian/etc.
@@ -20,14 +20,19 @@ distro_release = "16.04"
 # regex for `uname -r`
 # See also: regex-golang.appspot.com
 # stupid way to generate: $ echo '4.4.0-('$(seq 44 | xargs echo | sed 's/ /|/g')')-.*'
-release_mask = "4.10.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42)-.*"
+release_mask = "4[.]10[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42)-.*"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.11.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14)-.*"
+release_mask = "4[.]11[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14)-.*"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "16.04"
-release_mask = "4.13.0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21)-.*"
+# equivalent for "4[.]13[.]0-(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21)-.*"
+[supported_kernels.kernel]
+version = [ 4 ]
+major = [ 13 ]
+minor = [ 0 ]
+patch = [ 1, 21 ]

examples/kernel-exploit/CVE-2017-16995.c

@@ -317,6 +317,7 @@ void redact(const char *fmt, ...) {
 	va_start(args, fmt);
 	if(doredact) {
 		fprintf(stdout, "[!] ( ( R E D A C T E D ) )\n");
+		va_end(args);
 		return;
 	}
 	fprintf(stdout, "[*] ");

examples/kernel-module/.out-of-tree.toml

@@ -9,17 +9,22 @@ distro_type = "Ubuntu"
 distro_release = "16.04"
 # regex for `uname -r`
 # See also: regex-golang.appspot.com
-release_mask = "4.4.0-70-.*"
+release_mask = "4[.]4[.]0-70-.*"
 
 # [[supported_kernels]] may be defined unlimited number of times
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "18.04"
 # Also you can use only one kernel
-release_mask = "4.15.0-(24|29)-generic"
+release_mask = "4[.]15[.]0-(24|29)-generic"
 
 [[supported_kernels]]
 distro_type = "Ubuntu"
 distro_release = "18.04"
 # Also you can use only one kernel
-release_mask = "4.15.0-23-generic"
+release_mask = "4[.]15[.]0-23-generic"
+
+[[supported_kernels]]
+distro_type = "CentOS"
+distro_release = "7"
+release_mask = "3[.]10[.]0-862.el7.x86_64"

gen.go

@@ -7,8 +7,9 @@ package main
 import (
 	"fmt"
 
-	"github.com/jollheef/out-of-tree/config"
 	"github.com/naoina/toml"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
 )
 
 func genConfig(at config.ArtifactType) (err error) {
@@ -17,7 +18,9 @@ func genConfig(at config.ArtifactType) (err error) {
 		Type: at,
 	}
 	a.SupportedKernels = append(a.SupportedKernels, config.KernelMask{
-		config.Ubuntu, "18.04", ".*",
+		DistroType:    config.Ubuntu,
+		DistroRelease: "18.04",
+		ReleaseMask:   ".*",
 	})
 
 	buf, err := toml.Marshal(&a)

go.mod

@@ -0,0 +1,21 @@
+module code.dumpstack.io/tools/out-of-tree
+
+replace code.dumpstack.io/tools/out-of-tree/qemu => ./qemu
+
+replace code.dumpstack.io/tools/out-of-tree/config => ./config
+
+require (
+	github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc // indirect
+	github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf // indirect
+	github.com/mattn/go-runewidth v0.0.4 // indirect
+	github.com/mattn/go-sqlite3 v1.11.0
+	github.com/naoina/go-stringutil v0.1.0 // indirect
+	github.com/naoina/toml v0.1.1
+	github.com/olekukonko/tablewriter v0.0.1
+	github.com/otiai10/copy v1.0.1
+	github.com/remeh/sizedwaitgroup v0.0.0-20180822144253-5e7302b12cce
+	github.com/zcalusic/sysinfo v0.0.0-20190429151633-fbadb57345c2
+	golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5
+	gopkg.in/alecthomas/kingpin.v2 v2.2.6
+	gopkg.in/logrusorgru/aurora.v1 v1.0.0-20181002194514-a7b3b318ed4e
+)

go.sum

@@ -0,0 +1,33 @@
+bou.ke/monkey v1.0.1/go.mod h1:FgHuK96Rv2Nlf+0u1OOVDpCMdsWyOFmeeketDHE7LIg=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc h1:cAKDfWh5VpdgMhJosfJnn5/FoN2SRZ4p7fJNX58YPaU=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZqLG4oE62mJzwPIB8+Tee4RNCL9ulrY=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/mattn/go-runewidth v0.0.4 h1:2BvfKmzob6Bmd4YsL0zygOqfdFnK7GR4QL06Do4/p7Y=
+github.com/mattn/go-runewidth v0.0.4/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
+github.com/mattn/go-sqlite3 v1.11.0 h1:LDdKkqtYlom37fkvqs8rMPFKAMe8+SgjbwZ6ex1/A/Q=
+github.com/mattn/go-sqlite3 v1.11.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
+github.com/naoina/go-stringutil v0.1.0 h1:rCUeRUHjBjGTSHl0VC00jUPLz8/F9dDzYI70Hzifhks=
+github.com/naoina/go-stringutil v0.1.0/go.mod h1:XJ2SJL9jCtBh+P9q5btrd/Ylo8XwT/h1USek5+NqSA0=
+github.com/naoina/toml v0.1.1 h1:PT/lllxVVN0gzzSqSlHEmP8MJB4MY2U7STGxiouV4X8=
+github.com/naoina/toml v0.1.1/go.mod h1:NBIhNtsFMo3G2szEBne+bO4gS192HuIYRqfvOWb4i1E=
+github.com/olekukonko/tablewriter v0.0.1 h1:b3iUnf1v+ppJiOfNX4yxxqfWKMQPZR5yoh8urCTFX88=
+github.com/olekukonko/tablewriter v0.0.1/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo=
+github.com/otiai10/copy v1.0.1 h1:gtBjD8aq4nychvRZ2CyJvFWAw0aja+VHazDdruZKGZA=
+github.com/otiai10/copy v1.0.1/go.mod h1:8bMCJrAqOtN/d9oyh5HR7HhLQMvcGMpGdwRDYsfOCHc=
+github.com/otiai10/mint v1.2.3/go.mod h1:YnfyPNhBvnY8bW4SGQHCs/aAFhkgySlMZbrF5U0bOVw=
+github.com/remeh/sizedwaitgroup v0.0.0-20180822144253-5e7302b12cce h1:aP+C+YbHZfOQlutA4p4soHi7rVUqHQdWEVMSkHfDTqY=
+github.com/remeh/sizedwaitgroup v0.0.0-20180822144253-5e7302b12cce/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
+github.com/zcalusic/sysinfo v0.0.0-20190429151633-fbadb57345c2 h1:uMiaKNX5zFLOa6nNtun+d/lpV5bOBh7BvE4q9jfZacQ=
+github.com/zcalusic/sysinfo v0.0.0-20190429151633-fbadb57345c2/go.mod h1:zAn3FAIbgZPYnutDND49Ivf8sb/mXYk8UjZdqMswgHg=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5 h1:58fnuSXlxZmFdJyvtTFVmVhcMLU6v5fEb/ok4wyqtNU=
+golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/logrusorgru/aurora.v1 v1.0.0-20181002194514-a7b3b318ed4e h1:uKdf1KQDFZDYqNzSDhxB5hFxj5Fq4e3/C/ejtRJxlY0=
+gopkg.in/logrusorgru/aurora.v1 v1.0.0-20181002194514-a7b3b318ed4e/go.mod h1:DGR33jeYG1jxERD2W4hGjuW94Pxf3mkUf/Ddhf5BskA=

images.config.go

@@ -0,0 +1,7 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+const imagesBaseURL = "https://out-of-tree.fra1.digitaloceanspaces.com/1.0.0/"

images.go

@@ -0,0 +1,82 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"os"
+	"os/exec"
+)
+
+// inspired by Edd Turtle code
+func downloadFile(filepath string, url string) (err error) {
+	out, err := os.Create(filepath)
+	if err != nil {
+		return
+	}
+	defer out.Close()
+
+	resp, err := http.Get(url)
+	if err != nil {
+		return
+	}
+	defer resp.Body.Close()
+
+	switch resp.StatusCode {
+	case http.StatusOK:
+		break
+	case http.StatusForbidden, http.StatusNotFound:
+		err = fmt.Errorf("Cannot download %s. It looks like you need "+
+			"to generate it manually and place it "+
+			"to ~/.out-of-tree/images/. "+
+			"Check documentation for additional information.", url)
+		return
+	default:
+		err = fmt.Errorf("Something weird happens while "+
+			"download file: %d", resp.StatusCode)
+		return
+	}
+
+	_, err = io.Copy(out, resp.Body)
+	return
+}
+
+func unpackTar(archive, destination string) (err error) {
+	// NOTE: If you're change anything in tar command please check also
+	// BSD tar (or if you're using macOS, do not forget to check GNU Tar)
+	// Also make sure that sparse files are extracting correctly
+	cmd := exec.Command("tar", "-Sxf", archive)
+	cmd.Dir = destination + "/"
+
+	rawOutput, err := cmd.CombinedOutput()
+	if err != nil {
+		err = fmt.Errorf("%v: %s", err, rawOutput)
+		return
+	}
+
+	return
+}
+
+func downloadImage(path, file string) (err error) {
+	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_")
+	if err != nil {
+		return
+	}
+	defer os.RemoveAll(tmp)
+
+	archive := tmp + "/" + file + ".tar.gz"
+	url := imagesBaseURL + file + ".tar.gz"
+
+	err = downloadFile(archive, url)
+	if err != nil {
+		return
+	}
+
+	err = unpackTar(archive, path)
+	return
+}

kernel.go

@@ -9,16 +9,23 @@ import (
 	"fmt"
 	"io/ioutil"
 	"log"
+	"math"
+	"math/rand"
 	"os"
 	"os/exec"
 	"os/user"
 	"regexp"
 	"strings"
+	"time"
 
-	"github.com/jollheef/out-of-tree/config"
 	"github.com/naoina/toml"
+	"github.com/zcalusic/sysinfo"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
 )
 
+const kernelsAll int64 = math.MaxInt64
+
 func kernelListHandler(kcfg config.KernelConfig) (err error) {
 	if len(kcfg.Kernels) == 0 {
 		return errors.New("No kernels found")
@@ -29,34 +36,58 @@ func kernelListHandler(kcfg config.KernelConfig) (err error) {
 	return
 }
 
-func matchDebianKernelPkg(container, mask string, generic bool) (pkgs []string,
+func matchDebianHeadersPkg(container, mask string, generic bool) (
-	err error) {
+	pkgs []string, err error) {
 
-	cmd := "apt-cache search linux-image | cut -d ' ' -f 1"
+	cmd := "apt-cache search linux-headers | cut -d ' ' -f 1"
-	c := dockerCommand(container, "/tmp", "1m", cmd)
+	output, err := dockerRun(time.Minute, container, "/tmp", cmd)
-	rawOutput, err := c.CombinedOutput()
 	if err != nil {
 		return
 	}
 
-	r, err := regexp.Compile("linux-image-" + mask)
+	r, err := regexp.Compile("linux-headers-" + mask)
 	if err != nil {
 		return
 	}
 
-	kernels := r.FindAll(rawOutput, -1)
+	kernels := r.FindAll([]byte(output), -1)
 
 	for _, k := range kernels {
 		pkg := string(k)
 		if generic && !strings.HasSuffix(pkg, "generic") {
 			continue
 		}
+		if pkg == "linux-headers-generic" {
+			continue
+		}
 		pkgs = append(pkgs, pkg)
 	}
 
 	return
 }
 
+func matchCentOSDevelPkg(container, mask string, generic bool) (
+	pkgs []string, err error) {
+
+	cmd := "yum search kernel-devel --show-duplicates | " +
+		"grep '^kernel-devel' | cut -d ' ' -f 1"
+	output, err := dockerRun(time.Minute, container, "/tmp", cmd)
+	if err != nil {
+		return
+	}
+
+	r, err := regexp.Compile("kernel-devel-" + mask)
+	if err != nil {
+		return
+	}
+
+	for _, k := range r.FindAll([]byte(output), -1) {
+		pkgs = append(pkgs, string(k))
+	}
+
+	return
+}
+
 func dockerImagePath(sk config.KernelMask) (path string, err error) {
 	usr, err := user.Current()
 	if err != nil {
@@ -68,6 +99,16 @@ func dockerImagePath(sk config.KernelMask) (path string, err error) {
 	return
 }
 
+func vsyscallAvailable() (available bool, err error) {
+	buf, err := ioutil.ReadFile("/proc/self/maps")
+	if err != nil {
+		return
+	}
+
+	available = strings.Contains(string(buf), "[vsyscall]")
+	return
+}
+
 func generateBaseDockerImage(sk config.KernelMask) (err error) {
 	imagePath, err := dockerImagePath(sk)
 	if err != nil {
@@ -81,26 +122,52 @@ func generateBaseDockerImage(sk config.KernelMask) (err error) {
 		log.Printf("Base image for %s:%s found",
 			sk.DistroType.String(), sk.DistroRelease)
 		return
-	} else {
+	}
+
 	log.Printf("Base image for %s:%s not found, start generating",
 		sk.DistroType.String(), sk.DistroRelease)
 	os.MkdirAll(imagePath, os.ModePerm)
-	}
 
 	d += fmt.Sprintf("FROM %s:%s\n",
 		strings.ToLower(sk.DistroType.String()),
 		sk.DistroRelease,
 	)
 
+	vsyscall, err := vsyscallAvailable()
+	if err != nil {
+		return
+	}
+
 	switch sk.DistroType {
 	case config.Ubuntu:
 		d += "ENV DEBIAN_FRONTEND=noninteractive\n"
 		d += "RUN apt-get update\n"
 		d += "RUN apt-get install -y build-essential libelf-dev\n"
 		d += "RUN apt-get install -y wget git\n"
+		if sk.DistroRelease >= "14.04" {
+			d += "RUN apt-get install -y libseccomp-dev\n"
+		}
+		d += "RUN mkdir /lib/modules\n"
+	case config.CentOS:
+		if sk.DistroRelease < "7" && !vsyscall {
+			log.Println("Old CentOS requires `vsyscall=emulate` " +
+				"on the latest kernels")
+			log.Println("Check out `A note about vsyscall` " +
+				"at https://hub.docker.com/_/centos")
+			log.Println("See also https://lwn.net/Articles/446528/")
+			err = fmt.Errorf("vsyscall is not available")
+			return
+		}
+
+		// enable rpms from old minor releases
+		d += "RUN sed -i 's/enabled=0/enabled=1/' /etc/yum.repos.d/CentOS-Vault.repo\n"
+		// do not remove old kernels
+		d += "RUN sed -i 's;installonly_limit=;installonly_limit=100500;' /etc/yum.conf\n"
+		d += "RUN yum -y update\n"
+		d += "RUN yum -y groupinstall 'Development Tools'\n"
+		d += "RUN yum -y install deltarpm\n"
 	default:
-		s := fmt.Sprintf("%s not yet supported", sk.DistroType.String())
+		err = fmt.Errorf("%s not yet supported", sk.DistroType.String())
-		err = errors.New(s)
 		return
 	}
 
@@ -144,11 +211,32 @@ func dockerImageAppend(sk config.KernelMask, pkgname string) (err error) {
 		return
 	}
 
-	log.Printf("Start adding kernel %s for %s:%s",
+	var s string
-		pkgname, sk.DistroType.String(), sk.DistroRelease)
 
-	s := fmt.Sprintf("RUN apt-get install -y %s %s\n", pkgname,
+	switch sk.DistroType {
-		strings.Replace(pkgname, "image", "headers", -1))
+	case config.Ubuntu:
+		imagepkg := strings.Replace(pkgname, "headers", "image", -1)
+
+		log.Printf("Start adding kernel %s for %s:%s",
+			imagepkg, sk.DistroType.String(), sk.DistroRelease)
+
+		s = fmt.Sprintf("RUN apt-get install -y %s %s\n", imagepkg,
+			pkgname)
+	case config.CentOS:
+		imagepkg := strings.Replace(pkgname, "-devel", "", -1)
+
+		log.Printf("Start adding kernel %s for %s:%s",
+			imagepkg, sk.DistroType.String(), sk.DistroRelease)
+
+		version := strings.Replace(pkgname, "kernel-devel-", "", -1)
+
+		s = fmt.Sprintf("RUN yum -y install %s %s\n", imagepkg,
+			pkgname)
+		s += fmt.Sprintf("RUN dracut --add-drivers 'e1000 ext4' -f "+
+			"/boot/initramfs-%s.img %s\n", version, version)
+	default:
+		err = fmt.Errorf("%s not yet supported", sk.DistroType.String())
+	}
 
 	err = ioutil.WriteFile(imagePath+"/Dockerfile",
 		append(raw, []byte(s)...), 0644)
@@ -226,7 +314,7 @@ func copyKernels(name string) (err error) {
 
 func genKernelPath(files []os.FileInfo, kname string) string {
 	for _, file := range files {
-		if strings.Contains(file.Name(), "vmlinuz") {
+		if strings.HasPrefix(file.Name(), "vmlinuz") {
 			if strings.Contains(file.Name(), kname) {
 				return file.Name()
 			}
@@ -237,7 +325,9 @@ func genKernelPath(files []os.FileInfo, kname string) string {
 
 func genInitrdPath(files []os.FileInfo, kname string) string {
 	for _, file := range files {
-		if strings.Contains(file.Name(), "initrd") {
+		if strings.HasPrefix(file.Name(), "initrd") ||
+			strings.HasPrefix(file.Name(), "initramfs") {
+
 			if strings.Contains(file.Name(), kname) {
 				return file.Name()
 			}
@@ -246,13 +336,24 @@ func genInitrdPath(files []os.FileInfo, kname string) string {
 	return "unknown"
 }
 
-func genRootfsImage(d dockerImageInfo) string {
+func genRootfsImage(d dockerImageInfo, download bool) (rootfs string, err error) {
 	usr, err := user.Current()
 	if err != nil {
-		return fmt.Sprintln(err)
+		return
 	}
 	imageFile := d.ContainerName + ".img"
-	return usr.HomeDir + "/.out-of-tree/images/" + imageFile
+
+	imagesPath := usr.HomeDir + "/.out-of-tree/images/"
+	os.MkdirAll(imagesPath, os.ModePerm)
+
+	rootfs = imagesPath + imageFile
+	if !exists(rootfs) {
+		if download {
+			log.Println(imageFile, "not exists, start downloading...")
+			err = downloadImage(imagesPath, imageFile)
+		}
+	}
+	return
 }
 
 type dockerImageInfo struct {
@@ -296,16 +397,85 @@ func listDockerImages() (diis []dockerImageInfo, err error) {
 	return
 }
 
-func updateKernelsCfg() (err error) {
+func genHostKernels(download bool) (kcfg config.KernelConfig, err error) {
+	si := sysinfo.SysInfo{}
+	si.GetSysInfo()
+
+	distroType, err := config.NewDistroType(si.OS.Vendor)
+	if err != nil {
+		return
+	}
+
+	cmd := exec.Command("ls", "/lib/modules")
+	rawOutput, err := cmd.CombinedOutput()
+	if err != nil {
+		log.Println(string(rawOutput), err)
+		return
+	}
+
+	kernelsBase := "/boot/"
+	files, err := ioutil.ReadDir(kernelsBase)
+	if err != nil {
+		return
+	}
+
+	// only for compatibility, docker is not really used
+	dii := dockerImageInfo{
+		ContainerName: config.KernelMask{
+			DistroType:    distroType,
+			DistroRelease: si.OS.Version,
+		}.DockerName(),
+	}
+
+	rootfs, err := genRootfsImage(dii, download)
+	if err != nil {
+		return
+	}
+
+	for _, k := range strings.Fields(string(rawOutput)) {
+		ki := config.KernelInfo{
+			DistroType:    distroType,
+			DistroRelease: si.OS.Version,
+			KernelRelease: k,
+
+			KernelSource: "/lib/modules/" + k + "/build",
+
+			KernelPath: kernelsBase + genKernelPath(files, k),
+			InitrdPath: kernelsBase + genInitrdPath(files, k),
+			RootFS:     rootfs,
+		}
+
+		vmlinux := "/usr/lib/debug/boot/vmlinux-" + k
+		log.Println("vmlinux", vmlinux)
+		if exists(vmlinux) {
+			ki.VmlinuxPath = vmlinux
+		}
+
+		kcfg.Kernels = append(kcfg.Kernels, ki)
+	}
+
+	return
+}
+
+func updateKernelsCfg(host, download bool) (err error) {
+	newkcfg := config.KernelConfig{}
+
+	if host {
+		// Get host kernels
+		newkcfg, err = genHostKernels(download)
+		if err != nil {
+			return
+		}
+	}
+
+	// Get docker kernels
 	dockerImages, err := listDockerImages()
 	if err != nil {
 		return
 	}
 
-	newkcfg := config.KernelConfig{}
-
 	for _, d := range dockerImages {
-		err = genKernels(d, &newkcfg)
+		err = genDockerKernels(d, &newkcfg, download)
 		if err != nil {
 			log.Println("gen kernels", d.ContainerName, ":", err)
 			continue
@@ -342,8 +512,8 @@ func updateKernelsCfg() (err error) {
 	return
 }
 
-func genKernels(dii dockerImageInfo, newkcfg *config.KernelConfig) (
+func genDockerKernels(dii dockerImageInfo, newkcfg *config.KernelConfig,
-	err error) {
+	download bool) (err error) {
 
 	name := dii.ContainerName
 	cmd := exec.Command("docker", "run", name, "ls", "/lib/modules")
@@ -363,6 +533,11 @@ func genKernels(dii dockerImageInfo, newkcfg *config.KernelConfig) (
 		return
 	}
 
+	rootfs, err := genRootfsImage(dii, download)
+	if err != nil {
+		return
+	}
+
 	for _, k := range strings.Fields(string(rawOutput)) {
 		ki := config.KernelInfo{
 			DistroType:    dii.DistroType,
@@ -372,7 +547,7 @@ func genKernels(dii dockerImageInfo, newkcfg *config.KernelConfig) (
 
 			KernelPath: kernelsBase + genKernelPath(files, k),
 			InitrdPath: kernelsBase + genInitrdPath(files, k),
-			RootFS:     genRootfsImage(dii),
+			RootFS:     rootfs,
 		}
 		newkcfg.Kernels = append(newkcfg.Kernels, ki)
 	}
@@ -389,7 +564,75 @@ func hasKernel(ki config.KernelInfo, kcfg config.KernelConfig) bool {
 	return false
 }
 
-func kernelAutogenHandler(workPath string) (err error) {
+func shuffle(a []string) []string {
+	// Fisher–Yates shuffle
+	for i := len(a) - 1; i > 0; i-- {
+		j := rand.Intn(i + 1)
+		a[i], a[j] = a[j], a[i]
+	}
+	return a
+}
+
+func generateKernels(km config.KernelMask, max int64, download bool) (err error) {
+	log.Println("Generating for kernel mask", km)
+
+	_, err = genRootfsImage(dockerImageInfo{ContainerName: km.DockerName()},
+		download)
+	if err != nil {
+		return
+	}
+
+	err = generateBaseDockerImage(km)
+	if err != nil {
+		return
+	}
+
+	var pkgs []string
+	switch km.DistroType {
+	case config.Ubuntu:
+		pkgs, err = matchDebianHeadersPkg(km.DockerName(),
+			km.ReleaseMask, true)
+	case config.CentOS:
+		pkgs, err = matchCentOSDevelPkg(km.DockerName(),
+			km.ReleaseMask, true)
+	default:
+		err = fmt.Errorf("%s not yet supported", km.DistroType.String())
+	}
+	if err != nil {
+		return
+	}
+
+	for i, pkg := range shuffle(pkgs) {
+		if max <= 0 {
+			log.Println("Max is reached")
+			break
+		}
+
+		log.Println(i, "/", len(pkgs), pkg)
+
+		err = dockerImageAppend(km, pkg)
+		if err == nil {
+			max--
+		} else {
+			log.Println("dockerImageAppend", err)
+		}
+	}
+
+	err = kickImage(km.DockerName())
+	if err != nil {
+		log.Println("kick image", km.DockerName(), ":", err)
+		return
+	}
+
+	err = copyKernels(km.DockerName())
+	if err != nil {
+		log.Println("copy kernels", km.DockerName(), ":", err)
+		return
+	}
+	return
+}
+
+func kernelAutogenHandler(workPath string, max int64, host, download bool) (err error) {
 	ka, err := config.ReadArtifactConfig(workPath + "/.out-of-tree.toml")
 	if err != nil {
 		return
@@ -401,40 +644,17 @@ func kernelAutogenHandler(workPath string) (err error) {
 			return
 		}
 
-		err = generateBaseDockerImage(sk)
+		err = generateKernels(sk, max, download)
 		if err != nil {
 			return
 		}
+	}
 
-		var pkgs []string
+	err = updateKernelsCfg(host, download)
-		pkgs, err = matchDebianKernelPkg(sk.DockerName(),
-			sk.ReleaseMask, true)
-		if err != nil {
 	return
 }
 
-		for _, pkg := range pkgs {
+func kernelDockerRegenHandler(host, download bool) (err error) {
-			dockerImageAppend(sk, pkg)
-		}
-
-		err = kickImage(sk.DockerName())
-		if err != nil {
-			log.Println("kick image", sk.DockerName(), ":", err)
-			continue
-		}
-
-		err = copyKernels(sk.DockerName())
-		if err != nil {
-			log.Println("copy kernels", sk.DockerName(), ":", err)
-			continue
-		}
-	}
-
-	err = updateKernelsCfg()
-	return
-}
-
-func kernelDockerRegenHandler() (err error) {
 	dockerImages, err := listDockerImages()
 	if err != nil {
 		return
@@ -472,5 +692,24 @@ func kernelDockerRegenHandler() (err error) {
 		}
 	}
 
-	return updateKernelsCfg()
+	return updateKernelsCfg(host, download)
+}
+
+func kernelGenallHandler(distro, version string, host, download bool) (err error) {
+	distroType, err := config.NewDistroType(distro)
+	if err != nil {
+		return
+	}
+
+	km := config.KernelMask{
+		DistroType:    distroType,
+		DistroRelease: version,
+		ReleaseMask:   ".*",
+	}
+	err = generateKernels(km, kernelsAll, download)
+	if err != nil {
+		return
+	}
+
+	return updateKernelsCfg(host, download)
 }

log.go

@@ -0,0 +1,245 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"database/sql"
+	"encoding/json"
+	"fmt"
+	"log"
+	"math"
+	"os"
+
+	"github.com/olekukonko/tablewriter"
+	"gopkg.in/logrusorgru/aurora.v1"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
+)
+
+func logLogEntry(l logEntry) {
+	distroInfo := fmt.Sprintf("%s-%s {%s}", l.DistroType,
+		l.DistroRelease, l.KernelRelease)
+
+	artifactInfo := fmt.Sprintf("{[%s] %s}", l.Type, l.Name)
+
+	colored := ""
+	if l.Type == config.KernelExploit {
+		colored = aurora.Sprintf("[%4d %4s] [%s] %40s %40s: %s %s",
+			l.ID, l.Tag, l.Timestamp, artifactInfo, distroInfo,
+			genOkFail("BUILD", l.Build.Ok),
+			genOkFail("LPE", l.Test.Ok))
+	} else {
+		colored = aurora.Sprintf("[%4d %4s] [%s] %40s %40s: %s %s %s",
+			l.ID, l.Tag, l.Timestamp, artifactInfo, distroInfo,
+			genOkFail("BUILD", l.Build.Ok),
+			genOkFail("INSMOD", l.Run.Ok),
+			genOkFail("TEST", l.Test.Ok))
+	}
+
+	additional := ""
+	if l.KernelPanic {
+		additional = "(panic)"
+	} else if l.KilledByTimeout {
+		additional = "(timeout)"
+	}
+
+	if additional != "" {
+		fmt.Println(colored, additional)
+	} else {
+		fmt.Println(colored)
+	}
+}
+
+func logHandler(db *sql.DB, path, tag string, num int, rate bool) (err error) {
+	var les []logEntry
+
+	ka, kaErr := config.ReadArtifactConfig(path + "/.out-of-tree.toml")
+	if kaErr == nil {
+		log.Println(".out-of-tree.toml found, filter by artifact name")
+		les, err = getAllArtifactLogs(db, tag, num, ka)
+	} else {
+		les, err = getAllLogs(db, tag, num)
+	}
+	if err != nil {
+		return
+	}
+
+	s := "\nS"
+	if rate {
+		if kaErr != nil {
+			err = kaErr
+			return
+		}
+
+		s = fmt.Sprintf("{[%s] %s} Overall s", ka.Type, ka.Name)
+
+		les, err = getAllArtifactLogs(db, tag, math.MaxInt64, ka)
+		if err != nil {
+			return
+		}
+	} else {
+		for _, l := range les {
+			logLogEntry(l)
+		}
+	}
+
+	success := 0
+	for _, l := range les {
+		if l.Test.Ok {
+			success++
+		}
+	}
+
+	overall := float64(success) / float64(len(les))
+	fmt.Printf("%success rate: %.04f (~%.0f%%)\n",
+		s, overall, overall*100)
+
+	return
+}
+
+func logDumpHandler(db *sql.DB, id int) (err error) {
+	l, err := getLogByID(db, id)
+	if err != nil {
+		return
+	}
+
+	fmt.Println("ID:", l.ID)
+	fmt.Println("Date:", l.Timestamp)
+	fmt.Println("Tag:", l.Tag)
+	fmt.Println()
+
+	fmt.Println("Type:", l.Type.String())
+	fmt.Println("Name:", l.Name)
+	fmt.Println()
+
+	fmt.Println("Distro:", l.DistroType.String(), l.DistroRelease)
+	fmt.Println("Kernel:", l.KernelRelease)
+	fmt.Println()
+
+	fmt.Println("Build ok:", l.Build.Ok)
+	if l.Type == config.KernelModule {
+		fmt.Println("Insmod ok:", l.Run.Ok)
+	}
+	fmt.Println("Test ok:", l.Test.Ok)
+	fmt.Println()
+
+	fmt.Printf("Build output:\n%s\n", l.Build.Output)
+	fmt.Println()
+
+	if l.Type == config.KernelModule {
+		fmt.Printf("Insmod output:\n%s\n", l.Run.Output)
+		fmt.Println()
+	}
+
+	fmt.Printf("Test output:\n%s\n", l.Test.Output)
+	fmt.Println()
+
+	fmt.Printf("Qemu stdout:\n%s\n", l.Stdout)
+	fmt.Println()
+
+	fmt.Printf("Qemu stderr:\n%s\n", l.Stderr)
+	fmt.Println()
+
+	return
+}
+
+type runstat struct {
+	All, BuildOK, RunOK, TestOK, Timeout, Panic int
+}
+
+func getStats(db *sql.DB, path, tag string) (
+	distros map[string]map[string]map[string]runstat, err error) {
+
+	var les []logEntry
+
+	ka, kaErr := config.ReadArtifactConfig(path + "/.out-of-tree.toml")
+	if kaErr == nil {
+		les, err = getAllArtifactLogs(db, tag, -1, ka)
+	} else {
+		les, err = getAllLogs(db, tag, -1)
+	}
+	if err != nil {
+		return
+	}
+
+	distros = make(map[string]map[string]map[string]runstat)
+
+	for _, l := range les {
+		_, ok := distros[l.DistroType.String()]
+		if !ok {
+			distros[l.DistroType.String()] = make(map[string]map[string]runstat)
+		}
+
+		_, ok = distros[l.DistroType.String()][l.DistroRelease]
+		if !ok {
+			distros[l.DistroType.String()][l.DistroRelease] = make(map[string]runstat)
+		}
+
+		rs := distros[l.DistroType.String()][l.DistroRelease][l.KernelRelease]
+
+		rs.All++
+		if l.Build.Ok {
+			rs.BuildOK++
+		}
+		if l.Run.Ok {
+			rs.RunOK++
+		}
+		if l.Test.Ok {
+			rs.TestOK++
+		}
+		if l.KernelPanic {
+			rs.Panic++
+		}
+		if l.KilledByTimeout {
+			rs.Timeout++
+		}
+
+		distros[l.DistroType.String()][l.DistroRelease][l.KernelRelease] = rs
+	}
+
+	return
+}
+
+func logJSONHandler(db *sql.DB, path, tag string) (err error) {
+	distros, err := getStats(db, path, tag)
+	if err != nil {
+		return
+	}
+
+	bytes, err := json.Marshal(&distros)
+	if err != nil {
+		return
+	}
+
+	fmt.Println(string(bytes))
+	return
+}
+
+func logMarkdownHandler(db *sql.DB, path, tag string) (err error) {
+	distros, err := getStats(db, path, tag)
+	if err != nil {
+		return
+	}
+
+	table := tablewriter.NewWriter(os.Stdout)
+	table.SetHeader([]string{"Distro", "Release", "Kernel", "Reliability"})
+	table.SetBorders(tablewriter.Border{
+		Left: true, Top: false, Right: true, Bottom: false})
+	table.SetCenterSeparator("|")
+
+	for distro, releases := range distros {
+		for release, kernels := range releases {
+			for kernel, stats := range kernels {
+				all := float64(stats.All)
+				ok := float64(stats.TestOK)
+				r := fmt.Sprintf("%6.02f%%", (ok/all)*100)
+				table.Append([]string{distro, release, kernel, r})
+			}
+		}
+	}
+
+	table.Render()
+	return
+}

main.go

@@ -5,24 +5,86 @@
 package main
 
 import (
+	"fmt"
 	"log"
+	"math/rand"
 	"os"
 	"os/exec"
 	"os/user"
+	"runtime"
+	"sort"
+	"strconv"
+	"time"
 
 	kingpin "gopkg.in/alecthomas/kingpin.v2"
 
-	"github.com/jollheef/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/config"
 )
 
+func findFallback(kcfg config.KernelConfig, ki config.KernelInfo) (rootfs string) {
+	for _, k := range kcfg.Kernels {
+		if !exists(k.RootFS) || k.DistroType != ki.DistroType {
+			continue
+		}
+		if k.RootFS < ki.RootFS {
+			rootfs = k.RootFS
+			return
+		}
+	}
+	return
+}
+
+func handleFallbacks(kcfg config.KernelConfig) {
+	sort.Sort(sort.Reverse(config.ByRootFS(kcfg.Kernels)))
+
+	for i, k := range kcfg.Kernels {
+		if !exists(k.RootFS) {
+			newRootFS := findFallback(kcfg, k)
+
+			s := k.RootFS + " does not exists "
+			if newRootFS != "" {
+				s += "(fallback to " + newRootFS + ")"
+			} else {
+				s += "(no fallback found)"
+			}
+
+			kcfg.Kernels[i].RootFS = newRootFS
+			log.Println(s)
+		}
+	}
+}
+
+func checkRequiredUtils() (err error) {
+	// Check for required commands
+	for _, cmd := range []string{"docker", "qemu-system-x86_64"} {
+		_, err := exec.Command("which", cmd).CombinedOutput()
+		if err != nil {
+			return fmt.Errorf("Command not found: %s", cmd)
+		}
+	}
+	return
+}
+
+func checkDockerPermissions() (err error) {
+	output, err := exec.Command("docker", "ps").CombinedOutput()
+	if err != nil {
+		err = fmt.Errorf("%s", output)
+	}
+	return
+}
+
 func main() {
+	log.SetFlags(log.Lshortfile)
+
+	rand.Seed(time.Now().UnixNano())
+
 	app := kingpin.New(
 		"out-of-tree",
 		"kernel {module, exploit} development tool",
 	)
 
-	app.Author("Mikhail Klementev <jollheef@riseup.net>")
+	app.Author("Mikhail Klementev <root@dumpstack.io>")
-	app.Version("0.1.0")
+	app.Version("0.2.0")
 
 	pathFlag := app.Flag("path", "Path to work directory")
 	path := pathFlag.Default(".").ExistingDir()
@@ -31,10 +93,16 @@ func main() {
 	if err != nil {
 		return
 	}
+	os.MkdirAll(usr.HomeDir+"/.out-of-tree", os.ModePerm)
+
 	defaultKcfgPath := usr.HomeDir + "/.out-of-tree/kernels.toml"
 
 	kcfgPathFlag := app.Flag("kernels", "Path to main kernels config")
-	kcfgPath := kcfgPathFlag.Default(defaultKcfgPath).ExistingFile()
+	kcfgPath := kcfgPathFlag.Default(defaultKcfgPath).String()
+
+	defaultDbPath := usr.HomeDir + "/.out-of-tree/db.sqlite"
+	dbPathFlag := app.Flag("db", "Path to database")
+	dbPath := dbPathFlag.Default(defaultDbPath).String()
 
 	defaultUserKcfgPath := usr.HomeDir + "/.out-of-tree/kernels.user.toml"
 	userKcfgPathFlag := app.Flag("user-kernels", "User kernels config")
@@ -46,7 +114,15 @@ func main() {
 
 	dockerTimeoutFlag := app.Flag("docker-timeout", "Timeout for docker")
 	dockerTimeout := dockerTimeoutFlag.Default("1m").Duration()
+
 	pewCommand := app.Command("pew", "Build, run and test module/exploit")
+
+	pewMax := pewCommand.Flag("max", "Test no more than X kernels").
+		PlaceHolder("X").Default(fmt.Sprint(kernelsAll)).Int64()
+
+	pewRuns := pewCommand.Flag("runs", "Runs per each kernel").
+		Default("1").Int64()
+
 	pewKernelFlag := pewCommand.Flag("kernel", "Override kernel regex")
 	pewKernel := pewKernelFlag.String()
 
@@ -59,12 +135,37 @@ func main() {
 	pewTestFlag := pewCommand.Flag("test", "Override path test")
 	pewTest := pewTestFlag.String()
 
+	pewDistFlag := pewCommand.Flag("dist", "Build result path")
+	pewDist := pewDistFlag.Default(pathDevNull).String()
+
+	pewThreadsFlag := pewCommand.Flag("threads", "Build result path")
+	pewThreads := pewThreadsFlag.Default(strconv.Itoa(runtime.NumCPU())).Int()
+
+	pewTagFlag := pewCommand.Flag("tag", "Log tagging")
+	pewTag := pewTagFlag.String()
+
 	kernelCommand := app.Command("kernel", "Manipulate kernels")
+	kernelNoDownload := kernelCommand.Flag("no-download",
+		"Do not download qemu image while kernel generation").Bool()
+	kernelUseHost := kernelCommand.Flag("host", "Use also host kernels").Bool()
 	kernelListCommand := kernelCommand.Command("list", "List kernels")
 	kernelAutogenCommand := kernelCommand.Command("autogen",
 		"Generate kernels based on a current config")
+	kernelAutogenMax := kernelAutogenCommand.Flag("max",
+		"Download random kernels from set defined by regex in "+
+			"release_mask, but no more than X for each of "+
+			"release_mask").PlaceHolder("X").Default(
+		fmt.Sprint(kernelsAll)).Int64()
 	kernelDockerRegenCommand := kernelCommand.Command("docker-regen",
 		"Regenerate kernels config from out_of_tree_* docker images")
+	kernelGenallCommand := kernelCommand.Command("genall",
+		"Generate all kernels for distro")
+
+	genallDistroFlag := kernelGenallCommand.Flag("distro", "Distributive")
+	distro := genallDistroFlag.Required().String()
+
+	genallVerFlag := kernelGenallCommand.Flag("ver", "Distro version")
+	version := genallVerFlag.Required().String()
 
 	genCommand := app.Command("gen", "Generate .out-of-tree.toml skeleton")
 	genModuleCommand := genCommand.Command("module",
@@ -78,22 +179,83 @@ func main() {
 	debugFlagGDB := debugCommand.Flag("gdb", "Set gdb listen address")
 	debugGDB := debugFlagGDB.Default("tcp::1234").String()
 
-	bootstrapCommand := app.Command("bootstrap",
+	yekaslr := debugCommand.Flag("enable-kaslr", "Enable KASLR").Bool()
-		"Create directories && download images")
+	yesmep := debugCommand.Flag("enable-smep", "Enable SMEP").Bool()
+	yesmap := debugCommand.Flag("enable-smap", "Enable SMAP").Bool()
+	yekpti := debugCommand.Flag("enable-kpti", "Enable KPTI").Bool()
 
-	// Check for required commands
+	nokaslr := debugCommand.Flag("disable-kaslr", "Disable KASLR").Bool()
-	for _, cmd := range []string{"timeout", "docker", "qemu"} {
+	nosmep := debugCommand.Flag("disable-smep", "Disable SMEP").Bool()
-		_, err := exec.Command("which", cmd).CombinedOutput()
+	nosmap := debugCommand.Flag("disable-smap", "Disable SMAP").Bool()
+	nokpti := debugCommand.Flag("disable-kpti", "Disable KPTI").Bool()
+
+	bootstrapCommand := app.Command("bootstrap", "Apparently nothing")
+
+	logCommand := app.Command("log", "Logs")
+
+	logQueryCommand := logCommand.Command("query", "Query logs")
+	logNum := logQueryCommand.Flag("num", "How much lines").Default("50").Int()
+	logRate := logQueryCommand.Flag("rate", "Show artifact success rate").Bool()
+	logTag := logQueryCommand.Flag("tag", "Filter tag").String()
+
+	logDumpCommand := logCommand.Command("dump",
+		"Show all info for log entry with ID")
+	logDumpID := logDumpCommand.Arg("ID", "").Required().Int()
+
+	logJSONCommand := logCommand.Command("json", "Generate json statistics")
+	logJSONTag := logJSONCommand.Flag("tag", "Filter tag").Required().String()
+
+	logMarkdownCommand := logCommand.Command("markdown", "Generate markdown statistics")
+	logMarkdownTag := logMarkdownCommand.Flag("tag", "Filter tag").Required().String()
+
+	packCommand := app.Command("pack", "Exploit pack test")
+	packAutogen := packCommand.Flag("autogen", "Kernel autogeneration").Bool()
+	packNoDownload := packCommand.Flag("no-download",
+		"Do not download qemu image while kernel generation").Bool()
+	packExploitRuns := packCommand.Flag("exploit-runs",
+		"Amount of runs of each exploit").Default("4").Int64()
+	packKernelRuns := packCommand.Flag("kernel-runs",
+		"Amount of runs of each kernel").Default("1").Int64()
+
+	err = checkRequiredUtils()
 	if err != nil {
-			log.Fatalln("Command not found:", cmd)
+		log.Fatalln(err)
 	}
+
+	err = checkDockerPermissions()
+	if err != nil {
+		log.Println(err)
+		log.Println("You have two options:")
+		log.Println("\t1. Add user to group docker;")
+		log.Println("\t2. Run out-of-tree with sudo.")
+		os.Exit(1)
+	}
+
+	if !exists(usr.HomeDir + "/.out-of-tree/kernels.toml") {
+		log.Println("No ~/.out-of-tree/kernels.toml: Probably you " +
+			"need to run `out-of-tree kernel autogen` in " +
+			"directory that contains .out-of-tree.toml " +
+			"with defined kernel masks " +
+			"(see docs at https://out-of-tree.io)")
 	}
 
 	kingpin.MustParse(app.Parse(os.Args[1:]))
 
+	if *yekaslr && *nokaslr {
+		log.Fatalln("Only one of disable/enable can be used at once")
+	}
+
+	if *yesmep && *nosmep {
+		log.Fatalln("Only one of disable/enable can be used at once")
+	}
+
+	if *yesmap && *nosmap {
+		log.Fatalln("Only one of disable/enable can be used at once")
+	}
+
 	kcfg, err := config.ReadKernelConfig(*kcfgPath)
 	if err != nil {
-		log.Fatalln(err)
+		log.Println(err)
 	}
 
 	if exists(*userKcfgPath) {
@@ -109,28 +271,60 @@ func main() {
 		}
 	}
 
+	handleFallbacks(kcfg)
+
+	db, err := openDatabase(*dbPath)
+	if err != nil {
+		log.Fatalln(err)
+	}
+	defer db.Close()
+
 	switch kingpin.MustParse(app.Parse(os.Args[1:])) {
 	case pewCommand.FullCommand():
 		err = pewHandler(kcfg, *path, *pewKernel, *pewBinary,
-			*pewTest, *pewGuess, *qemuTimeout, *dockerTimeout)
+			*pewTest, *pewGuess, *qemuTimeout, *dockerTimeout,
+			*pewMax, *pewRuns, *pewDist, *pewTag, *pewThreads, db)
 	case kernelListCommand.FullCommand():
 		err = kernelListHandler(kcfg)
 	case kernelAutogenCommand.FullCommand():
-		err = kernelAutogenHandler(*path)
+		err = kernelAutogenHandler(*path, *kernelAutogenMax,
+			*kernelUseHost, !*kernelNoDownload)
 	case kernelDockerRegenCommand.FullCommand():
-		err = kernelDockerRegenHandler()
+		err = kernelDockerRegenHandler(*kernelUseHost, !*kernelNoDownload)
+	case kernelGenallCommand.FullCommand():
+		err = kernelGenallHandler(*distro, *version,
+			*kernelUseHost, !*kernelNoDownload)
 	case genModuleCommand.FullCommand():
 		err = genConfig(config.KernelModule)
 	case genExploitCommand.FullCommand():
 		err = genConfig(config.KernelExploit)
 	case debugCommand.FullCommand():
 		err = debugHandler(kcfg, *path, *debugKernel, *debugGDB,
-			*dockerTimeout)
+			*dockerTimeout, *yekaslr, *yesmep, *yesmap, *yekpti,
+			*nokaslr, *nosmep, *nosmap, *nokpti)
 	case bootstrapCommand.FullCommand():
-		err = bootstrapHandler()
+		fmt.Println("bootstrap is no more required, " +
+			"now images downloading on-demand")
+		fmt.Println("please, remove it from any automation scripts, " +
+			"because it'll be removed in the next release")
+	case logQueryCommand.FullCommand():
+		err = logHandler(db, *path, *logTag, *logNum, *logRate)
+	case logDumpCommand.FullCommand():
+		err = logDumpHandler(db, *logDumpID)
+	case logJSONCommand.FullCommand():
+		err = logJSONHandler(db, *path, *logJSONTag)
+	case logMarkdownCommand.FullCommand():
+		err = logMarkdownHandler(db, *path, *logMarkdownTag)
+	case packCommand.FullCommand():
+		err = packHandler(db, *path, kcfg, *packAutogen,
+			!*packNoDownload, *packExploitRuns, *packKernelRuns)
 	}
 
 	if err != nil {
 		log.Fatalln(err)
 	}
+
+	if somethingFailed {
+		os.Exit(1)
+	}
 }

pack.go

@@ -0,0 +1,56 @@
+// Copyright 2019 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"runtime"
+	"time"
+
+	"code.dumpstack.io/tools/out-of-tree/config"
+)
+
+func packHandler(db *sql.DB, path string, kcfg config.KernelConfig,
+	autogen, download bool, exploitRuns, kernelRuns int64) (err error) {
+
+	dockerTimeout := time.Minute
+	qemuTimeout := time.Minute
+	threads := runtime.NumCPU()
+
+	tag := fmt.Sprintf("pack_run_%d", time.Now().Unix())
+	log.Println("Tag:", tag)
+
+	files, err := ioutil.ReadDir(path)
+	if err != nil {
+		return
+	}
+
+	for _, f := range files {
+		workPath := path + "/" + f.Name()
+
+		if !exists(workPath + "/.out-of-tree.toml") {
+			continue
+		}
+
+		if autogen {
+			var perRegex int64 = 1
+			err = kernelAutogenHandler(workPath, perRegex, false, download)
+			if err != nil {
+				return
+			}
+		}
+
+		log.Println(f.Name())
+
+		pewHandler(kcfg, workPath, "", "", "", false,
+			dockerTimeout, qemuTimeout,
+			kernelRuns, exploitRuns, pathDevNull, tag, threads, db)
+	}
+
+	return
+}

pew.go

@@ -5,29 +5,52 @@
 package main
 
 import (
+	"database/sql"
 	"errors"
 	"fmt"
+	"io"
 	"io/ioutil"
 	"log"
 	"math/rand"
 	"os"
 	"os/exec"
-	"runtime"
+	"os/user"
 	"strings"
 	"time"
 
-	"github.com/logrusorgru/aurora"
 	"github.com/otiai10/copy"
 	"github.com/remeh/sizedwaitgroup"
+	"gopkg.in/logrusorgru/aurora.v1"
 
-	"github.com/jollheef/out-of-tree/config"
+	"code.dumpstack.io/tools/out-of-tree/config"
-	qemu "github.com/jollheef/out-of-tree/qemu"
+	"code.dumpstack.io/tools/out-of-tree/qemu"
 )
 
-func dockerCommand(container, workdir, timeout, command string) *exec.Cmd {
+var somethingFailed = false
-	return exec.Command("timeout", "-k", timeout, timeout, "docker", "run",
+
-		"-v", workdir+":/work", container,
+const pathDevNull = "/dev/null"
-		"bash", "-c", "cd /work && "+command)
+
+func dockerRun(timeout time.Duration, container, workdir, command string) (
+	output string, err error) {
+
+	cmd := exec.Command("docker", "run", "-v", workdir+":/work",
+		container, "bash", "-c", "cd /work && "+command)
+
+	timer := time.AfterFunc(timeout, func() {
+		cmd.Process.Kill()
+	})
+	defer timer.Stop()
+
+	raw, err := cmd.CombinedOutput()
+	if err != nil {
+		e := fmt.Sprintf("error `%v` for cmd `%v` with output `%v`",
+			err, command, string(raw))
+		err = errors.New(e)
+		return
+	}
+
+	output = string(raw)
+	return
 }
 
 func build(tmp string, ka config.Artifact, ki config.KernelInfo,
@@ -48,38 +71,37 @@ func build(tmp string, ka config.Artifact, ki config.KernelInfo,
 	}
 
 	kernel := "/lib/modules/" + ki.KernelRelease + "/build"
+	if ki.KernelSource != "" {
+		kernel = ki.KernelSource
+	}
 
-	seconds := fmt.Sprintf("%ds", dockerTimeout/time.Second)
+	if ki.ContainerName != "" {
-	cmd := dockerCommand(ki.ContainerName, tmpSourcePath, seconds,
+		output, err = dockerRun(dockerTimeout, ki.ContainerName,
-		"make KERNEL="+kernel+" TARGET="+target)
+			tmpSourcePath, "make KERNEL="+kernel+" TARGET="+target+
-	rawOutput, err := cmd.CombinedOutput()
+				" && chmod -R 777 /work")
-	output = string(rawOutput)
+	} else {
+		command := "make KERNEL=" + kernel + " TARGET=" + target
+		cmd := exec.Command("bash", "-c", "cd "+tmpSourcePath+" && "+command)
+		timer := time.AfterFunc(dockerTimeout, func() {
+			cmd.Process.Kill()
+		})
+		defer timer.Stop()
+
+		var raw []byte
+		raw, err = cmd.CombinedOutput()
 		if err != nil {
-		err = errors.New("make execution error")
+			e := fmt.Sprintf("error `%v` for cmd `%v` with output `%v`",
+				err, command, string(raw))
+			err = errors.New(e)
 			return
 		}
 
-	return
+		output = string(raw)
-}
-
-func cleanDmesg(q *qemu.QemuSystem) (err error) {
-	start := time.Now()
-	for {
-		_, err = q.Command("root", "dmesg -c")
-		if err == nil {
-			break
-		}
-		time.Sleep(time.Second)
-
-		if time.Now().After(start.Add(time.Minute)) {
-			err = errors.New("Can't connect to qemu")
-			break
-		}
 	}
 	return
 }
 
-func testKernelModule(q *qemu.QemuSystem, ka config.Artifact,
+func testKernelModule(q *qemu.System, ka config.Artifact,
 	test string) (output string, err error) {
 
 	output, err = q.Command("root", test)
@@ -87,7 +109,7 @@ func testKernelModule(q *qemu.QemuSystem, ka config.Artifact,
 	return
 }
 
-func testKernelExploit(q *qemu.QemuSystem, ka config.Artifact,
+func testKernelExploit(q *qemu.System, ka config.Artifact,
 	test, exploit string) (output string, err error) {
 
 	output, err = q.Command("user", "chmod +x "+exploit)
@@ -111,18 +133,48 @@ func testKernelExploit(q *qemu.QemuSystem, ka config.Artifact,
 	return
 }
 
-func genOkFail(name string, ok bool) aurora.Value {
+func genOkFail(name string, ok bool) (aurv aurora.Value) {
 	if ok {
 		s := " " + name + " SUCCESS "
-		return aurora.BgGreen(aurora.Black(s))
+		aurv = aurora.BgGreen(aurora.Black(s))
 	} else {
+		somethingFailed = true
 		s := " " + name + " FAILURE "
-		return aurora.BgRed(aurora.Gray(aurora.Bold(s)))
+		aurv = aurora.BgRed(aurora.Gray(aurora.Bold(s)))
+	}
+	return
+}
+
+type phasesResult struct {
+	BuildArtifact    string
+	Build, Run, Test struct {
+		Output string
+		Ok     bool
 	}
 }
 
-func dumpResult(q *qemu.QemuSystem, ka config.Artifact, ki config.KernelInfo,
+func copyFile(sourcePath, destinationPath string) (err error) {
-	build_ok, run_ok, test_ok *bool) {
+	sourceFile, err := os.Open(sourcePath)
+	if err != nil {
+		return
+	}
+	defer sourceFile.Close()
+
+	destinationFile, err := os.Create(destinationPath)
+	if err != nil {
+		return err
+	}
+	if _, err := io.Copy(destinationFile, sourceFile); err != nil {
+		destinationFile.Close()
+		return err
+	}
+	return destinationFile.Close()
+}
+
+func dumpResult(q *qemu.System, ka config.Artifact, ki config.KernelInfo,
+	res *phasesResult, dist, tag, binary string, db *sql.DB) {
+
+	// TODO merge (problem is it's not 100% same) with log.go:logLogEntry
 
 	distroInfo := fmt.Sprintf("%s-%s {%s}", ki.DistroType,
 		ki.DistroRelease, ki.KernelRelease)
@@ -130,13 +182,13 @@ func dumpResult(q *qemu.QemuSystem, ka config.Artifact, ki config.KernelInfo,
 	colored := ""
 	if ka.Type == config.KernelExploit {
 		colored = aurora.Sprintf("[*] %40s: %s %s", distroInfo,
-			genOkFail("BUILD", *build_ok),
+			genOkFail("BUILD", res.Build.Ok),
-			genOkFail("LPE", *test_ok))
+			genOkFail("LPE", res.Test.Ok))
 	} else {
 		colored = aurora.Sprintf("[*] %40s: %s %s %s", distroInfo,
-			genOkFail("BUILD", *build_ok),
+			genOkFail("BUILD", res.Build.Ok),
-			genOkFail("INSMOD", *run_ok),
+			genOkFail("INSMOD", res.Run.Ok),
-			genOkFail("TEST", *test_ok))
+			genOkFail("TEST", res.Test.Ok))
 	}
 
 	additional := ""
@@ -151,22 +203,121 @@ func dumpResult(q *qemu.QemuSystem, ka config.Artifact, ki config.KernelInfo,
 	} else {
 		fmt.Println(colored)
 	}
+
+	err := addToLog(db, q, ka, ki, res, tag)
+	if err != nil {
+		log.Println("[db] addToLog (", ka, ") error:", err)
+	}
+
+	if binary == "" && dist != pathDevNull {
+		err = os.MkdirAll(dist, os.ModePerm)
+		if err != nil {
+			log.Println("os.MkdirAll (", ka, ") error:", err)
+		}
+
+		path := fmt.Sprintf("%s/%s-%s-%s", dist, ki.DistroType,
+			ki.DistroRelease, ki.KernelRelease)
+		if ka.Type != config.KernelExploit {
+			path += ".ko"
+		}
+
+		err = copyFile(res.BuildArtifact, path)
+		if err != nil {
+			log.Println("copyFile (", ka, ") error:", err)
+		}
+	}
+}
+
+func copyArtifactAndTest(q *qemu.System, ka config.Artifact,
+	res *phasesResult, remoteTest string) (err error) {
+
+	switch ka.Type {
+	case config.KernelModule:
+		res.Run.Output, err = q.CopyAndInsmod(res.BuildArtifact)
+		if err != nil {
+			log.Println(res.Run.Output, err)
+			return
+		}
+		res.Run.Ok = true
+
+		res.Test.Output, err = testKernelModule(q, ka, remoteTest)
+		if err != nil {
+			log.Println(res.Test.Output, err)
+			return
+		}
+		res.Test.Ok = true
+	case config.KernelExploit:
+		remoteExploit := fmt.Sprintf("/tmp/exploit_%d", rand.Int())
+		err = q.CopyFile("user", res.BuildArtifact, remoteExploit)
+		if err != nil {
+			return
+		}
+
+		res.Test.Output, err = testKernelExploit(q, ka, remoteTest,
+			remoteExploit)
+		if err != nil {
+			log.Println(res.Test.Output)
+			return
+		}
+		res.Run.Ok = true // does not really used
+		res.Test.Ok = true
+	default:
+		log.Println("Unsupported artifact type")
+	}
+
+	return
+}
+
+func copyTest(q *qemu.System, testPath string, ka config.Artifact) (
+	remoteTest string, err error) {
+
+	remoteTest = fmt.Sprintf("/tmp/test_%d", rand.Int())
+	err = q.CopyFile("user", testPath, remoteTest)
+	if err != nil {
+		if ka.Type == config.KernelExploit {
+			q.Command("user",
+				"echo -e '#!/bin/sh\necho touch $2 | $1' "+
+					"> "+remoteTest+
+					" && chmod +x "+remoteTest)
+		} else {
+			q.Command("user", "echo '#!/bin/sh' "+
+				"> "+remoteTest+" && chmod +x "+remoteTest)
+		}
+	}
+
+	_, err = q.Command("root", "chmod +x "+remoteTest)
+	return
 }
 
 func whatever(swg *sizedwaitgroup.SizedWaitGroup, ka config.Artifact,
 	ki config.KernelInfo, binaryPath, testPath string,
-	qemuTimeout, dockerTimeout time.Duration) {
+	qemuTimeout, dockerTimeout time.Duration, dist, tag string,
+	db *sql.DB) {
 
 	defer swg.Done()
 
 	kernel := qemu.Kernel{KernelPath: ki.KernelPath, InitrdPath: ki.InitrdPath}
-	q, err := qemu.NewQemuSystem(qemu.X86_64, kernel, ki.RootFS)
+	q, err := qemu.NewSystem(qemu.X86x64, kernel, ki.RootFS)
 	if err != nil {
 		log.Println("Qemu creation error:", err)
 		return
 	}
 	q.Timeout = qemuTimeout
 
+	if ka.Qemu.Timeout.Duration != 0 {
+		q.Timeout = ka.Qemu.Timeout.Duration
+	}
+	if ka.Qemu.Cpus != 0 {
+		q.Cpus = ka.Qemu.Cpus
+	}
+	if ka.Qemu.Memory != 0 {
+		q.Memory = ka.Qemu.Memory
+	}
+
+	q.SetKASLR(!ka.Mitigations.DisableKaslr)
+	q.SetSMEP(!ka.Mitigations.DisableSmep)
+	q.SetSMAP(!ka.Mitigations.DisableSmap)
+
 	err = q.Start()
 	if err != nil {
 		log.Println("Qemu start error:", err)
@@ -174,106 +325,73 @@ func whatever(swg *sizedwaitgroup.SizedWaitGroup, ka config.Artifact,
 	}
 	defer q.Stop()
 
-	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_")
+	usr, err := user.Current()
+	if err != nil {
+		return
+	}
+	tmpdir := usr.HomeDir + "/.out-of-tree/tmp"
+	os.MkdirAll(tmpdir, os.ModePerm)
+
+	tmp, err := ioutil.TempDir(tmpdir, "out-of-tree_")
 	if err != nil {
 		log.Println("Temporary directory creation error:", err)
 		return
 	}
 	defer os.RemoveAll(tmp)
 
-	build_ok := false
+	result := phasesResult{}
-	run_ok := false
+	defer dumpResult(q, ka, ki, &result, dist, tag, binaryPath, db)
-	test_ok := false
-	defer dumpResult(q, ka, ki, &build_ok, &run_ok, &test_ok)
 
-	var outFile, output string
 	if binaryPath == "" {
-		// TODO Write build log to file or database
+		result.BuildArtifact, result.Build.Output, err = build(tmp, ka,
-		outFile, output, err = build(tmp, ka, ki, dockerTimeout)
+			ki, dockerTimeout)
 		if err != nil {
-			log.Println(output)
+			log.Println(err)
 			return
 		}
-		build_ok = true
+		result.Build.Ok = true
 	} else {
-		outFile = binaryPath
+		result.BuildArtifact = binaryPath
-		build_ok = true
+		result.Build.Ok = true
-	}
-
-	err = cleanDmesg(q)
-	if err != nil {
-		return
 	}
 
 	if testPath == "" {
-		testPath = outFile + "_test"
+		testPath = result.BuildArtifact + "_test"
-	}
+		if !exists(testPath) {
-
+			testPath = tmp + "/" + "test.sh"
-	remoteTest := fmt.Sprintf("/tmp/test_%d", rand.Int())
-	err = q.CopyFile("user", testPath, remoteTest)
-	if err != nil {
-		if ka.Type == config.KernelExploit {
-			log.Println("Use `echo touch FILE | exploit` for test")
-			q.Command("user",
-				"echo -e '#!/bin/sh\necho touch $2 | $1' "+
-					"> "+remoteTest+
-					" && chmod +x "+remoteTest)
-		} else {
-			log.Println("copy file err", err)
-			// we should not exit because of testing 'insmod' part
-			// for kernel module
 		}
 	}
 
-	_, err = q.Command("root", "chmod +x "+remoteTest)
+	remoteTest, err := copyTest(q, testPath, ka)
 	if err != nil {
 		return
 	}
 
-	if ka.Type == config.KernelModule {
+	copyArtifactAndTest(q, ka, &result, remoteTest)
-		// TODO Write insmod log to file or database
-		output, err := q.CopyAndInsmod(outFile)
-		if err != nil {
-			log.Println(output, err)
-			return
-		}
-		run_ok = true
-
-		// TODO Write test results to file or database
-		output, err = testKernelModule(q, ka, remoteTest)
-		if err != nil {
-			log.Println(output, err)
-			return
-		}
-		test_ok = true
-	} else if ka.Type == config.KernelExploit {
-		remoteExploit := fmt.Sprintf("/tmp/exploit_%d", rand.Int())
-		err = q.CopyFile("user", outFile, remoteExploit)
-		if err != nil {
-			return
 }
 
-		// TODO Write test results to file or database
+func shuffleKernels(a []config.KernelInfo) []config.KernelInfo {
-		output, err = testKernelExploit(q, ka, remoteTest, remoteExploit)
+	// Fisher–Yates shuffle
-		if err != nil {
+	for i := len(a) - 1; i > 0; i-- {
-			log.Println(output)
+		j := rand.Intn(i + 1)
-			return
+		a[i], a[j] = a[j], a[i]
 	}
-		run_ok = true // does not really used
+	return a
-		test_ok = true
-	} else {
-		err = errors.New("Unsupported artifact type")
-	}
-	return
 }
 
 func performCI(ka config.Artifact, kcfg config.KernelConfig, binaryPath,
-	testPath string, qemuTimeout, dockerTimeout time.Duration) (err error) {
+	testPath string, qemuTimeout, dockerTimeout time.Duration,
+	max, runs int64, dist, tag string, threads int,
+	db *sql.DB) (err error) {
 
 	found := false
 
-	swg := sizedwaitgroup.New(runtime.NumCPU())
+	swg := sizedwaitgroup.New(threads)
-	for _, kernel := range kcfg.Kernels {
+	for _, kernel := range shuffleKernels(kcfg.Kernels) {
+		if max <= 0 {
+			break
+		}
+
 		var supported bool
 		supported, err = ka.Supported(kernel)
 		if err != nil {
@@ -282,9 +400,13 @@ func performCI(ka config.Artifact, kcfg config.KernelConfig, binaryPath,
 
 		if supported {
 			found = true
+			max--
+			for i := int64(0); i < runs; i++ {
 				swg.Add()
-			go whatever(&swg, ka, kernel, binaryPath, testPath,
+				go whatever(&swg, ka, kernel, binaryPath,
-				qemuTimeout, dockerTimeout)
+					testPath, qemuTimeout, dockerTimeout,
+					dist, tag, db)
+			}
 		}
 	}
 	swg.Wait()
@@ -319,9 +441,27 @@ func kernelMask(kernel string) (km config.KernelMask, err error) {
 	return
 }
 
+func genAllKernels() (sk []config.KernelMask, err error) {
+	for _, dType := range config.DistroTypeStrings {
+		var dt config.DistroType
+		dt, err = config.NewDistroType(dType)
+		if err != nil {
+			return
+		}
+
+		sk = append(sk, config.KernelMask{
+			DistroType:  dt,
+			ReleaseMask: ".*",
+		})
+	}
+	return
+}
+
 func pewHandler(kcfg config.KernelConfig,
 	workPath, ovrrdKrnl, binary, test string, guess bool,
-	qemuTimeout, dockerTimeout time.Duration) (err error) {
+	qemuTimeout, dockerTimeout time.Duration,
+	max, runs int64, dist, tag string, threads int,
+	db *sql.DB) (err error) {
 
 	ka, err := config.ReadArtifactConfig(workPath + "/.out-of-tree.toml")
 	if err != nil {
@@ -343,20 +483,14 @@ func pewHandler(kcfg config.KernelConfig,
 	}
 
 	if guess {
-		ka.SupportedKernels = []config.KernelMask{}
+		ka.SupportedKernels, err = genAllKernels()
-		for _, dType := range config.DistroTypeStrings {
-			var dt config.DistroType
-			dt, err = config.NewDistroType(dType)
 		if err != nil {
 			return
 		}
-
-			km := config.KernelMask{DistroType: dt, ReleaseMask: ".*"}
-			ka.SupportedKernels = append(ka.SupportedKernels, km)
-		}
 	}
 
-	err = performCI(ka, kcfg, binary, test, qemuTimeout, dockerTimeout)
+	err = performCI(ka, kcfg, binary, test, qemuTimeout, dockerTimeout,
+		max, runs, dist, tag, threads, db)
 	if err != nil {
 		return
 	}

pew_test.go

@@ -0,0 +1,44 @@
+// Copyright 2018 Mikhail Klementev. All rights reserved.
+// Use of this source code is governed by a AGPLv3 license
+// (or later) that can be found in the LICENSE file.
+
+package main
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestDockerRun(t *testing.T) {
+	tmp, err := ioutil.TempDir("/tmp/", "out-of-tree_test_")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(tmp)
+
+	start := time.Now()
+
+	timeout := time.Second
+
+	_, err = dockerRun(timeout, "ubuntu", tmp, "sleep 5s")
+	if err == nil {
+		t.Fatal("docker is not killed by timeout")
+	}
+	if time.Since(start) > 3*time.Second {
+		t.Fatal(fmt.Sprintf("timeout failed (%v instead of %v)",
+			time.Since(start), time.Second))
+	}
+
+	output, err := dockerRun(time.Minute, "ubuntu", tmp, "echo hello")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if !strings.Contains(output, "hello") {
+		t.Fatal("wrong output (" + output + ")")
+	}
+}

qemu/README.md

@@ -9,7 +9,7 @@ Features:
 
 ## Installation
 
-    $ go get github.com/jollheef/out-of-tree/qemu
+    $ go get code.dumpstack.io/tools/out-of-tree/qemu
 
 ### Generate root image
 
@@ -30,12 +30,12 @@ Note: qemu on macOS since v2.12 (24 April 2018) supports Hypervisor.framework.
 
 #### Generate image
 
-    $ cd $GOPATH/src/github.com/jollheef/out-of-tree/tools/qemu-debian-img
+    $ cd $GOPATH/src/code.dumpstack.io/tools/out-of-tree/tools/qemu-debian-img
     $ ./bootstrap.sh
 
 ### Fill configuration file
 
-    $ $EDITOR $GOPATH/src/github.com/jollheef/out-of-tree/qemu/test.config.go
+    $ $EDITOR $GOPATH/src/code.dumpstack.io/tools/out-of-tree/qemu/test.config.go
 
 ### Run tests
 
@@ -43,7 +43,7 @@ Note: qemu on macOS since v2.12 (24 April 2018) supports Hypervisor.framework.
 
 ## Usage
 
-    $ go get github.com/jollheef/out-of-tree/qemu
+    $ go get code.dumpstack.io/tools/out-of-tree/qemu
 
 Minimal example:
 
@@ -52,7 +52,7 @@ Minimal example:
 		KernelPath: "/path/to/vmlinuz",
 		InitrdPath: "/path/to/initrd", // if required
 	}
-	q, err := qemu.NewQemuSystem(qemu.X86_64, kernel, "/path/to/qcow2")
+	q, err := qemu.NewSystem(qemu.X86_64, kernel, "/path/to/qcow2")
 	if err != nil {
 		log.Fatal(err)
 	}
@@ -71,4 +71,4 @@ Minimal example:
 
 More information and list of all functions see at go documentation project, or just run locally:
 
-    $ godoc github.com/jollheef/out-of-tree/qemu
+    $ godoc code.dumpstack.io/tools/out-of-tree/qemu

qemu/qemu-kernel.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a AGPLv3 license
 // (or later) that can be found in the LICENSE file.
 
-package qemukernel
+package qemu
 
 import (
 	"bytes"
@@ -44,8 +44,10 @@ func readUntilEOF(pipe io.ReadCloser, buf *[]byte) (err error) {
 type arch string
 
 const (
-	X86_64 arch = "x86_64"
+	// X86x64 is the qemu-system-x86_64
-	I386        = "i386"
+	X86x64 arch = "x86_64"
+	// X86x32 is the qemu-system-i386
+	X86x32 = "i386"
 	// TODO add other
 
 	unsupported = "unsupported" // for test purposes
@@ -58,8 +60,8 @@ type Kernel struct {
 	InitrdPath string
 }
 
-// QemuSystem describe qemu parameters and runned process
+// System describe qemu parameters and runned process
-type QemuSystem struct {
+type System struct {
 	arch      arch
 	kernel    Kernel
 	drivePath string
@@ -70,6 +72,11 @@ type QemuSystem struct {
 	debug bool
 	gdb   string // tcp::1234
 
+	noKASLR bool
+	noSMEP  bool
+	noSMAP  bool
+	noKPTI  bool
+
 	// Timeout works after Start invocation
 	Timeout         time.Duration
 	KilledByTimeout bool
@@ -93,12 +100,12 @@ type QemuSystem struct {
 	exitErr error
 }
 
-// NewQemuSystem constructor
+// NewSystem constructor
-func NewQemuSystem(arch arch, kernel Kernel, drivePath string) (q *QemuSystem, err error) {
+func NewSystem(arch arch, kernel Kernel, drivePath string) (q *System, err error) {
 	if _, err = exec.LookPath("qemu-system-" + string(arch)); err != nil {
 		return
 	}
-	q = &QemuSystem{}
+	q = &System{}
 	q.arch = arch
 
 	if _, err = os.Stat(kernel.KernelPath); err != nil {
@@ -156,10 +163,19 @@ func kvmExists() bool {
 	if _, err := os.Stat("/dev/kvm"); err != nil {
 		return false
 	}
+
+	file, err := os.OpenFile("/dev/kvm", os.O_WRONLY, 0666)
+	if err != nil {
+		if os.IsPermission(err) {
+			return false
+		}
+	}
+	file.Close()
+
 	return true
 }
 
-func (q *QemuSystem) panicWatcher() {
+func (q *System) panicWatcher() {
 	for {
 		time.Sleep(time.Second)
 		if bytes.Contains(q.Stdout, []byte("Kernel panic")) {
@@ -172,15 +188,36 @@ func (q *QemuSystem) panicWatcher() {
 	}
 }
 
+func (q System) cmdline() (s string) {
+	s = "root=/dev/sda ignore_loglevel console=ttyS0 rw"
+
+	if q.noKASLR {
+		s += " nokaslr"
+	}
+
+	if q.noSMEP {
+		s += " nosmep"
+	}
+
+	if q.noSMAP {
+		s += " nosmap"
+	}
+
+	if q.noKPTI {
+		s += " nokpti"
+	}
+
+	return
+}
+
 // Start qemu process
-func (q *QemuSystem) Start() (err error) {
+func (q *System) Start() (err error) {
 	rand.Seed(time.Now().UnixNano()) // Are you sure?
 	q.sshAddrPort = getFreeAddrPort()
 	hostfwd := fmt.Sprintf("hostfwd=tcp:%s-:22", q.sshAddrPort)
 	qemuArgs := []string{"-snapshot", "-nographic",
 		"-hda", q.drivePath,
 		"-kernel", q.kernel.KernelPath,
-		"-append", "root=/dev/sda ignore_loglevel console=ttyS0 rw",
 		"-smp", fmt.Sprintf("%d", q.Cpus),
 		"-m", fmt.Sprintf("%d", q.Memory),
 		"-device", "e1000,netdev=n1",
@@ -195,14 +232,16 @@ func (q *QemuSystem) Start() (err error) {
 		qemuArgs = append(qemuArgs, "-initrd", q.kernel.InitrdPath)
 	}
 
-	if (q.arch == X86_64 || q.arch == I386) && kvmExists() {
+	if (q.arch == X86x64 || q.arch == X86x32) && kvmExists() {
-		qemuArgs = append(qemuArgs, "-enable-kvm")
+		qemuArgs = append(qemuArgs, "-enable-kvm", "-cpu", "host")
 	}
 
-	if q.arch == X86_64 && runtime.GOOS == "darwin" {
+	if q.arch == X86x64 && runtime.GOOS == "darwin" {
 		qemuArgs = append(qemuArgs, "-accel", "hvf", "-cpu", "host")
 	}
 
+	qemuArgs = append(qemuArgs, "-append", q.cmdline())
+
 	q.cmd = exec.Command("qemu-system-"+string(q.arch), qemuArgs...)
 
 	if q.pipe.stdin, err = q.cmd.StdinPipe(); err != nil {
@@ -250,7 +289,7 @@ func (q *QemuSystem) Start() (err error) {
 }
 
 // Stop qemu process
-func (q *QemuSystem) Stop() {
+func (q *System) Stop() {
 	// 1  00/01   01  01  SOH  (Ctrl-A)  START OF HEADING
 	fmt.Fprintf(q.pipe.stdin, "%cx", 1)
 	// wait for die
@@ -262,7 +301,7 @@ func (q *QemuSystem) Stop() {
 	}
 }
 
-func (q QemuSystem) ssh(user string) (client *ssh.Client, err error) {
+func (q System) ssh(user string) (client *ssh.Client, err error) {
 	cfg := &ssh.ClientConfig{
 		User:            user,
 		HostKeyCallback: ssh.InsecureIgnoreHostKey(),
@@ -273,7 +312,7 @@ func (q QemuSystem) ssh(user string) (client *ssh.Client, err error) {
 }
 
 // Command executes shell commands on qemu system
-func (q QemuSystem) Command(user, cmd string) (output string, err error) {
+func (q System) Command(user, cmd string) (output string, err error) {
 	client, err := q.ssh(user)
 	if err != nil {
 		return
@@ -291,7 +330,7 @@ func (q QemuSystem) Command(user, cmd string) (output string, err error) {
 }
 
 // AsyncCommand executes command on qemu system but does not wait for exit
-func (q QemuSystem) AsyncCommand(user, cmd string) (err error) {
+func (q System) AsyncCommand(user, cmd string) (err error) {
 	client, err := q.ssh(user)
 	if err != nil {
 		return
@@ -308,7 +347,7 @@ func (q QemuSystem) AsyncCommand(user, cmd string) (err error) {
 }
 
 // CopyFile is copy file from local machine to remote through ssh/scp
-func (q *QemuSystem) CopyFile(user, localPath, remotePath string) (err error) {
+func (q *System) CopyFile(user, localPath, remotePath string) (err error) {
 	addrPort := strings.Split(q.sshAddrPort, ":")
 	addr := addrPort[0]
 	port := addrPort[1]
@@ -325,7 +364,8 @@ func (q *QemuSystem) CopyFile(user, localPath, remotePath string) (err error) {
 	return
 }
 
-func (q *QemuSystem) CopyAndInsmod(localKoPath string) (output string, err error) {
+// CopyAndInsmod copy kernel module to temporary file on qemu then insmod it
+func (q *System) CopyAndInsmod(localKoPath string) (output string, err error) {
 	remoteKoPath := fmt.Sprintf("/tmp/module_%d.ko", rand.Int())
 	err = q.CopyFile("root", localKoPath, remoteKoPath)
 	if err != nil {
@@ -336,7 +376,7 @@ func (q *QemuSystem) CopyAndInsmod(localKoPath string) (output string, err error
 }
 
 // CopyAndRun is copy local file to qemu vm then run it
-func (q *QemuSystem) CopyAndRun(user, path string) (output string, err error) {
+func (q *System) CopyAndRun(user, path string) (output string, err error) {
 	remotePath := fmt.Sprintf("/tmp/executable_%d", rand.Int())
 	err = q.CopyFile(user, path, remotePath)
 	if err != nil {
@@ -346,12 +386,54 @@ func (q *QemuSystem) CopyAndRun(user, path string) (output string, err error) {
 	return q.Command(user, "chmod +x "+remotePath+" && "+remotePath)
 }
 
-func (q *QemuSystem) Debug(conn string) {
+// Debug is for enable qemu debug and set hostname and port for listen
+func (q *System) Debug(conn string) {
 	q.debug = true
 	q.gdb = conn
 }
 
-func (q QemuSystem) GetSshCommand() (cmd string) {
+// SetKASLR is changing KASLR state through kernel boot args
+func (q *System) SetKASLR(state bool) {
+	q.noKASLR = !state
+}
+
+// SetSMEP is changing SMEP state through kernel boot args
+func (q *System) SetSMEP(state bool) {
+	q.noSMEP = !state
+}
+
+// SetSMAP is changing SMAP state through kernel boot args
+func (q *System) SetSMAP(state bool) {
+	q.noSMAP = !state
+}
+
+// SetKPTI is changing KPTI state through kernel boot args
+func (q *System) SetKPTI(state bool) {
+	q.noKPTI = !state
+}
+
+// GetKASLR is retrieve KASLR settings
+func (q *System) GetKASLR() bool {
+	return !q.noKASLR
+}
+
+// GetSMEP is retrieve SMEP settings
+func (q *System) GetSMEP() bool {
+	return !q.noSMEP
+}
+
+// GetSMAP is retrieve SMAP settings
+func (q *System) GetSMAP() bool {
+	return !q.noSMAP
+}
+
+// GetKPTI is retrieve KPTI settings
+func (q *System) GetKPTI() bool {
+	return !q.noKPTI
+}
+
+// GetSSHCommand returns command for connect to qemu machine over ssh
+func (q System) GetSSHCommand() (cmd string) {
 	addrPort := strings.Split(q.sshAddrPort, ":")
 	addr := addrPort[0]
 	port := addrPort[1]

qemu/qemu-kernel_test.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a AGPLv3 license
 // (or later) that can be found in the LICENSE file.
 
-package qemukernel
+package qemu
 
 import (
 	"crypto/sha512"
@@ -20,46 +20,46 @@ func init() {
 	rand.Seed(time.Now().UnixNano())
 }
 
-func TestQemuSystemNew_InvalidKernelPath(t *testing.T) {
+func TestSystemNew_InvalidKernelPath(t *testing.T) {
 	kernel := Kernel{Name: "Invalid", KernelPath: "/invalid/path"}
-	if _, err := NewQemuSystem(X86_64, kernel, "/bin/sh"); err == nil {
+	if _, err := NewSystem(X86x64, kernel, "/bin/sh"); err == nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemNew_InvalidQemuArch(t *testing.T) {
+func TestSystemNew_InvalidQemuArch(t *testing.T) {
 	kernel := Kernel{Name: "Valid path", KernelPath: testConfigVmlinuz}
-	if _, err := NewQemuSystem(unsupported, kernel, "/bin/sh"); err == nil {
+	if _, err := NewSystem(unsupported, kernel, "/bin/sh"); err == nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemNew_InvalidQemuDrivePath(t *testing.T) {
+func TestSystemNew_InvalidQemuDrivePath(t *testing.T) {
 	kernel := Kernel{Name: "Valid path", KernelPath: testConfigVmlinuz}
-	if _, err := NewQemuSystem(X86_64, kernel, "/invalid/path"); err == nil {
+	if _, err := NewSystem(X86x64, kernel, "/invalid/path"); err == nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemNew(t *testing.T) {
+func TestSystemNew(t *testing.T) {
 	kernel := Kernel{Name: "Valid path", KernelPath: testConfigVmlinuz}
-	if _, err := NewQemuSystem(X86_64, kernel, "/bin/sh"); err != nil {
+	if _, err := NewSystem(X86x64, kernel, "/bin/sh"); err != nil {
 		t.Fatal(err)
 	}
 }
 
-func TestQemuSystemStart(t *testing.T) {
+func TestSystemStart(t *testing.T) {
 	kernel := Kernel{Name: "Test kernel", KernelPath: testConfigVmlinuz}
-	qemu, err := NewQemuSystem(X86_64, kernel, "/bin/sh")
+	q, err := NewSystem(X86x64, kernel, "/bin/sh")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	if err = qemu.Start(); err != nil {
+	if err = q.Start(); err != nil {
 		t.Fatal(err)
 	}
 
-	qemu.Stop()
+	q.Stop()
 }
 
 func TestGetFreeAddrPort(t *testing.T) {
@@ -71,39 +71,39 @@ func TestGetFreeAddrPort(t *testing.T) {
 	ln.Close()
 }
 
-func TestQemuSystemStart_Timeout(t *testing.T) {
+func TestSystemStart_Timeout(t *testing.T) {
 	t.Parallel()
 	kernel := Kernel{Name: "Test kernel", KernelPath: testConfigVmlinuz}
-	qemu, err := NewQemuSystem(X86_64, kernel, "/bin/sh")
+	q, err := NewSystem(X86x64, kernel, "/bin/sh")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	qemu.Timeout = time.Second
+	q.Timeout = time.Second
 
-	if err = qemu.Start(); err != nil {
+	if err = q.Start(); err != nil {
 		t.Fatal(err)
 	}
 
 	time.Sleep(2 * time.Second)
 
-	if !qemu.Died {
+	if !q.Died {
 		t.Fatal("qemu does not died :c")
 	}
 
-	if !qemu.KilledByTimeout {
+	if !q.KilledByTimeout {
 		t.Fatal("qemu died not because of timeout O_o")
 	}
 }
 
-func startTestQemu(t *testing.T, timeout time.Duration) (q *QemuSystem, err error) {
+func startTestQemu(t *testing.T, timeout time.Duration) (q *System, err error) {
 	t.Parallel()
 	kernel := Kernel{
 		Name:       "Test kernel",
 		KernelPath: testConfigVmlinuz,
 		InitrdPath: testConfigInitrd,
 	}
-	q, err = NewQemuSystem(X86_64, kernel, testConfigRootfs)
+	q, err = NewSystem(X86x64, kernel, testConfigRootfs)
 	if err != nil {
 		return
 	}
@@ -119,14 +119,14 @@ func startTestQemu(t *testing.T, timeout time.Duration) (q *QemuSystem, err erro
 	return
 }
 
-func TestQemuSystemCommand(t *testing.T) {
+func TestSystemCommand(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
-	output, err := qemu.Command("root", "cat /etc/shadow")
+	output, err := q.Command("root", "cat /etc/shadow")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -134,7 +134,7 @@ func TestQemuSystemCommand(t *testing.T) {
 		t.Fatal("Wrong output from `cat /etc/shadow` by root")
 	}
 
-	output, err = qemu.Command("user", "cat /etc/passwd")
+	output, err = q.Command("user", "cat /etc/passwd")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -142,18 +142,19 @@ func TestQemuSystemCommand(t *testing.T) {
 		t.Fatal("Wrong output from `cat /etc/passwd` by user")
 	}
 
-	output, err = qemu.Command("user", "cat /etc/shadow")
+	_, err = q.Command("user", "cat /etc/shadow")
-	if err == nil { // unsucessful is good because user must not read /etc/shadow
+	// unsuccessful is good because user must not read /etc/shadow
+	if err == nil {
 		t.Fatal("User have rights for /etc/shadow. WAT?!")
 	}
 }
 
-func TestQemuSystemCopyFile(t *testing.T) {
+func TestSystemCopyFile(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	localPath := "/bin/sh"
 
@@ -162,30 +163,31 @@ func TestQemuSystemCopyFile(t *testing.T) {
 		return
 	}
 
-	sha_local := fmt.Sprintf("%x", sha512.Sum512(content))
+	shaLocal := fmt.Sprintf("%x", sha512.Sum512(content))
 
-	err = qemu.CopyFile("user", localPath, "/tmp/test")
+	err = q.CopyFile("user", localPath, "/tmp/test")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	sha_remote, err := qemu.Command("user", "sha512sum /tmp/test")
+	shaRemote, err := q.Command("user", "sha512sum /tmp/test")
 	if err != nil {
 		t.Fatal(err)
 	}
-	sha_remote = strings.Split(sha_remote, " ")[0]
+	shaRemote = strings.Split(shaRemote, " ")[0]
 
-	if sha_local != sha_remote {
+	if shaLocal != shaRemote {
-		t.Fatal(fmt.Sprintf("Broken file (%s instead of %s)", sha_remote, sha_local))
+		t.Fatal(fmt.Sprintf("Broken file (%s instead of %s)",
+			shaRemote, shaLocal))
 	}
 }
 
-func TestQemuSystemCopyAndRun(t *testing.T) {
+func TestSystemCopyAndRun(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	randStr := fmt.Sprintf("%d", rand.Int())
 	content := []byte("#!/bin/sh\n echo -n " + randStr + "\n")
@@ -203,34 +205,35 @@ func TestQemuSystemCopyAndRun(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	output, err := qemu.CopyAndRun("user", tmpfile.Name())
+	output, err := q.CopyAndRun("user", tmpfile.Name())
 	if err != nil {
 		t.Fatal(err)
 	}
 
 	if output != randStr {
-		t.Fatal("Wrong output from copyied executable (" + output + "," + randStr + ")")
+		t.Fatal("Wrong output from copyied executable (" +
+			output + "," + randStr + ")")
 	}
 }
 
-func TestQemuSystemCopyAndInsmod(t *testing.T) {
+func TestSystemCopyAndInsmod(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
-	lsmodBefore, err := qemu.Command("root", "lsmod | wc -l")
+	lsmodBefore, err := q.Command("root", "lsmod | wc -l")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	_, err = qemu.CopyAndInsmod(testConfigSampleKo)
+	_, err = q.CopyAndInsmod(testConfigSampleKo)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	lsmodAfter, err := qemu.Command("root", "lsmod | wc -l")
+	lsmodAfter, err := q.Command("root", "lsmod | wc -l")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -240,21 +243,21 @@ func TestQemuSystemCopyAndInsmod(t *testing.T) {
 	}
 }
 
-func TestQemuSystemKernelPanic(t *testing.T) {
+func TestSystemKernelPanic(t *testing.T) {
-	qemu, err := startTestQemu(t, time.Minute)
+	q, err := startTestQemu(t, 5*time.Minute)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	// Enable sysrq
-	_, err = qemu.Command("root", "echo 1 > /proc/sys/kernel/sysrq")
+	_, err = q.Command("root", "echo 1 > /proc/sys/kernel/sysrq")
 	if err != nil {
 		t.Fatal(err)
 	}
 
 	// Trigger kernel panic
-	err = qemu.AsyncCommand("root", "sleep 1s && echo c > /proc/sysrq-trigger")
+	err = q.AsyncCommand("root", "sleep 1s && echo c > /proc/sysrq-trigger")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -262,40 +265,41 @@ func TestQemuSystemKernelPanic(t *testing.T) {
 	// Wait for panic watcher timeout
 	time.Sleep(5 * time.Second)
 
-	if qemu.KilledByTimeout {
+	if q.KilledByTimeout {
 		t.Fatal("qemu is killed by timeout, not because of panic")
 	}
 
-	if !qemu.Died {
+	if !q.Died {
 		t.Fatal("qemu is not killed after kernel panic")
 	}
 
-	if !qemu.KernelPanic {
+	if !q.KernelPanic {
 		t.Fatal("qemu is died but there's no information about panic")
 	}
 }
 
-func TestQemuSystemRun(t *testing.T) {
+func TestSystemRun(t *testing.T) {
-	qemu, err := startTestQemu(t, 0)
+	q, err := startTestQemu(t, 0)
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer qemu.Stop()
+	defer q.Stop()
 
 	for {
-		_, err := qemu.Command("root", "echo")
+		_, err := q.Command("root", "echo")
 		if err == nil {
 			break
 		}
 	}
 
 	start := time.Now()
-	err = qemu.AsyncCommand("root", "sleep 10s")
+	err = q.AsyncCommand("root", "sleep 1m")
 	if err != nil {
 		t.Fatal(err)
 	}
-	if time.Since(start) > time.Second {
+	if time.Since(start) > 10*time.Second {
-		t.Fatalf("qemu.Run does not async (waited %s)", +time.Since(start))
+		t.Fatalf("q.AsyncCommand does not async (waited %s)",
+			time.Since(start))
 	}
 
 }
@@ -309,13 +313,13 @@ func openedPort(port int) bool {
 	return true
 }
 
-func TestQemuSystemDebug(t *testing.T) {
+func TestSystemDebug(t *testing.T) {
 	t.Parallel()
 	kernel := Kernel{
 		KernelPath: testConfigVmlinuz,
 		InitrdPath: testConfigInitrd,
 	}
-	q, err := NewQemuSystem(X86_64, kernel, testConfigRootfs)
+	q, err := NewSystem(X86x64, kernel, testConfigRootfs)
 	if err != nil {
 		return
 	}

qemu/test.config.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a AGPLv3 license
 // (or later) that can be found in the LICENSE file.
 
-package qemukernel
+package qemu
 
 const testConfigVmlinuz = "../tools/qemu-debian-img/ubuntu1804.vmlinuz"
 const testConfigInitrd = "../tools/qemu-debian-img/ubuntu1804.initrd"

tools/kernel-factory/.gitignore

@@ -1 +0,0 @@
-output

tools/kernel-factory/Ubuntu/14.04/Dockerfile

@@ -1,9 +0,0 @@
-FROM ubuntu:14.04
-
-RUN apt-get update
-
-# for linux-image-3*-generic, so... CRUTCHES!
-# E: Unable to locate package linux-image-3*-generic
-ENV DEBIAN_FRONTEND=noninteractive
-RUN apt search linux-image | grep -e linux-image-3 -e linux-image-4 | grep generic | cut -d '/' -f 1 | xargs apt install -y
-RUN apt search linux-headers | grep -e linux-headers-3 -e linux-headers-4 | grep generic | cut -d '/' -f 1 | xargs apt install -y

tools/kernel-factory/Ubuntu/14.04/Image

@@ -1 +0,0 @@
-../../../qemu-debian-img/ubuntu1404.img

tools/kernel-factory/Ubuntu/16.04/Dockerfile

@@ -1,4 +0,0 @@
-FROM ubuntu:16.04
-
-RUN apt-get update
-RUN DEBIAN_FRONTEND=noninteractive apt-get install -y linux-image-4*-generic linux-headers-*-generic build-essential wget git

tools/kernel-factory/Ubuntu/16.04/Image

@@ -1 +0,0 @@
-../../../qemu-debian-img/ubuntu1604.img

tools/kernel-factory/Ubuntu/18.04/Dockerfile

@@ -1,4 +0,0 @@
-FROM ubuntu:18.04
-
-RUN apt-get update
-RUN DEBIAN_FRONTEND=noninteractive apt-get install -y linux-image-4*-generic linux-headers-*-generic build-essential wget git libelf-dev

tools/kernel-factory/Ubuntu/18.04/Image

@@ -1 +0,0 @@
-../../../qemu-debian-img/ubuntu1804.img

tools/kernel-factory/bootstrap.sh

@@ -1,29 +0,0 @@
-#!/bin/sh -eux
-mkdir -p output
-echo > output/kernels.toml
-find | grep Docker | sed 's/Dockerfile//' | while read DOCKER; do
-    CONTAINER_NAME=$(echo $DOCKER | sed -e 's;/;;g' -e 's;\.;;g' -e 's;\(.*\);\L\1;')
-    docker build -t ${CONTAINER_NAME} ${DOCKER}
-    docker run ${CONTAINER_NAME} bash -c 'ls /boot'
-    CONTAINER_ID=$(docker ps -a | grep ${CONTAINER_NAME} | awk '{print $1}' | head -n 1)
-    docker cp ${CONTAINER_ID}:/boot/. output/
-    DISTRO_NAME=$(echo $DOCKER | cut -d '/' -f 2)
-    DISTRO_VER=$(echo $DOCKER | cut -d '/' -f 3)
-
-    BOOT_FILES="$(docker run $CONTAINER_NAME ls /boot)"
-    for KERNEL_RELEASE in $(docker run $CONTAINER_NAME ls /lib/modules); do
-	echo '[[Kernels]]' >> output/kernels.toml
-	echo 'distro_type =' \"$DISTRO_NAME\" >> output/kernels.toml
-	echo 'distro_release =' \"$DISTRO_VER\" >> output/kernels.toml
-	echo 'kernel_release =' \"$KERNEL_RELEASE\" >> output/kernels.toml
-	echo 'container_name =' \"$CONTAINER_NAME\" >> output/kernels.toml
-	KERNEL_PATH=$(echo $BOOT_FILES | sed  's/ /\n/g' | grep $KERNEL_RELEASE | grep vmlinuz)
-	echo 'kernel_path =' \"$(realpath output/$KERNEL_PATH)\" >> output/kernels.toml
-	INITRD_PATH=$(echo $BOOT_FILES | sed  's/ /\n/g' | grep $KERNEL_RELEASE | grep init)
-	echo 'initrd_path =' \"$(realpath output/$INITRD_PATH)\" >> output/kernels.toml
-	ROOTFS_PATH=$(realpath $DOCKER/Image)
-	echo 'root_f_s =' \"$ROOTFS_PATH\" >> output/kernels.toml
-	echo >> output/kernels.toml
-    done
-done
-rm -rf output/grub

tools/qemu-centos-img/7/Dockerfile

@@ -1,4 +1,4 @@
-# Copyright 2018 Mikhail Klementev. All rights reserved.
+# Copyright 2019 Mikhail Klementev. All rights reserved.
 # Use of this source code is governed by a AGPLv3 license
 # (or later) that can be found in the LICENSE file.
 #
@@ -7,11 +7,13 @@
 #     $ docker build -t gen-centos7-image .
 #     $ docker run --privileged -v $(pwd):/shared -t gen-centos7-image
 #
-# centos7.img will be created in current directory. You can change $(pwd) to
+# out_of_tree_centos_7.img will be created in current directory.
-# different directory to use different destination for image.
+# You can change $(pwd) to different directory to use different destination
+# for image.
 #
 FROM centos:7
 
+RUN yum -y update
 RUN yum -y groupinstall "Development Tools"
 RUN yum -y install qemu-img e2fsprogs
 
@@ -26,7 +28,7 @@ RUN yum --installroot=$TMPDIR \
         --releasever=7 \
         --disablerepo='*' \
         --enablerepo=base \
-        -y install openssh-server
+        -y install openssh-server openssh-clients
 
 RUN chroot $TMPDIR /bin/sh -c 'useradd -m user'
 RUN sed -i 's/root:\*:/root::/' $TMPDIR/etc/shadow
@@ -37,13 +39,11 @@ RUN sed -i '/PermitRootLogin/d' $TMPDIR/etc/ssh/sshd_config
 RUN echo PermitRootLogin yes >> $TMPDIR/etc/ssh/sshd_config
 
 # network workaround
-# FIXME kernel module compatibility issues
 RUN chmod +x $TMPDIR/etc/rc.local
-RUN echo 'find /lib/modules | grep e1000.ko | xargs insmod -f' >> $TMPDIR/etc/rc.local
 RUN echo 'dhclient' >> $TMPDIR/etc/rc.local
 
 ENV IMAGEDIR=/tmp/image
-ENV IMAGE=/shared/centos7.img
+ENV IMAGE=/shared/out_of_tree_centos_7.img
 
 RUN mkdir $IMAGEDIR
 

tools/qemu-debian-img/Dockerfile

@@ -7,7 +7,7 @@
 #     $ docker build -t gen-ubuntu1804-image .
 #     $ docker run --privileged -v $(pwd):/shared -t gen-ubuntu1804-image
 #
-# centos7.img will be created in current directory. You can change $(pwd) to
+# ubuntu1804.img will be created in current directory. You can change $(pwd) to
 # different directory to use different destination for image.
 #
 FROM ubuntu:18.04

tools/qemu-debian-img/bootstrap.sh

@@ -1,4 +1,6 @@
 #!/bin/sh -eux
+cd $(dirname $(realpath $0))
+
 docker build -t gen-ubuntu1804-image .
 docker run --privileged -v $(pwd):/shared -t gen-ubuntu1804-image
 RUN="docker run -v $(pwd):/shared -t gen-ubuntu1804-image"