2
0
Fork 0

Optimize Dhall processing

This commit is contained in:
Ehmry - 2020-04-05 13:31:11 +05:30
parent 049b0d3053
commit 8cdc415417
10 changed files with 312 additions and 161 deletions

View File

@ -7,21 +7,21 @@
/_/ /____/
```
[Genodepkgs](https://git.sr.ht/~ehmry/genodepkgs) is nexus of
[Genodepkgs](https://git.sr.ht/~ehmry/genodepkgs) is nexus of
[Nix](https://nixos.org/nix/) expressions related to the
[Genode OS framework](https://genode.org).
# Packaging
Packaging is done using standard Nixpkgs methods, a `stdenv` is available for
Packaging is done using standard Nixpkgs methods, a `stdenv` is available for
cross-compilation. See [Solo5](./packages/solo5/default.nix) as an example.
# System description format
The high-level interface to system building are boot descriptions. These Dhall
The high-level interface to system building are boot descriptions. These Dhall
documents describe the configuration of an [Init](https://genode.org/documentation/genode-foundations/19.05/system_configuration/The_init_component.html)
instance and a store of ROM (Read-Only Memory) modules. These description can be
used to build firmware-like binary images or merged and nested within other
instance and a store of ROM (Read-Only Memory) modules. These description can be
used to build firmware-like binary images or merged and nested within other
descriptions. In theory these descriptions can arrange file-systems, but those
functions are missing.
@ -30,10 +30,10 @@ https://git.sr.ht/~ehmry/dhall-genode/tree/master/Boot/package.dhall
# Package manifests
Software packages are described my manifest documents, which can be used to
build the ROM map in boot descriptions. This manifests can be automatically
generated and made available within Nix builds and dev-shells via environmental
variables. This may seem awkward, but consider that this is to support building
Software packages are described my manifest documents, which can be used to
build the ROM map in boot descriptions. This manifests can be automatically
generated and made available within Nix builds and dev-shells via environmental
variables. This may seem awkward, but consider that this is to support building
firmware systems without a $PATH or even a file-system.
An example manifest:
@ -62,21 +62,21 @@ in { config = Genode.Init.default, rom = rom } : Genode.Boot.Type
# Contributing
At the moment every user needs to also act as distro developer, designing how
packaging works needs to be done before packages can be submitted at will.
At the moment every user needs to also act as distro developer, designing how
packaging works needs to be done before packages can be submitted at will.
Patches can be submitted and issues reported via the
[mailing list](https://lists.sr.ht/~ehmry/genodepkgs). The mailing list is the
[mailing list](https://lists.sr.ht/~ehmry/genodepkgs). The mailing list is the
minimum viable mechanism for community development, and may be replaced later.
The worksites at the moment are:
- Generation of documentation on internal types and functions. By generating a
comprehensive single document early, we should be able to refactor and prune
- Generation of documentation on internal types and functions. By generating a
comprehensive single document early, we should be able to refactor and prune
the internals faster.
- Testing framework for simple and multi-machine tests.
- Patching standard Nixpkgs packages via an overlay. Workflow and tooling
- Patching standard Nixpkgs packages via an overlay. Workflow and tooling
needs to be explored for building emulated UNIX environments.
- LLVM testing and upstreaming patches.
@ -87,13 +87,13 @@ The worksites at the moment are:
# Nix Flakes
This repository is structured as a Nix flake, which is an experimental extension
of Nix tooling. The flakes capable Nix frontend is available from Nixpkgs as
This repository is structured as a Nix flake, which is an experimental extension
of Nix tooling. The flakes capable Nix frontend is available from Nixpkgs as
`nixFlakes`.
## Switching to flakes
All users regardless of distro must add the following to
All users regardless of distro must add the following to
*~/.config/nix/nix.conf* to enable flakes:
```
@ -156,27 +156,27 @@ $ nix log .#checks.x86_64-linux-x86_64-genode.nova-pci
# Toolchain
The Genode Labs toolchain is available here only for Linux and in binary form.
However, an experimental LLVM cross-compiler is provided here. Bootstrapping GCC
However, an experimental LLVM cross-compiler is provided here. Bootstrapping GCC
has not been practical due to its recursive autotools build system.
# Building
Bootstrapping the build tooling can take several hours on a laptop, therefore it
is recommended enable the binary caches and to farm massive rebuilds to large
multicore machines. The Nix flake is prepared for continuous builds with a
[Hydra](https://nixos.org/hydra/), but only the experimental flakes capable
Bootstrapping the build tooling can take several hours on a laptop, therefore it
is recommended enable the binary caches and to farm massive rebuilds to large
multicore machines. The Nix flake is prepared for continuous builds with a
[Hydra](https://nixos.org/hydra/), but only the experimental flakes capable
Hydra.
## C3D2 Hydra
A publicly available Hydra is provided by [CCC Dresden](https://c3d2.de) at
https://server7.hq.c3d2.de/. The machine is only reachable over IPv6 and
A publicly available Hydra is provided by [CCC Dresden](https://c3d2.de) at
https://server7.hq.c3d2.de/. The machine is only reachable over IPv6 and
therefore inaccessible to SculptOS.
# Testing
Tests are performed using QEMU and a test driver similiar to what is used by
NixOS (ours has not been ported to Python however). The test artifacts are built
Tests are performed using QEMU and a test driver similiar to what is used by
NixOS (ours has not been ported to Python however). The test artifacts are built
as follows:
```shell
@ -186,9 +186,6 @@ nix build .#checks.x86_64-linux-x86_64-genode.nova-solo5-net
# Build an ISO of the test run:
nix build .#checks.x86_64-linux-x86_64-genode.nova-solo5-net.iso
# Build the top-levl XML configuration of a test:
nix build .#checks.x86_64-linux-x86_64-genode.nova-solo5-net.xml
# Build the [SOTEST](https://opensource.sotest.io/) artifacts of a test run.
nix build .#checks.x86_64-linux-x86_64-genode.nova-solo5-net.sotest
```
@ -224,53 +221,53 @@ Others are simply introduced as an aid to the reader:
- *Downstream* - repositories depending on this repository
- *OS ABI* - The binary interface to an operating system. In the case of Genode
this does not include kernel interfaces, as system calls are abstracted via
- *OS ABI* - The binary interface to an operating system. In the case of Genode
this does not include kernel interfaces, as system calls are abstracted via
dynamically linked procedures.
- `localSystem` - the CPU and OS ABI pair used to evaluate and compile
packages. Genode cannot natively instantiate Nix derivations, therefore it
- `localSystem` - the CPU and OS ABI pair used to evaluate and compile
packages. Genode cannot natively instantiate Nix derivations, therefore it
cannot be a `localSystem` ABI.
- `crossSystem` - the target CPU and OS ABI pair for packages. In the common
case the ABI will be Genode. In the case of tooling it may be the same as
`localSystem`. For software that runs bare-metal the CPU is significant but
- `crossSystem` - the target CPU and OS ABI pair for packages. In the common
case the ABI will be Genode. In the case of tooling it may be the same as
`localSystem`. For software that runs bare-metal the CPU is significant but
ABI is irrelevant.
- `buildPlatform` - equivalent to `localSystem`.
- `hostPlatform` - the platform on which a package will execute.
- `targetPlatform` - the platform for which a package will produce artifacts.
This is only significant for software such as compilers. The respective
- `targetPlatform` - the platform for which a package will produce artifacts.
This is only significant for software such as compilers. The respective
build, host, and target platform of a compiler that was built on a Solaris,
executes on Genode, and produces only Haiku binaries would be Solaris,
executes on Genode, and produces only Haiku binaries would be Solaris,
Genode, and Haiku.
- *Flake* - a code repository containing a `flake.nix` file describing
- *Flake* - a code repository containing a `flake.nix` file describing
dependency repositories. The evaluation of flakes may only form ascyclic
graphs. Repositories packaged here may themselves contain a `flake.nix`
file that refers to this repository, but these repositories cannot be inputs
to the Genodepkgs flake.
- *Genode core* - Genode component of highest privilege. The core is located
conceptually at the bottom and root of the component tree. Core might be
directly above the kernel, or acting as the kernel. The role of core is a
- *Genode core* - Genode component of highest privilege. The core is located
conceptually at the bottom and root of the component tree. Core might be
directly above the kernel, or acting as the kernel. The role of core is a
subset of the role of the kernel in a monolithic system.
- *Lower* - lowering a subsystem is moving it closer to the Genode core
and increasing privilege.
- *Raise* - raising a subsystem is moving it further away from the Genode core
- *Raise* - raising a subsystem is moving it further away from the Genode core
and decreasing privilege.
- *Package* - *undefined*
# Legal
This repository is for trivial packaging algebra and metadata that exists only
for the sake of richer and more meaningful code in external repositories. All
works committed to this repository, unless otherwise noted, shall be implicitly
This repository is for trivial packaging algebra and metadata that exists only
for the sake of richer and more meaningful code in external repositories. All
works committed to this repository, unless otherwise noted, shall be implicitly
licensed under a [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/deed)
public domain dedication.

View File

@ -91,9 +91,6 @@ in dhallApps // {
in {
type = "app";
program = "${drv}/bin/nova-image";
function = attrs: bootDesc:
nixpkgs.runCommand "image.elf" attrs
''XDG_CACHE_HOME=$TMPDIR ${drv}/bin/nova-image "${bootDesc}" > $out'';
};
hw-iso = let

View File

@ -16,26 +16,32 @@ in nixpkgs.writeScriptBin "hw-image" (with nixpkgs.buildPackages;
TMPDIR="$(${coreutils}/bin/mktemp -d)"
trap "rm -rf $TMPDIR" err exit
export DHALL_GENODE=${packages.dhallGenode}/source.dhall
export DHALL_GENODE=''${DHALL_GENODE:-${packages.dhallGenode}/binary.dhall}
if [ -z "$XDG_CACHE_HOME" ]
then
export XDG_CACHE_HOME="$TMPDIR"
${buildPackages.xorg.lndir}/bin/lndir -silent \
"${packages.dhallGenode}/.cache" \
"$XDG_CACHE_HOME"
fi
build_core() {
local lib="$1"
local modules="$2"
local link_address="$3"
${apps.dhall.program} text <<< \
"(${../modules.as.dhall}).to64bitImage ($modules)" \
> "$TMPDIR/modules.as"
local lib="$1"
local modules="$2"
local link_address="$3"
${apps.dhall.program} text > "$TMPDIR/modules.as" \
<<< "(${../modules.as.dhall}).to64bitImage ($modules)" \
# compile the boot modules into one object file
$CC -c -x assembler -o "$TMPDIR/boot_modules.o" "$TMPDIR/modules.as"
# compile the boot modules into one object file
$CC -c -x assembler -o "$TMPDIR/boot_modules.o" "$TMPDIR/modules.as"
# link final image
$LD \
--strip-all \
-T${genodeSources}/repos/base/src/ld/genode.ld \
-z max-page-size=0x1000 \
-Ttext=$link_address -gc-sections \
"$lib" "$TMPDIR/boot_modules.o"
# link final image
$LD \
--strip-all \
-T${genodeSources}/repos/base/src/ld/genode.ld \
-z max-page-size=0x1000 \
-Ttext=$link_address -gc-sections \
"$lib" "$TMPDIR/boot_modules.o"
cat a.out
}

View File

@ -111,7 +111,7 @@
apps' = forAllCrossSystems ({ system, localSystem, crossSystem }:
import ./apps {
self = self.apps.${system};
nixpkgs = self.legacyPackages.${system};
nixpkgs = nixpkgsFor.${system};
nixpkgsLocal = nixpkgsFor.${localSystem};
packages = self.packages.${system};
});

125
lib/compile-boot.dhall Normal file
View File

@ -0,0 +1,125 @@
-- SPDX-License-Identifier: CC0-1.0
let Genode = env:DHALL_GENODE
let Prelude = Genode.Prelude
let BootModules = Genode.BootModules
let RomEntry = Prelude.Map.Entry Text BootModules.ROM.Type
let compile =
λ(addressType : Text)
→ λ(boot : Genode.Boot.Type)
→ λ(out : Text)
→ let NaturalIndex = { index : Natural, value : Text }
let TextIndex = { index : Text, value : Text }
let moduleKeys =
Prelude.Map.keys Text BootModules.ROM.Type boot.rom # [ "config" ]
let moduleValues =
let f =
λ(e : RomEntry)
→ merge
{ RomText = λ(text : Text) → ".ascii ${Text/show text}"
, RomPath = λ(path : Text) → ".incbin ${Text/show path}"
}
e.mapValue
in Prelude.List.map RomEntry Text f boot.rom
# [ ".incbin \"${out}/config\"" ]
let map =
λ(list : List Text)
→ λ(f : TextIndex → Text)
→ let indexedNatural = Prelude.List.indexed Text list
let indexed =
Prelude.List.map
NaturalIndex
TextIndex
( λ(x : NaturalIndex)
→ { index = Prelude.Natural.show x.index
, value = x.value
}
)
indexedNatural
let texts = Prelude.List.map TextIndex Text f indexed
in Prelude.Text.concatSep "\n" texts
let mapNames = map moduleKeys
let mapValues = map moduleValues
let asm =
''
.set MIN_PAGE_SIZE_LOG2, 12
.set DATA_ACCESS_ALIGNM_LOG2, 3
.section .data
.p2align DATA_ACCESS_ALIGNM_LOG2
.global _boot_modules_headers_begin
_boot_modules_headers_begin:
''
++ mapNames
( λ(m : TextIndex)
→ ''
${addressType} _boot_module_${m.index}_name
${addressType} _boot_module_${m.index}_begin
${addressType} _boot_module_${m.index}_end - _boot_module_${m.index}_begin
''
)
++ ''
.global _boot_modules_headers_end
_boot_modules_headers_end:
''
++ mapNames
( λ(m : TextIndex)
→ ''
.p2align DATA_ACCESS_ALIGNM_LOG2
_boot_module_${m.index}_name:
.string "${m.value}"
.byte 0
''
)
++ ''
.section .data.boot_modules_binaries
.global _boot_modules_binaries_begin
_boot_modules_binaries_begin:
''
++ mapValues
( λ(m : TextIndex)
→ ''
.p2align MIN_PAGE_SIZE_LOG2
_boot_module_${m.index}_begin:
${m.value}
_boot_module_${m.index}_end:
''
)
++ ''
.p2align MIN_PAGE_SIZE_LOG2
.global _boot_modules_binaries_end
_boot_modules_binaries_end:
''
in { config = Genode.Init.render boot.config
, modules_asm = asm
, stats =
let sum = Genode.Init.resources boot.config
in "RAM=${Prelude.Natural.show sum.ram}"
}
let funcs = { to32bitImage = compile ".long", to64bitImage = compile ".quad" }
in funcs.to64bitImage

View File

@ -4,22 +4,75 @@
let
thisSystem = builtins.getAttr system;
hostPkgs = nixpkgs.legacyPackages.${localSystem};
inherit (nixpkgs) buildPackages;
testPkgs = thisSystem genodepkgs.packages;
in {
dhallText = name: source:
hostPkgs.runCommand name {
inherit name source;
preferLocalBuild = true;
buildInputs = [ hostPkgs.dhall ];
DHALL_PRELUDE = "${testPkgs.dhallPackages.prelude}/package.dhall";
DHALL_GENODE = "${testPkgs.dhallPackages.genode}/package.dhall";
} ''
in rec {
linuxScript = name: env: bootDhall:
buildPackages.runCommand name ({
inherit name;
buildInputs = [ buildPackages.dhall ];
DHALL_GENODE = "${testPkgs.dhallGenode}/binary.dhall";
} // env) ''
export XDG_CACHE_HOME=$NIX_BUILD_TOP
dhall text < $source > $out
${buildPackages.xorg.lndir}/bin/lndir -silent \
${testPkgs.dhallGenode}/.cache \
$XDG_CACHE_HOME
dhall to-directory-tree --output $out \
<<< "${./linux-script.dhall} (${bootDhall}) \"$out\""
'';
compileBoot = name: env: bootDhall:
buildPackages.runCommand name ({
inherit name;
buildInputs = [ buildPackages.dhall ];
DHALL_GENODE = "${testPkgs.dhallGenode}/binary.dhall";
} // env) ''
export XDG_CACHE_HOME=$NIX_BUILD_TOP
${buildPackages.xorg.lndir}/bin/lndir -silent \
${testPkgs.dhallGenode}/.cache \
$XDG_CACHE_HOME
dhall to-directory-tree --output $out \
<<< "${./compile-boot.dhall} (${bootDhall}) \"$out\""
substituteInPlace "$out/modules_asm" \
--replace '.incbin "./config"' ".incbin \"$out/config\""
${buildPackages.libxml2}/bin/xmllint \
-schema ${testPkgs.genodeSources}/repos/os/src/init/config.xsd \
-noout $out/config
'';
novaImage = name: env: boot:
let inherit (nixpkgs.stdenv) cc;
in nixpkgs.stdenv.mkDerivation {
name = name + ".image.elf";
# buildInputs = with buildPackages; [ stdenv.cc ];
build = compileBoot name env boot;
# CC="${cc}/bin/${cc.targetPrefix}cc"
buildCommand = ''
# compile the boot modules into one object file
$CC -c -x assembler -o "boot_modules.o" "$build/modules_asm"
# link final image
LD="${buildPackages.binutils}/bin/${buildPackages.binutils.targetPrefix}ld"
$LD --strip-all -nostdlib \
-T${testPkgs.genodeSources}/repos/base/src/ld/genode.ld \
-T${testPkgs.genodeSources}/repos/base-nova/src/core/core-bss.ld \
-z max-page-size=0x1000 \
-Ttext=0x100000 -gc-sections \
"${testPkgs.base-nova.coreObj}" boot_modules.o \
-o $out
'';
};
mergeManifests = inputs:
nixpkgs.writeTextFile {
name = "manifest.dhall";

42
lib/linux-script.dhall Normal file
View File

@ -0,0 +1,42 @@
-- SPDX-License-Identifier: CC0-1.0
let Genode = env:DHALL_GENODE
let Prelude = Genode.Prelude
let Args = { config : Genode.Init.Type, rom : Genode.BootModules.Type } : Type
let RomEntry = Prelude.Map.Entry Text Genode.BootModules.ROM.Type
let addLine =
λ(e : RomEntry)
→ λ(script : Text)
→ merge
{ RomText =
λ(rom : Text)
→ ''
${script}
echo ${Text/show rom} > ${Text/show e.mapKey}
''
, RomPath =
λ(rom : Text)
→ ''
${script}
ln -s ${Text/show rom} ${Text/show e.mapKey}
''
}
e.mapValue
in λ(args : Args) -> \(out : Text)
{ config = Genode.Init.render args.config
, script = Prelude.List.fold
RomEntry
args.rom
Text
addLine
''
#!/bin/sh
ln -s ${out}/config config
''
}

View File

@ -46,6 +46,8 @@ let
XDG_CACHE_HOME = "/tmp";
} // env;
build = lib.linuxScript name env' testConfig';
toExports = env:
with builtins;
let
@ -71,7 +73,7 @@ let
global env
set env(XDG_CACHE_HOME) /tmp
exec ${apps.linux-image.program} ${testConfig'}
exec sh ${build}/script
spawn ./core-linux
wait_for_output $wait_for_re $timeout_value $spawn_id
@ -97,7 +99,7 @@ let
test = runTests driver;
in test // {
inherit driver test;
inherit build driver test;
config = buildPkgs.runCommand (name + ".dhall") env' ''
${apps.dhall.program} <<< "(${testConfig'}).config" > $out
'';
@ -107,7 +109,7 @@ let
image = buildPkgs.runCommand (name + ".image.elf") env' ''
mkdir -p $out
pushd $out
${apps.linux-image.program} "${testConfig'}"
. ${build}/script
'';
};

View File

@ -49,7 +49,8 @@ let
XDG_CACHE_HOME = "/tmp";
} // testEnv;
image = apps.nova-image.function testEnv' testConfig';
image = lib.novaImage name testEnv' testConfig';
build = lib.compileBoot name testEnv' testConfig';
baseSetup = ''
##
@ -115,13 +116,13 @@ let
mkdir -p $out/bin
echo "$testScript" > $out/test-script
echo "$baseSetup" > $out/base-setup
eval $(${apps.dhall-to-bash.program} --declare "TEST_RAM" \
<<< "${./boot-ram.dhall} (${testConfig'})")
source ${build}/stats
ln -s ${testDriver}/bin/genode-test-driver $out/bin/
wrapProgram $out/bin/genode-test-driver \
--set testScript "$testScript" \
--set baseSetup "$baseSetup" \
--set TEST_RAM $TEST_RAM \
--set TEST_RAM $RAM \
'';
passMeta = drv:
@ -131,17 +132,15 @@ let
test = passMeta (runTests driver);
in test // {
inherit driver image test manifest;
inherit build driver image test manifest;
config = buildPkgs.runCommand (name + ".dhall") testEnv' ''
export XDG_CACHE_HOME=''${TMPDIR:-/tmp}
${apps.dhall.program} <<< "${testConfig'}" > $out
'';
iso = apps.nova-iso.function testEnv' "${testConfig'}";
compile = lib.compileBoot name testConfig';
xml = buildPkgs.runCommand (name + ".config") testEnv'
''${apps.render-init.program} <<< "(${testConfig'}).config" > $out'';
iso = apps.nova-iso.function testEnv' "${testConfig'}";
sotest = buildPkgs.runCommand "nova-${name}-sotest" testEnv' ''
cp "${testPkgs.bender}" bender

View File

@ -1,75 +1,5 @@
#!/usr/bin/env expect
##
# Generate assembly code aggregating boot-module data from specified files.
#
proc generate_boot_modules_asm {modules} {
# architecture dependent definitions
set address_type ".quad"
# header
set asm_src {}
append asm_src ".set MIN_PAGE_SIZE_LOG2, 12\n"
append asm_src ".set DATA_ACCESS_ALIGNM_LOG2, 3\n"
append asm_src "\n"
append asm_src ".section .data\n"
append asm_src "\n"
append asm_src ".p2align DATA_ACCESS_ALIGNM_LOG2\n"
append asm_src ".global _boot_modules_headers_begin\n"
append asm_src "_boot_modules_headers_begin:\n"
append asm_src "\n"
# module list
set i 0
foreach module $modules {
incr i
append asm_src "${address_type} _boot_module_${i}_name\n"
append asm_src "${address_type} _boot_module_${i}_begin\n"
append asm_src "${address_type} _boot_module_${i}_end -"
append asm_src " _boot_module_${i}_begin\n"
append asm_src "\n"
}
append asm_src ".global _boot_modules_headers_end\n"
append asm_src "_boot_modules_headers_end:\n"
append asm_src "\n"
# module names
set i 0
foreach module $modules {
incr i
append asm_src ".p2align DATA_ACCESS_ALIGNM_LOG2\n"
append asm_src "_boot_module_${i}_name:\n"
append asm_src ".string \"${module}\"\n"
append asm_src ".byte 0\n"
append asm_src "\n"
}
# header end
append asm_src ".section .data.boot_modules_binaries\n"
append asm_src "\n"
append asm_src ".global _boot_modules_binaries_begin\n"
append asm_src "_boot_modules_binaries_begin:\n"
append asm_src "\n"
# module data
set i 0
foreach module $modules {
incr i
append asm_src ".p2align MIN_PAGE_SIZE_LOG2\n"
append asm_src "_boot_module_${i}_begin:\n"
append asm_src ".incbin \"${module}\"\n"
append asm_src "_boot_module_${i}_end:\n"
append asm_src "\n"
}
append asm_src ".p2align MIN_PAGE_SIZE_LOG2\n"
append asm_src ".global _boot_modules_binaries_end\n"
append asm_src "_boot_modules_binaries_end:\n"
return $asm_src
}
eval $env(baseSetup)
eval $env(testScript)