From 456dda9ac0198678eae5bac198c829abe7d1afe1 Mon Sep 17 00:00:00 2001 From: Norman Feske Date: Mon, 26 Nov 2018 15:26:02 +0100 Subject: [PATCH] Release notes for version 18.11 --- doc/release_notes-18-11.txt | 1107 +++++++++++++++++++++++++++++++++++ 1 file changed, 1107 insertions(+) create mode 100644 doc/release_notes-18-11.txt diff --git a/doc/release_notes-18-11.txt b/doc/release_notes-18-11.txt new file mode 100644 index 000000000..e3845731e --- /dev/null +++ b/doc/release_notes-18-11.txt @@ -0,0 +1,1107 @@ + + + =============================================== + Release notes for the Genode OS Framework 18.11 + =============================================== + + Genode Labs + + + +Our [https://genode.org/about/road-map - road map] for 2018 emphasized +software quality and resilience as one of the major topics of the year. +The release 18.11 pays tribute to this plan on multiple levels. + +First, by integrating *static code analysis* into Genode's build and packaging +tools, we aim to cultivate rigid code analysis as a regular part of our work +flows to catch bugs as early as possible - before running any code +(Section [Static code analysis]). + +The second line of defense are intensive automated tests, which are steadily +extended. The tests can roughly be classified into unit tests, component +tests, and integration tests. By moving over 70 *component tests* from +individual test scenarios (run scripts) to Genode packages, we became able to +run all these tests as one big integration test. As described in +Section [Automated test infrastructure hosted on top of Genode], the large +batch of component tests, their orchestration, and the analysis of the results +are performed directly on the target platform. This approach not only stresses +the resilience of the base framework and the underlying kernels, but it also +makes each individual test easily reproducible, e.g., on top of Sculpt OS. + +For unit tests, *coverage metrics* are a useful tool to aid the test +development. The current release integrates the support of the gcov tool into +the build system and test infrastructure so that these metrics become easy to +obtain. + +To improve the resilience of Genode systems that contain parts that are +known/expected to sometimes fail, e.g., because they depend on hugely complex +software stacks, the new release features the ability to *monitor* *the* +*health* of components (Section [Component health monitoring]). Using this new +introspection mechanism, Genode systems become able to respond to such +conditions by restarting the affected component or by logging the event. + +The most sophisticated integration test is certainly the interactive use of +*Sculpt OS* on a daily basis. On that account, we are happy to introduce a few +new toys to play with. By introducing a new window layouter and modularizing +Sculpt's window-management packages, window decorators can now be swapped out +at runtime, window layouts are preserved across reboots, and the layout rules +can be edited on the fly (Section [Enhanced window-management flexibility]). + +Also outlined on our road map, we strive to apply Genode for network +appliances and typical server applications. With the new ability to host +MirageOS unikernels directly on Genode +(Section [Genode as a platform for Mirage-OS unikernels]), MirageOS-based +server applications can be readily integrated into Genode systems. Another +key component for server use cases is the new SSH server described in +Section [SSH terminal server], which allows for the friction-less remote +administration of Genode-based servers. + +Further highlights of the current release are the improved network performance +on Xilinx Zynq, the initial version of a Genode SDK, performance improvements +of the base-hw kernel on NXP i.MX platforms, and the updated language support +for Ada and Java. + + +Raising the bar of quality assurance +#################################### + +Automated test infrastructure hosted on top of Genode +===================================================== + +When it comes to testing, Genode's run tool provides a convenient way for +building, configuring, executing, and evaluating system scenarios. Throughout +the years, the base framework became equipped with a broad variety of run +scripts, each dedicated to another aspect of Genode. Most of them support +being integrated into a fully automated test infrastructure. For instance, all +the tests listed in _tool/autopilot.list_ are automatically evaluated each +night at Genode Labs for different platforms and kernels. This exposes the +current state of development to a variety of base-system behavior (e.g., +scheduling) and quickly reveals regression bugs. + +[image depot_autopilot_before] + +However, the existing approach had its drawbacks: + +* As each test was realized through a run script executed individually, + the target platform had to be booted over and over again. So, for larger + lists of tests like _tool/autopilot.list_, boot times could sum up to a + significant magnitude. Moreover, continuous on and off switching of + hardware may have a negative effect on its lifetime. + +* Most of the mentioned run scripts neither needed more than a few minutes to + finish nor did they include a lot of dynamics regarding, for instance, the + creation and destruction of components. So the software system as a whole + was stressed only to a very limited extent. + +* The description of the test scenarios could not be used for running tests + from within an on-target Genode natively. But especially in the context of + the ever growing use of the Sculpt desktop system, this becomes more and + more desirable. The run script mechanism, however, is designed for a + Linux-based host platform remote-controlling the target. + +This brought us to the idea of creating a new component that takes sets of +tests that are described through Genode means only as input. It evaluates one +after the other in a sandbox environment to, eventually, provide the user with +an easy-to-read result list. When it came to choosing the test-description +format, the already existing depot mechanism - as used within Sculpt OS - was +an obvious candidate. + +Depot packages are designed to describe dynamically loadable sub trees for +Genode's component hierarchy, i.e., exactly for what a test scenario should +be in the context of the new component (plus some extra information about +platform requirements and success conditions). Furthermore for managing +package depots, Genode already provides a comprehensive set of well-tested +tools, most noteworthy in this context the depot-query application. This +component makes querying the contents of packages as easy as writing the +package path to a report and awaiting an according blueprint through a ROM +dataspace. That said, the choice of the back-end mechanism was clear and +consequently, we named the new component _depot autopilot_. + + +The depot autopilot +------------------- + +As starting point for the development of the depot autopilot, the depot deploy +component (introduced with the 17.08 release), served us well. Both components +have the purpose of loading Genode scenarios from packages and start them +inside a nicely separated instance of Genode's init runtime. For this reason, +a good part of the depot autopilot configuration interface might look familiar +to you, as shall be depicted by this example: + +! +! +! +! +! +! +! ... +! +! +! +! +! +! +! ... +! +! +! +! +! ... +! +! +! +! + +Like in depot deploy configurations, there are '' nodes to declare, +which scenarios to run and from which packages to load them, there is an +'arch' attribute to select the packages architecture variant, and also the +'' and '' nodes that install common parts of the configuration. +The same goes for the 'runtime' files in the test packages. They are actually +fully compatible with the depot deploy tool: + +! +! +! +! +! +! +! +! +! +! +! ... +! ... +! +! +! +! +! +! +! +! + +But there are also new things as you can see. We will explain them below. + +Roughly outlined, the concepts of the depot autopilot differs in three points +from that of the depot deploy component: + +# The outcome of loaded scenarios is captured and interpreted in order + to support failure detection and analysis, +# Scenarios do not run in parallel but in a given order to mitigate + interfering influences and make results reproducible, and +# Scenarios must be considered to trigger bugs that + may lead to situations where the scenario itself or even the whole system + hangs or crashes. + +The first point is addressed by the new '' tag in _runtime_ files. It +allows for listing events that might trigger during a test and specify the +reaction of the depot autopilot. Currently, two types of events, the '' +event and the '' event, are available, which cover most of the use +cases found in tests of the Genode main repositories. Here's a short example +of how to use them: + +! +! +! +! [init -> test-example] Test succeeded! +! Error: +! +! ... +! + +The content of each '' node is matched continuously against the output of +each LOG client of a test. The first tag whose content matches completely +decides on the tests result through its 'meaning' attribute and causes the +depot autopilot to replace the current test scenario with its successor. In +order to be able to do the matching, the depot autopilot acts as LOG session +server to all components of a test scenario, given that incoming LOG session +requests can be correlated to these components. This behavior leads to the +requirement that the depot autopilot receives the session-label prefix of the +runtime init through the new 'children_label_prefix' attribute of its +configuration: + +! + +As soon as the '' event matching is done, the test output is incorporated +into the autopilot's LOG output, so, later debugging may benefit from it as +well. + +A '' node, on the other hand, causes the autopilot to set a timeout +according to the 'sec' attribute when starting the test. Should the timeout +trigger before the test terminated in another way, it's up to the '' +node to define the test's result and terminate the test. + +The second point, regarding the scheduling of tests, is an easy one. While for +the depot deploy tool, the order of '' nodes is irrelevant, for the +depot autopilot, it defines the order in which tests shall be executed. + +The last point raised the question of what to do when a test gets stuck or, +even worse, compromises the whole system. The former is already handled by +using the above mentioned '' events. The latter, on the other hand, +brings the '' node of the autopilot configuration into play. +With this node, the depot autopilot instance can be equipped with the results +of a previously running instance. This way, the target platform can be +rebooted when a system crash is detected and can proceed with the remaining +tests without losing the information gathered during earlier boot cycles. This +is a short example: + +! +! test-1 ok 0.563 log "Succeeded" +! test-2 failed 20.000 reboot +! ... +! + +The so configured depot-autopilot instance will put the content string of +the '' node at the top of its result list. The values of the +attributes of the tag ('succeeded', 'failed', ...), at the other hand, are +relevant for result statistics. + +Once all tests are evaluated, the depot autopilot outputs a result overview +comprising a list of single test results and some statistical data: + +! --- Finished after 1207.154 sec --- +! +! test-init failed 300.015 timeout 300 sec +! test-init_loop ok 57.228 log "child "test-init_loop" exited wi ..." +! test-ldso ok 1.779 log "[init -> test-ldso] Lib_2_global ..." +! +! succeeded: 2 failed: 1 skipped: 0 + +For a comprehensive description of all the features of the depot autopilot, +please have a look at _repos/gems/src/app/depot_autopilot/README_. + + +The run script +-------------- + +With the depot autopilot component, testing becomes more independent from a +non-Genode host system that executes run scripts. However, we did not intend +to port all existing tests at once nor do we know yet if this is desirable. +In our nightly testing infrastructure, tasks like powering on and off the +target platform or managing the log archives are still bound to Linux and the +time-tested run mechanism. Thus, a mediation between the depot autopilot and +the existing test infrastructure is required. + +[image depot_autopilot_after] + +The solution is the new run script _gems/run/depot_autopilot.run_. This run +script contains a scenario that combines a TAR VFS, a depot query component, +the depot autopilot, and an init runtime resulting in a fully automated meta +test-unit for the run mechanism. Inside the script, you have a convenient and +small set of configuration variables. Normally, you just list the names of the +test packages to evaluate and define platform incompatibilities through +optional 'skip_test_pkg' variables: + +! set avail_test_pkgs { +! test-lx_block +! test-log +! test-tcp_bulk_lwip +! test-mmio +! ... } +! +! set skip_test_pkg(test-lx_block) [expr ![have_spec linux]] +! set skip_test_pkg(test-tcp_bulk_lwip) [expr ![have_spec x86]] + +Together with the '--depot-user' run parameter, the run script composes an +archive of a 'depot' directory containing the correct versions of all needed +packages. As usual, the run modules manage the boot process of the target +platform and the loading of the system image, thereby shipping the depot +archive as Genode boot module. The archive can then be accessed by the depot +query tool on target through the VFS server. The depot autopilot, on the other +hand, gets configured to successively request the exact same packages from the +depot query tool and evaluates them using the init runtime named "dynamic". + +[image depot_autopilot_arch] + +The run script can also handle system crashes. Through the depot autopilot's +log output, it keeps track of every test started and its individual timeout. +If a timeout goes by without the depot autopilot having terminated the current +test, the run script intervenes. As it also records the individual test +results printed by the depot autopilot up to this point, it can now reboot the +system to continue testing. In this follow-up attempt, everything remains the +same with two exceptions. First, the list of tests is pruned by the already +evaluated tests plus the test that caused the reboot, and second, the +'' node of the depot autopilot is used to pass on the +findings of former attempts. This way, the final result list of the depot +autopilot should always be reached and always be complete. + +Another feature of the run script is that it offers an alternate mode for +debugging a single test without having to switch the scenario. Through three +variables you can deactivate all but one test and apply source code changes +without having to rebuild packages. For example: + +! set single_test_pkg "test-libc_vfs" +! set single_test_build { server/ram_fs test/libc_vfs } +! set single_test_modules { ram_fs vfs.lib.so } + +This would only run the _libc_vfs_ test and overlays the package contents with +the _ram_fs_ and _vfs.lib.so_ images, which are freshly built from the local +source repository. + +A more detailed documentation of the run script can be found at +_repos/gems/src/app/depot_autopilot/README_ and inside the run script itself. + + +The test packages +----------------- + +Of course, the depot autopilot isn't worth much without any test packages in +place and all of our tests existed solely in the format expected by the run +tool. But we had the feeling that a significant amount of them should be +portable without great difficulty. One potential limiting factor was that +drivers should not be an individual part of test runtimes as, so far, +restarting drivers causes problems in most cases. Therefore, they must be +loaded by the surrounding system. However, holding drivers for all hardware +available just in case that a test needs them would unnecessarily raise the +scenario's complexity and disturb less demanding but more performance critical +tests. Thus, we decided to let the depot-autopilot run script provide merely +the timer driver as most of the component and unit tests are satisfied with +this. As soon as we want to port tests using other drivers, like the +network-bench suite, a dedicated depot-autopilot run script would be the way +to go. Another noteworthy point is the rather basic feature set for doing +result evaluation with the depot autopilot. As it turned out, most tests +comply with this limitation because tricky success conditions aren't +encouraged by the run mechanism either. + +All in all, we managed to port 75 tests from the repositories 'base', 'os', +'libports', and 'ports' to the depot autopilot during this release cycle. To +avoid redundancy, we removed the corresponding run scripts at the same time. +Each package that represents a test is prefixed with "test-" so you can +quickly receive an overview of all available tests by doing: + +! find /repos -type d -wholename *recipes/pkg/test-* + + +New tooling for obtaining test-coverage metrics +=============================================== + +The [https://gcc.gnu.org/onlinedocs/gcc/Gcov.html - gcov] tool can analyze how +often each source-code line of a test program has been executed. + +On POSIX-like systems the usual steps to use this tool are as follows: + +* When building a test program, special compiler flags add instrumentation code + to the program and generate a gcov "note" file along with each object file. + These note files are needed later on for the final analysis. The test + program is also linked against the gcov library. + +* When the test program is running, it collects coverage data and writes it + into a gcov "data" file for each object file when the program exits. + +* The gcov program uses the source, note, and data files to generate annotated + source-code files with the number of times each line has been executed. + +We ported the gcov tool to Genode to use it with our native (non-libc) Genode +tests and the 'depot_autopilot' run script. It is integrated in the following +way: + +* When adding 'COVERAGE = yes' to a _target.mk_ file of a test program, + the build system adds the gcov-specific compiler flags and links the + program with the gcov library. + +* The generated gcov note files get a symlink in the _build/.../bin/gcov_data_ + directory, which is added to the binary depot package of the test program. + This way, the note files become available within the binary archives of the + depot as used by the "depot_autopilot" system scenario described in + Section [Automated test infrastructure hosted on top of Genode]. + +* When the test program starts, it calls 'env.exec_static_constructors()'. + Each instrumented object file registers itself at the gcov library from a + static constructor (auto-generated with the gcov compiler flags). + +* The gcov library needs the Genode environment to write the collected data to + a file system, so the test program also calls the 'gcov_init()' function + with a reference to the environment. + +* When the test program has finished, it calls 'genode_exit()', which in turn + calls an exit handler registered by the gcov library. The gcov library then + writes the coverage data files to a RAM file system with a path that matches + the path to the note files in the depot in the same file system. + +* When all tests have finished, the depot_autopilot scenario runs the gcov + program, which scans the file system for the collected gcov data files and + writes the final analysis for each file to the log. If the name of the test + package is included in the 'avail_test_src_pkgs' list of the + _depot_autopilot.run_ script, the source and API archives of the test are + added to the depot and the source code is printed together with the counts + and line numbers, otherwise only the counts and line numbers with the text + '/*EOF*/' on each line are printed. + +So far, we enabled the code coverage analysis feature for the 'xml_generator' +test and it can be seen in action by running the _depot_autopilot.run_ script. + + +Static code analysis +==================== + +The static analyzer tool of [https://clang-analyzer.llvm.org] can analyze +source code in C and C++ projects to find bugs at compile time. + +With this tool enabled, Genode users can check and ensure the quality of +Genode components. The tool can be invoked during make invocations and during +the creation of Genode packages. + +For the invocation of _make_ within a Genode build directory, the new +STATIC_ANALYZE variable on the command line will prompt the static analyzer to +run next to the actual build step. + +! STATIC_ANALYZE=1 make -C build/x86_64 KERNEL=... run/... + +For analyzing Genode packages, a new wrapper tool _tool/depot/static_analyze_ +becomes handy. It can be combined with the existing _tool/depot/*_ tools to +take effect, e.g.: + +! tool/depot/static_analyze tool/depot/create /pkg/... + +The results of the static-analyzer tool are generated in the form of html +pages and can be inspected afterwards. The following example output showcases +a run of the static analyzer tool: + +! +! make: Entering directory '../genode/build/x86_64' +! checking library dependencies... +! scan-build: Using '/usr/lib/llvm-6.0/bin/clang' for static analysis +! ... +! +! LINK init +! scan-build: 0 bugs found. +! scan-build: The analyzer encountered problems on some source files. +! scan-build: Preprocessed versions of these sources were deposited in +! '/tmp/scan-build-2018-11-28-111203-20081-1/failures'. + +During our enablement of this feature we used Clang 6.0 on Ubuntu 16.04. The +steps to provide the required tools on Linux are like follows. + +! sudo apt install clang-tools-6.0 +! cd $HOME/bin +! ln -s $(which scan-build-6.0) scan-build + + +Genode as a platform for Mirage-OS unikernels +############################################# + +This year, we collaborated with [https://mirage.io - MirageOS] +at [https://bornhack.dk/bornhack-2018/ - Bornhack] and the MirageOS hack +retreat in Marrakesh to bring unikernel applications to Genode as native +components. MirageOS is an application framework that provides all necessary +OS abstractions for network applications as a library. A MirageOS unikernel +requires a machine operating system to provide a thread of execution, system +time, a network interface, and block storage. Higher-level OS facilities such +as multitasking, TCP/IP, and object storage are implemented as statically +linked libraries written in the same language as the application, OCaml in the +case of MirageOS. Relative to traditional applications that rely on a POSIX +runtime this allows for significant code pruning at compile-time and simple +and precise host environment requirements. + +MirageOS was originally implemented with [https://xenproject.org/ - Xen] as +the machine OS but to port MirageOS to other hypervisors a general sandboxing +middleware library was created. This library is +[https://github.com/Solo5/solo5 - Solo5], which provides the simple +abstractions necessary for running a unikernel as paravirtualized guest or as +sandboxed process of another OS. After investigating both approaches, we chose +to implement a native Solo5 bindings library. This follows our belief that an +operating system can provide process isolation equal to and more efficiently +than hardware assisted virtualization. We share this belief with the unikernel +community and expect it as an +[https://dl.acm.org/citation.cfm?id=3267845 - inevitable trend] +in hosted computing. Furthermore, the burden of developing and maintaining a +unikernel support layer is trivial compared to a virtual machine monitor. + +Building and deploying MirageOS with Genode requires coordination at both +sides. For this reason, Genode target support was added to the Mirage tooling +in the +[https://github.com/mirage/mirage/releases/tag/3.3.0 - 3.3.0 release]. +The Genode platform layer of Solo5 and Mirage differs most from other targets +such as [https://muen.sk/articles.html#mirageos-unikernels - Muen] in that it +is dynamically linked. This allows a Mirage image to remain viable across +Genode API and ABI changes. + +To compile Mirage code into a Genode compatible binary, one may follow the +standard build procedure using the _mirage_ tool just as one would for Muen or +Unix. + +! # See https://mirage.io/docs/ for instructions +! # on building a MirageOS application. +! +! cd mirage-skeleton/tutorial/hello/ +! mirage configure --target genode --dhcp=true +! make depends +! make build + +The output is a Genode program dynamically linked to a Solo5 bindings library. +This library is built using the Genode tool chain and is available as a +[https://depot.genode.org/genodelabs/bin/x86_64/solo5 - depot package]. +Finding a smooth workflow for deploying unikernels with Genode is a topic open +for discussion, but for now, we offer a run scenario for creating a standalone +x86_64 MirageOS boot image with an ethernet driver. + +! # See https://genode.org/documentation/developer-resources/build_system +! # for instructions on preparing the Genode build system. +! +! cp mirage-skeleton/tutorial/hello/hello.genode \ +! genode/build/x86_64/bin/mirage +! cd genode/build/x86_64 +! +! # Build and run a minimal image (serial logging only): +! make run/mirage_net KERNEL=hw +! +! # Build and run an image with a graphical console: +! make run/mirage_pretty KERNEL=hw +! +! # To prepare a USB image for booting real hardware: +! sudo cp var/run/mirage_pretty.iso «your USB stick device file» + +For a minimal network scenario, a Mirage instance requires only a handful of +components, time sources, a network driver, and an I/O hardware-access +multiplexer. + +[image solo5_tcb] A minimum viable Genode system for hosting Mirage, + the trusted computing base is highlighted in red + +A simple demo not withstanding, we can justify Genode as a unikernel hosting +platform by analyzing the complexity of a Genode host system by measuring the +lines of C++ code used to produce such an image. This yields an initial metric +for assessing the security of a system as well as its cost to develop and +maintain. By compiling each component with debugging symbols, parsing binaries +for file paths injected by the compiler, and collecting a sum of the number of +lines found in each file, we get an estimate of the total line count for each +component. Building a set of every source file used in the minimal system, +excluding the unikernel itself, the sum of lines of code does not exceed +50,000. We assert that this is a fraction of the size of any Unix-derived +hosting environment. + +[image solo5_tcb_sloc] + + +Base framework and OS-level infrastructure +########################################## + +Component health monitoring +=========================== + +Scenarios where components are known to sometimes fail call for a mechanism +that continuously checks the health of components and reports anomalies. To +accommodate such use cases, we introduced a low-level health-monitoring +mechanism into the foundation of the framework and made this mechanism +available via init's configuration concept. + +At the lowest level, the parent interface received two new RPC functions. The +'heartbeat_sigh' function allows a child to register a signal handler for +heartbeat signals. The 'heartbeat_response' function allows a child to confirm +its health to the parent. With this basic interface, a parent becomes able to +periodically ask for a life sign from each of its children. + +[image heartbeat] + +Each component installs a heartbeat signal handler during its initialization. +This happens under the hood and is thereby transparent to application code. +The default heartbeat handler invokes the 'heartbeat_response' function at +the parent interface and thereby confirms that the component is still able to +respond to external events. + +With this low-level mechanism in place, we enhanced the init component with the +ability to monitor components hosted on top of init. A global (for this init +instance) heartbeat rate can be configured via a '' +node at the top level of init's configuration. The heartbeat rate can be +specified in milliseconds. If configured, init uses a dedicated timer session +for performing health checks periodically. Each component that hosts a +'' node inside its '' node is monitored. In each period, +init requests heartbeat responses from all monitored children and maintains a +count of outstanding heartbeats for each component. The counter is incremented +in each period and reset whenever the child responds to init's heartbeat +request. Whenever the number of outstanding heartbeats of a child becomes +higher than 1, the child may be in trouble. Init reports this information in +its state report via the new attribute 'skipped_heartbeats="N"' where N +denotes the number of periods since the child became unresponsive. + +Of course, the mechanism won't deliver 100% accuracy. There may be situations +like long-running calculations where long times of unresponsiveness are +expected from a healthy component. Vice versa, in a multi-threaded +application, the crash of a secondary thread may go undetected if the primary +(checked) thread stays responsive. However, in the majority of cases where a +component crashes (page fault, stack overflow), gets stuck in a busy loop, +produces a deadlock, or throws an unhandled exception (abort), the mechanism +nicely reflects the troublesome situation to the outside. + + +Enhanced window-management flexibility +====================================== + +Genode's +[https://genode.org/documentation/release-notes/14.08#New_GUI_architecture - custom GUI architecture] +consists of multiple components with strictly distinguished roles. The central +broker between those components is a low-complexity window-manager (wm) +component, which is complemented by a so-called decorator that defines how +windows look, and a layouter that defines how they behave. Since the layouter +and decorator are sandboxed components without access to the application +window's content nor the user input into the applications, those components +are largely uncritical for the information security of GUI applications. + + +New window layouter +~~~~~~~~~~~~~~~~~~~ + +The window layouter complements the window manager with the policy of how +windows are positioned on screen and how they behave when the user interacts +with window elements like the maximize button or the window title. The current +release replaces the former "floating_window_layouter" with a new +"window_layouter" component that supports the subdivision of screen space into +columns and rows, the concept of layers, and the principle ability to store +window layout information across reboots. + + +Layout rules +------------ + +The window layouter positions windows according to rules defined by the +component's configuration. The rules consist of two parts, the definition of +the screen's layout and the assignment of client windows to the defined parts +of the screen's layout. + +! +! +! +! ...definition of screen layout... +! +! +! ,,, +! +! ... +! + +The '' node can host any number of '' nodes, which partition +the screen horizontally into columns. By default, each column has the same +size. By specifying an optional 'weight' attribute, column sizes can be +weighted relative to one another. The default weight is '1'. Alternatively, +the 'width' of a column can be explicitly specified in pixels. +Each column can host any number of '' nodes, which subdivide the column +vertically. Analogously to columns, rows can be dimensioned via an optional +'weight' attribute or an explicit 'height' in pixels. A '' can, in turn, +contain '' nodes, thereby further subdividing the screen. +Each '' or '' can be used as window-placement target when +equipped with a 'name' attribute. Each name must occur only once within the +'' node. In the following, a named column or row is referred to as +_target_. Each target can host an optional 'layer' attribute. If not +specified, the layer 9999 is assumed. A target with a lower layer overlaps +targets with higher layers. + +The assignment of windows to targets is defined via '' nodes. Each +'' node must be equipped with a 'label', 'label_prefix', or +'label_suffix' attribute, which is used to match window labels. For a given +window, the first matching '' node takes effect. + +Each '' node must have a 'target' attribute that refers to the name +of a column or row. By default, the window is sized to fit the target area. +However, it is possible to position the window relative to the target area by +specifying the 'xpos', 'ypos', 'width', and 'height' attributes together with +the 'maximized="no"' attribute. + +If multiple windows are assigned to the same target area, the order of their +'' rules defines their stacking order. The window with the earliest +'' rule is displayed in front. + + +Dynamic layouts +--------------- + +The window layouter is able to respond to rule changes at runtime. + +By specifying the '' attribute 'rules="rom"', the window layouter +tries to obtain the layout rules from a distinct ROM module. Should the ROM +module not contain valid rules, the '' sub node of the '' comes +into effect. + +Any window-layout change such as the movement of a floating window is realized +as a change of the window-layout rules. To support interactive adjustments of +the window layout, the layouter responds to certain user interactions by +generating new rules by itself in the form of a "rules" report. The generation +of such rules can be enabled via the '' sub node of the configuration: + +! +! +! ... +! + +By feeding back the rules generated by the window layouter into the window +layouter itself via a 'report_rom' service, the window layout becomes +adjustable interactively. As the rules entail the complete state of the +present window layout, it is possible to save/restore the layout state. + + +Dynamic rule-generation mechanism +--------------------------------- + +Whenever a new window appears that solely matches a wildcard '' rule +(one that uses a 'label_prefix' or 'label_suffix'), the layouter generates a +new '' rule with the window's label as 'label' attribute. The +explicitly labeled '' rules appear before any wildcard '' +rules. + +If the user brings a window to front, the window layouter will change the +order of the explicit '' rules such that the window's '' rule +comes first. When moving or resizing a window, the 'xpos', 'ypos', 'width', +and 'height' attribute of the window's assign rule are updated. When +maximizing or unmaximizing a window, the 'maximized' attribute of its +'' rule is toggled. + + +Modularized window system in Sculpt OS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sculpt OS used to come with a package called "wm", which comprised a +ready-to-use composition of window-system components, including the layouter, +a decorator, and the configuration of these components. Even though this +package was convenient for a start, we desire more flexibility. For example, a +user may wish to replace the decorator at runtime or tweak the configuration +of the individual components at runtime. For this reason, we split up the wm +package. For the convenient use in run scripts, we preserved the former wm +package as "motif_wm". + +With the move of the window layouter and window decorators into dedicated +packages, those components can now be combined with the "wm" server at runtime +and restarted/reconfigured/swapped-out independently. To use the window +manager in Sculpt, one must launch the "wm", "window_layouter", and one of the +"motif_decorator" or "themed_decorator" packages. It is possible to replace +one decorator by the other at runtime. + +The window layouter package depends on a so-called "recall_fs" component, +which is a file system used for *remembering the window layouter's state*. +Sculpt comes with a ready-to-use "recall_fs" launcher, which hands out the +directory _/recall_ of the used file system. When the window layouter is +started, a file _/recall/window_layouter/rules_ reflects the current state of +the layouter. It serves two purposes. First, it preserves the window layout +across reboots. Second, it can be edited by the user or (potentially) by other +components. Thereby, the layout policy can be tweaked without limits, and the +rules can simply be swapped out by mere file operations. + + +Network-stack improvements +========================== + +The transition from lwIP as a plugin to the libc library to a plugin to the +VFS library is complete. As a result, POSIX applications can no longer be +linked to lwIP through the 'libc_lwip' library. Instead they need only to link +with the 'libc' library and be configured at runtime to open network +connections through the local VFS. The 'libc' library now features networking +support by default with the actual TCP/IP stack implemented in dynamically +loaded VFS plugins. An example configuration for using lwIP with DHCP follows: + +! +! +! +! +! +! +! +! +! +! +! +! + +The lwIP library has also been updated to the latest 2.1.2 release. + + +Java language runtime +===================== + +OpenJDK support has been added to Genode with release 18.05. At that time, we +were only able to offer support for Java's interpreter mode, because of issues +that arose on ARM platforms (please refer to the release notes of +[https://genode.org/documentation/release-notes/18.05#Java_language_support - Genode 18.05]). +In this mode, byte code is not compiled but interpreted which, of course, +introduces a serious performance penalty. Since 18.05, we worked on enabling +Java's just-in-time compiler (JIT), which translates byte code to actual +machine code. We were able to add support for both ARM and x86 platforms, but +for this release, we stopped short of enabling the compiler by default because +some stability issues remain. + +Please note: OpenJDK has been moved from Genode's base system to the +[https://github.com/genodelabs/genode-world Genode-world] repository and is +now also available as a Genode package. + + +Ada language runtime +==================== + +Genode's support for the Ada programming language is maintained by +[https://componolit.com - Componolit]. With the current release, the +runtime-specific code moved to a +[https://github.com/Componolit/ada-runtime - distinct repository], which is +integrated with Genode as a port. It can be installed by executing the +following command from the base of Genode's source tree: + +! ./tool/ports/prepare_port ada-runtime + + +C language runtime +================== + +The pthread library has been integrated into the libc library with this +release. This has no effect on the semantics of pthreads. Applications simply +do not need to link to this library anymore. This is in contrast to normal +Unix systems that provide a _libpthread_ library. Our POSIX implementation +exists in userspace while supporting the blocking I/O patterns commonly used +in Unix. To do this, we run a "kernel" task within the libc library that +dispatches I/O as application threads are blocked. This task operates on a +secondary stack of the primary thread but is also re-entrant from secondary +threads. Maintaining a common implementation for multi-threaded and +single-threaded applications has been the most practical approach, and merging +pthreads completely into the libc clears the path to improving multithreaded +I/O during the next releases. + + +Improved performance of Zynq network driver +=========================================== + +Since the network driver for the Xilinx Zynq-7000 platform was burdened with +performance issues, it underwent a review to identify and mitigate conceptual +bottlenecks. Although the Gigabit Ethernet MAC (GEM) controller allows +DMA-accelerated transmission and reception of network packets, the driver still +needed to copy each packet from the packet-stream interface to the DMA memory +region. As the latter was additionally attached as uncached memory, the driver +showed a very poor performance (i.e. throughput). + +Fortunately, the design of the packet-stream interface enabled a zero-copy +implementation of the Zynq network driver. Instead of copying each packet from +the bulk buffer of the packet-stream interface to a dedicated DMA memory +region, the allocated packet buffers are now directly handed over to the GEM +for DMA-accelerated packet transmission and reception. In particular, the +driver will allocate up to 1024 packet buffers, depending on the bulk buffer +size provided by its Nic client. + +In addition to the zero-copy implementation, checksum offloading has been +enabled. For outbound packets, the GEM automatically fills in the IP, UDP and +TCP checksum header fields. Inbound packets will be dropped if their checksum +fields are incorrect. Note, that IP checksum offloading is not implemented by +qemu. + +Furthermore, MAC 802.3 pause frames have been enabled. + + +Base API changes +================ + +The framework's base API received two minor improvements. + +First, the _util/reconstructible.h_ utility has a new method called +'conditional', which simplifies the typical use case for 'Constructible' +objects where the constructed/destructed state depends on a configuration +parameter. The method alleviates the need to re-implement the logic manually. + +Second, when creating a secondary entrypoint, the CPU affinity of the +entrypoint is now specified as an argument. + + +Libraries and applications +########################## + +Initial version of the Genode SDK +================================= + +To allow Genode components to be built independently from our build system, we +have added a script to generate a Software Development Kit (SDK) that contains +headers, pre-built libraries, linker scripts, and +[https://www.freedesktop.org/wiki/Software/pkg-config/ - pkg-config] +metadata. The SDK is still experimental and lacks documentation, but has been +shown to be useful when combined with some external Make-based build systems +as well as the Nimble package manager provided with the Nim programming +language. + +The SDK includes support for native Genode services as well a C runtime and +the C++ standard library. + +A simple [https://github.com/libretro/nxengine-libretro/commit/f6ea7cd2e260731a00e20ebe239ccdc6bbf31e50 - example] +of using the SDK within a makefile to build a library: + +! ... +! ifeq ($(platform), genode) +! CC := $(shell pkg-config genode-base --variable=cc) +! CXX := $(shell pkg-config genode-base --variable=cxx) +! LD := $(shell pkg-config genode-base --variable=ld) +! AR := $(shell pkg-config genode-base --variable=ar) -rcs +! CFLAGS += $(shell pkg-config --cflags genode-libc) +! CXXFLAGS += $(shell pkg-config --cflags genode-stdcxx) +! LDFLAGS += $(shell pkg-config --libs genode-lib genode-libc genode-stdcxx) +! endif +! +! %.o: %.c +! $(CC) $(CFLAGS) -c $< -o $@ +! +! %.o: %.cpp +! $(CXX) $(CXXFLAGS) -c $< -o $@ +! +! $(TARGET): $(OBJECTS) +! $(LD) $(OBJECTS) $(LDFLAGS) -o $@ +! ... + +To link a program, the link flags may be obtained with 'pkg-config --libs genode-prg', +for libraries with 'pkg-config --libs genode-lib'. + +In any case, the tools that invoke pkg-config, Make or otherwise, need a +'PKG_CONFIG_PATH' environmental variable set to the location of the +'pkgconfig' directory in the SDK, such as with the following shell command: + +! PKG_CONFIG_PATH=/opt/genode-sdk-x86_64-18.08/pkgconfig make + +It is important to stress that shared libraries must be sourced exclusively +from the SDK or otherwise built by the SDK. It is not possible to mix host +libraries with cross-compiled Genode libraries. In practice it has not been a +problem to build libraries commonly available in the form of shared libraries +as static bundled dependencies, such as libogg, libpng, SQLite, etc. + +In the future we will investigate expanding the scope of the SDK or +complimenting it with add-on SDKs, such as for Qt or SDL. + + +SSH terminal server +=================== + +Up to now, we mostly focused on covering the direct interaction of a user with +the system. Besides a minimal TCP terminal, there was no component to +accommodate remote access. Instead of extending this component to add vital +features like TLS or user authentication, we turned to using an existing +protocol. For this use case the SSH protocol is the most popular choice as +clients and server programs are available for virtually all OSes. + +So with this release, we introduce a component that makes Genode's terminal +session accessible via SSH. It is based on the server implementation of +libssh, which was updated to version 0.8.4. On the Genode side of things, the +component acts as a terminal-session server, to which Terminal clients can +connect. From the outside, users may access a specific terminal session by +logging in via SSH given they provide the proper login credentials. For now, +only the SSH channel 'shell' and 'term' requests have been implemented, i.e., +the server can be used for interactive sessions but not for scp/rsync or other +operations that rely on the 'exec' request. Since the component merely makes +terminal sessions available, the SSH connection needs to be forcefully closed +by issuing the well-known '~.' sequence rather than using '^D' (EOF) which the +underlying terminal session may not handle as expected. + +The following exemplary snippet shows how the component can by configured: + +! +! +! +! +! +! +! +! +! +! +! +! +! +! + +The component is configured to listen on port '2022' for incoming SSH +connections and allows for logging in by using either a password or public +key. The '' configuration is used to link the terminal session to the +SSH login. In this case, the terminal session of the client with the label +'noux-system' may be accessed via the 'root' login. The '' configures the +file system of the component. Besides access to the needed services, like the +real-time clock, a random device, and the TCP/IP stack a file system session +mounted at '/etc' for housing the files required by the component. In +particular, these files are the SSH server's host key and the public key for +'user' login. + +For more information, please read _repos/gems/src/server/ssh_terminal/README_ +and take _repos/gems/run/ssh_terminal.run_ for a ride. + + +Removed networking support from Noux +==================================== + +Support for networking has been removed from the Noux runtime as a side effect +of moving IP stacks from libc plugins into VFS plugins. In practice, Noux has +seldom been used for networking and our recommendation remains that networked +applications should link with the libc library and not use the Noux runtime. + + +New depot packages +================== + +Our work on the Genode-based test-automation framework prompted us to a +package for each individual test. As a nice byproduct, we introduced depot +recipes of all components the tests depend on, and a few more. Thereby, the +following new depot content has become available: + +Source archives for the base-hw microkernel for various platforms: +* src/base-hw-arndale +* src/base-hw-imx53_qsb +* src/base-hw-imx53_qsb_tz +* src/base-hw-odroid_xu +* src/base-hw-panda +* src/base-hw-rpi +* src/base-hw-wand_quad + +Source archives for the Fiasco.OC microkernel for a few platforms: +* src/base-foc-pc +* src/base-foc-arndale +* src/base-foc-pbxa9 + +Source archives of components used by the test scenarios: +* src/nic_bridge +* src/python + +Source archives and package runtimes needed for hosting the gcov tool and +the GCC tool chain: +* src/gmp, src/mpc, src/mpfr +* src/binutils_x86, src/gcc_x86 +* src/gnumake, src/sed, src/tar, src/which +* pkg/gcov +* pkg/noux-build-x86 + + +Platforms +######### + +NOVA microhypervisor +==================== + +Up to now, the NOVA kernel reserved a statically configured part of the system +memory as kernel memory. The configured memory had to be chosen at link time. +However, in the case of Sculpt, the actual target machine and its available +system memory is unknown beforehand, which makes it hard to choose a +kernel-memory amount well suited to a such broad use case. + +We can't lift this structural issue, but we were able to mitigate it to some +degree. The kernel now looks up the overall available system memory at boot +time and allocates the kernel memory depending on three build-time +configuration options. So, the overall kernel memory is still static, but +dimensioned depending on the target machine. + +The three configuration options can be adjusted in the Makefile of the kernel. +CONFIG_MEMORY_BOOT is the amount of kernel memory allocated in the BSS +statically, effectively a link time decision. +CONFIG_MEMORY_DYN_MIN and CONFIG_MEMORY_DYN_PER_MILL configures the dynamic +part of the kernel-memory allocation applied during early kernel boot time. +CONFIG_MEMORY_DYN_MIN is the amount of memory which should be allocated at +least. The default is 28 MiB. CONFIG_MEMORY_DYN_PER_MILL defines the amount of +the system memory in per mill, which should be allocated at most, and has a +default value of 10‰. The overall maximum kernel memory is restricted to ~1G +for 64bit, due to the chosen internal virtual memory layout. + + +Execution on bare hardware (base-hw) +==================================== + +Regarding our own kernel development, the current release increases the +performance on supported ARM platforms. Most significantly, the FPU is enabled +by default on all boards. All components built for ARMv6 and ARMv7a are now +compiled to take advantage of this feature. Therefore, the runtime behaviour +of certain scenarios can change significantly on ARM when using the 18.11 +release. + +Moreover, after recognizing poor memory-access latency on several Cortex-A9 +platforms, a regression in the enabling of the L2 cache got discovered and +fixed. Especially, the i.MX6 based Wandboard Quad cache, clocking and power +settings got tweaked well to achieve reasonable CPU and memory performance. + +This performance-wise line of work will be continued with regard to the next +release.