Final touches of the release notes 14.05

This commit is contained in:
Martin Stein 2014-05-28 15:21:18 +02:00 committed by Norman Feske
parent 2034180d87
commit a7f003edf4
1 changed files with 82 additions and 83 deletions

View File

@ -263,7 +263,7 @@ switching Git branches that use different versions of the same port, the build
system automatically finds the right port version as expected by the currently system automatically finds the right port version as expected by the currently
active branch. active branch.
For step-by-step instructions of how to add a port using the new mechanism, For step-by-step instructions on how to add a port using the new mechanism,
please refer to the updated porting guide: please refer to the updated porting guide:
:Genode Porting Guide: :Genode Porting Guide:
@ -324,18 +324,18 @@ encrypted data from various operating systems. In our case, we want to
use the data from Genode as well as from our current development use the data from Genode as well as from our current development
platform Linux. platform Linux.
In Genode 14.02, we introduced a port of the NetBSD based rumpkernels In Genode 14.02, we introduced a port of the NetBSD based rump kernels
to leverage file-system implementations, e.g., ext2. Beside file to leverage file-system implementations, e.g., ext2. Beside file
systems, NetBSD itself also offers block-level encryption in form of systems, NetBSD itself also offers block-level encryption in form of
its cryptographic disk-driver _cgd(4)_. In line with our roadmap, we its cryptographic disk-driver _cgd(4)_. In line with our roadmap, we
enabled the cryptographic-device driver in our rumpkernels port as a enabled the cryptographic-device driver in our rump-kernels port as a
first step to exploring block-level encryption on Genode. first step to explore block-level encryption on Genode.
:[https://www.netbsd.org/docs/guide/en/chap-cgd.html]: :[https://www.netbsd.org/docs/guide/en/chap-cgd.html]:
NetBSD cryptographic-device driver (CGD) NetBSD cryptographic-device driver (CGD)
The heart of our CGD port is the _rump_cgd_ server, which encapsulates The heart of our CGD port is the _rump_cgd_ server, which encapsulates
the rumpkernels and the cgd device. The server uses a block session to the rump kernels and the cgd device. The server uses a block session to
get access to an existing block device and, in return, provides a get access to an existing block device and, in return, provides a
block session to its client. Each block written or read by the client block session to its client. Each block written or read by the client
is transparently encrypted resp. decrypted by the server with a given is transparently encrypted resp. decrypted by the server with a given
@ -358,7 +358,7 @@ the moment. Note, the server serves only one client as it
transparently encrypts/decrypts one back-end block session. Though transparently encrypts/decrypts one back-end block session. Though
_rump_cgd_ is currently limited with regard to the used cipher and the _rump_cgd_ is currently limited with regard to the used cipher and the
way key input is handled, we plan to extend this way key input is handled, we plan to extend this
rumpkernel-based component step by step in the future. rump-kernel-based component step by step in the future.
If you want to get some hands on with CGD, the first step is to If you want to get some hands on with CGD, the first step is to
prepare a raw encrypted and ext2-formatted partition image by using prepare a raw encrypted and ext2-formatted partition image by using
@ -407,10 +407,10 @@ Currently, the key to access the cryptographically secured device must
be specified before using the device. Implementing a mechanism which be specified before using the device. Implementing a mechanism which
asks for the key on the first attempt is in the works. asks for the key on the first attempt is in the works.
By using the rumpkernels and the cryptographic-device driver, we are By using the rump kernels and the cryptographic-device driver, we are
able to use block-level encryption on Genode and on Linux. able to use block-level encryption on Genode and on Linux.
In Linux case, we depend on _rumprun_, which can In Linux case, we depend on _rumprun_, which can
run unmodified NetBSD userland tools on top of the rumpkernels to run unmodified NetBSD userland tools on top of the rump kernels to
manage the cgd device. To ease this task, we provide the manage the cgd device. To ease this task, we provide the
aforementioned _rump_ wrapper script. aforementioned _rump_ wrapper script.
@ -473,7 +473,7 @@ an issue if the shell, which is used, is maintaining a history of
executed commands. executed commands.
For the sake of completeness let us put all examples together by creating an For the sake of completeness let us put all examples together by creating an
encrypted Ext2 image that will contain all files of Genode's _demo_ encrypted ext2 image that will contain all files of Genode's _demo_
scenario: scenario:
! dd if=/dev/urandom of=/tmp/demo.img bs=1M count=16 ! dd if=/dev/urandom of=/tmp/demo.img bs=1M count=16
@ -491,12 +491,12 @@ this tool can be obtained by running:
! rump -h ! rump -h
Since _tool/rump_ just utilizes the rumpkernels running on the host Since _tool/rump_ just utilizes the rump kernels running on the host
system to do its duty, there is a script called _tool/rump_cgdconf_ system to do its duty, there is a script called _tool/rump_cgdconf_
that extracts the key from a 'cgdconfig(8)' generated configuration that extracts the key from a 'cgdconfig(8)' generated configuration
file and is also able to generate such a file from a given key. file and is also able to generate such a file from a given key.
Thereby, we try accommodate the interoperability between the general Thereby, we try to accommodate the interoperability between the general
rumpkernels-based tools and the _rump_cgd_ server used on Genode. rump-kernel-based tools and the _rump_cgd_ server used on Genode.
Per-process virtual file systems Per-process virtual file systems
@ -504,7 +504,7 @@ Per-process virtual file systems
Our C runtime served us quite well over the years. At its core, it has a Our C runtime served us quite well over the years. At its core, it has a
flexible plugin architecture that allows us to combine different back ends flexible plugin architecture that allows us to combine different back ends
such the lwIP socket API (using libc_lwip_nic_dhcp), using LOG as stdout via such as the lwIP socket API (using libc_lwip_nic_dhcp), using LOG as stdout
(via libc_log), or using a ROM dataspace as a file (via libc_rom). Recently (via libc_log), or using a ROM dataspace as a file (via libc_rom). Recently
however, the original design has started to show its limitations: however, the original design has started to show its limitations:
@ -551,12 +551,12 @@ types. It is stable and complete enough to run our tool chain to build Genode
on Genode. Wouldn't it be a good idea to reuse the Noux VFS for the normal on Genode. Wouldn't it be a good idea to reuse the Noux VFS for the normal
libc? With the current release cycle, we pursued this line of thoughts. libc? With the current release cycle, we pursued this line of thoughts.
The first step was transplanting the VFS code from the Noux runtime to an The first step was transplanting the VFS code from the Noux runtime to a
free-standing library. The most substantial free-standing library. The most substantial
change was the decoupling of the VFS interfaces from the types provided by change was the decoupling of the VFS interfaces from the types provided by
Noux. All those types had been moved to the VFS library. In the process Noux. All those types had been moved to the VFS library. In the process
of reshaping the Noux VFS into a library, several existing pseudo file systems of reshaping the Noux VFS into a library, several existing pseudo file systems
received a welcome cleanup, and some new ones were added. In particular, received a welcome clean-up, and some new ones were added. In particular,
there is a new "log" file system for writing data to a LOG session, a "rom" there is a new "log" file system for writing data to a LOG session, a "rom"
file system for reading ROM modules, and an "inline" file system for file system for reading ROM modules, and an "inline" file system for
reading data defined within the VFS configuration. reading data defined within the VFS configuration.
@ -611,7 +611,7 @@ archive (which is obtained from a ROM module named "website.tar"). There
are two pseudo devices "/dev/log" and "/dev/null", to which the are two pseudo devices "/dev/log" and "/dev/null", to which the
"stdin", "stdout", and "stderr" attributes refer. The "log" file system "stdin", "stdout", and "stderr" attributes refer. The "log" file system
consists of a single node that represents a LOG session. The web server consists of a single node that represents a LOG session. The web server
configuration is supplied inline as part of the config. (Btw, you can configuration is supplied inline as part of the configuration. (BTW, you can
try out a very similar scenario using the 'ports/genode_org.run' script) try out a very similar scenario using the 'ports/genode_org.run' script)
The VFS implementation resides at 'os/include/vfs/'. This is where you The VFS implementation resides at 'os/include/vfs/'. This is where you
@ -692,11 +692,11 @@ the following minor changes.
:Framebuffer session: :Framebuffer session:
We simplified the framebuffer-session interface by removing the We simplified the framebuffer-session interface by removing the
'Framebuffer::Session::release()' function. This step makes the mode-change 'Framebuffer::Session::release()' method. This step makes the mode-change
protocol consistent with the way the ROM-session interface handles protocol consistent with the way the ROM-session interface handles
ROM-module changes. That is, the client acknowledges the release of its ROM-module changes. That is, the client acknowledges the release of its
current dataspace by requesting a new dataspace via the current dataspace by requesting a new dataspace via the
'Framebuffer::Session::dataspace()' function. 'Framebuffer::Session::dataspace()' method.
To enable framebuffer clients to synchronize their operations with the To enable framebuffer clients to synchronize their operations with the
display frequency, the session interface received the new 'sync_sigh' display frequency, the session interface received the new 'sync_sigh'
@ -721,8 +721,8 @@ Ported 3rd-party software
VirtualBox on NOVA VirtualBox on NOVA
================== ==================
With Genode 14.02, we successfully executed more than seven With Genode 14.02, we successfully executed more than seven
guest-operating systemsa, including MS Windows 7, on top of Genode/NOVA. Based guest-operating systems, including MS Windows 7, on top of Genode/NOVA. Based
on this proof of concept, we invested significant efforts to stabilize on this proof of concept, we invested significant efforts to stabilize
and extend our port of VirtualBox during the last three months. We and extend our port of VirtualBox during the last three months. We
also paid attention to user friendliness (i.e., features) by enabling also paid attention to user friendliness (i.e., features) by enabling
@ -732,7 +732,7 @@ Regarding stability, one issue we encountered has been occasional
synchronization problems during the early VMM bootstrap phase. Several synchronization problems during the early VMM bootstrap phase. Several
internal threads in the VMM are started concurrently, like the timer internal threads in the VMM are started concurrently, like the timer
thread, emulation thread (EMT), virtual CPU handler thread, hard-disk thread, emulation thread (EMT), virtual CPU handler thread, hard-disk
thread, and user-interface frontend thread. Some of these threads are thread, and user-interface front-end thread. Some of these threads are
favoured regarding their execution over others according to their favoured regarding their execution over others according to their
importance. VirtualBox expresses this by host-specific mechanisms like importance. VirtualBox expresses this by host-specific mechanisms like
priorities and nice levels of the host operating system. For Genode, priorities and nice levels of the host operating system. For Genode,
@ -779,7 +779,7 @@ process-absolute base and a base-local offset. These structures can
thereby be shared over different protection domains where the base thereby be shared over different protection domains where the base
pointer typically differs (shared memory attached at different pointer typically differs (shared memory attached at different
addresses). For the Genode port, we actually don't need this shared addresses). For the Genode port, we actually don't need this shared
memory features, however we had to recognize that the space for the memory features, however, we had to recognize that the space for the
offset value is a signed integer (int32_t). On a 64bit host, this offset value is a signed integer (int32_t). On a 64bit host, this
feature caused trouble if the distance of two memory pointers was feature caused trouble if the distance of two memory pointers was
larger than 31 bit (2 GiB). Fortunately, each memory-allocation larger than 31 bit (2 GiB). Fortunately, each memory-allocation
@ -852,16 +852,16 @@ USB 3.0 for x86-based platforms
=============================== ===============================
Having support for USB 3.0 or XHCI host controllers on the Exynos 5 platform Having support for USB 3.0 or XHCI host controllers on the Exynos 5 platform
since mid 2013, we decided it was about time to enable USB 3.0 on x86 platforms. since mid 2013, we decided it was about time to enable USB 3.0 on x86
Because XHCI is a standardized interface, which is also exposed by the Exynos 5 platforms. Because XHCI is a standardized interface, which is also exposed by
host controller, the enablement was relatively straight forward. The major the Exynos 5 host controller, the enablement was relatively straight forward.
open issue for x86 was the missing connection of the USB controller to the PCI The major open issue for x86 was the missing connection of the USB controller
bus. For this, we ported the XHCI-PCI part from Linux and connected it with the to the PCI bus. For this, we ported the XHCI-PCI part from Linux and connected
internal-PCI driver of our _dde_linux_ environment. This step enabled basic XHCI it with the internal-PCI driver of our _dde_linux_ environment. This step
support for x86 platforms. Unfortunately, there seems not to be a single USB 3.0 enabled basic XHCI support for x86 platforms. Unfortunately, there seems not
controller without quirks. Thus, we tested some PCI cards and notebooks and to be a single USB 3.0 controller without quirks. Thus, we tested some PCI
added controller-specific quirks as needed. These quirks may not cover all current cards and notebooks and added controller-specific quirks as needed. These
production chips though. quirks may not cover all current production chips though.
We also enabled and tested the HID, storage, and network profiles for USB 3.0, We also enabled and tested the HID, storage, and network profiles for USB 3.0,
where the supported network chip is, as for Exynos 5, the ASIX AX88179 where the supported network chip is, as for Exynos 5, the ASIX AX88179
@ -879,32 +879,32 @@ Multi-processor support
When we started to contemplate the support for symmetric multiprocessing When we started to contemplate the support for symmetric multiprocessing
within the base-hw kernel, a plenty of fresh influences on this subject within the base-hw kernel, a plenty of fresh influences on this subject
floated around in our minds. Most notably, the NOVA port of Genode recently obtained floated around in our minds. Most notably, the NOVA port of Genode recently
SMP support in the course of a prototypically comparison of different models obtained SMP support in the course of a prototypically comparison of different
for inter-processor communication. In addition to the very insightful models for inter-processor communication. In addition to the very insightful
conclusions of this evaluation, our knowledge about other kernel projects and conclusions of this evaluation, our knowledge about other kernel projects and
their way to SMP went in. In general, this showed us that the subject - if their way to SMP went in. In general, this showed us that the subject - if
addressed too ambitious - may boast lots of complex stabilization problems, and addressed too ambitious - may boast lots of complex stabilization problems, and
coping with them easily draws down SMP efficiency in the aftermath. coping with them easily draws down SMP efficiency in the aftermath.
Against this backdrop, we decided - as so often in the evolution of the base-hw kernel - to Against this backdrop, we decided - as so often in the evolution of the base-hw
pick the easiest-to-reach and easiest-to-grasp solution first with preliminary kernel - to pick the easiest-to-reach and easiest-to-grasp solution first with
disregard to secondary requirements like scalability. As the base-hw kernel preliminary disregard to secondary requirements like scalability. As the
is single-threaded on uniprocessor systems, it was obvious to maintain base-hw kernel is single-threaded on uniprocessor systems, it was obvious to
one kernel thread per SMP processor and, as far as possible, let them all maintain one kernel thread per SMP processor and, as far as possible, let them
work in a similar way. To moreover keep the code base of the kernel as all work in a similar way. To moreover keep the code base of the kernel as
unmodified as possible, while introducing SMP, access to kernel objects get fully unmodified as possible while introducing SMP, access to kernel objects get
serialized by one global spin lock. Therewith, we had a very minimalistic fully serialized by one global spin lock. Therewith, we had a very minimalistic
starting point for what shall emerge on the kernel side. starting point for what shall emerge on the kernel side.
Likewise, we started with a feature set narrowed to only the essentials on the Likewise, we started with a feature set narrowed to only the essentials on the
user side, prohibiting thread migration, any kind of inter-processor user side, prohibiting thread migration, any kind of inter-processor
communication, and also the unmapping of dataspaces, as this would have communication, and also the unmapping of dataspaces, as this would have
raised the need for synchronization of TLBs. While thread migration raised the need for synchronization of TLBs. While thread migration
is still an open issue, means of inter-processor communication and MMU is still an open issue, means of inter-processor communication and TLB
synchronization were added successively after having the basic work stable. synchronization were added successively after having the basics work stable.
First of all, the start-up code of the kernel had to be adapted. The simple First of all, the startup code of the kernel had to be adapted. The simple
uniprocessor instantiation was split into three phases: At the very beginning, uniprocessor instantiation was split into three phases: At the very beginning,
the primary processor runs alone and initializes everything that is needed for the primary processor runs alone and initializes everything that is needed for
calling a simple C function, which then prepares and performs the activation of calling a simple C function, which then prepares and performs the activation of
@ -917,58 +917,59 @@ accesses at this level can't be synchronized. Therefore, the first
initialization phase prepares everything in such a way, that the second phase initialization phase prepares everything in such a way, that the second phase
can be done without writing to global memory. As soon as the processors are can be done without writing to global memory. As soon as the processors are
done with the second phase, they acquire the global spin lock that protects all done with the second phase, they acquire the global spin lock that protects all
kernel data. This way, all processors consecutively pass the third initialization kernel data. This way, all processors consecutively pass the third
phase that handles all remaining drivers and kernel objects. This is the last initialization phase that handles all remaining drivers and kernel objects.
time the primary processor plays a special role by doing all the work that This is the last time the primary processor plays a special role by doing all
isn't related to processor-local resources. Afterwards the processors can the work that isn't related to processor-local resources. Afterwards the
proceed to the main function that is called on every kernel pass. processors can proceed to the main function that is called on every kernel
pass.
Another main challenge was the mode-transition assembler code path that Another main challenge was the mode-transition assembler code path that
performs both performs both
transitions from a processor exception to the call of the kernel-main function transitions from a processor exception to the call of the kernel-main function
and from the return of the kernel-main function back to the user and from the return of the kernel-main function back to the user
space. As this can't be synchronized, all corresponding data must be provided per space. As this can't be synchronized, all corresponding data must be provided
processor. This brought in additional offset calculations, which were a little per processor. This brought in additional offset calculations, which were a
tricky to achieve without polluting the user state. But after we managed little tricky to achieve without polluting the user state. But after we managed
to do so, the kernel was already able to handle user threads on different processors to do so, the kernel was already able to handle user threads on different
as long as they didn't interact with each other. processors as long as they didn't interact with each other.
When it came to synchronous and asynchronous inter-processor communication, When it came to synchronous and asynchronous inter-processor communication,
we enjoyed a big benefit of our approach. Due to fully serializing all kernel we enjoyed a big benefit of our approach. Due to fully serializing all kernel
code paths, none of the communication models had changed with SMP. Thanks to the code paths, none of the communication models had changed with SMP. Thanks to
cache coherence of ARM hardware, even shared memory amongst processors isn't a the cache coherence of ARM hardware, even shared memory amongst processors
problem. The only difference is that now a processor may change the schedule of isn't a problem. The only difference is that now a processor may change the
another processor by unblocking one of its threads on communication feedback. schedule of another processor by unblocking one of its threads on communication
This may rescind the current scheduling choice of the other processor. To feedback. This may rescind the current scheduling choice of the other
avoid lags in this case, we let the unaware processor trap into an IPI. As the processor. To avoid lags in this case, we let the unaware processor trap into
IPI sender doesn't have to wait for an answer, this isn't a big deal neither an IPI. As the IPI sender doesn't have to wait for an answer, this isn't a big
conceptually nor according to performance. deal neither conceptually nor according to performance.
The last problem we had to solve for common Genode scenarios was the coherency The last problem we had to solve for common Genode scenarios was the coherency
of the TLB caches. When unmapping a dataspace at one processor, the corresponding of the TLBs. When unmapping a dataspace at one processor, the corresponding
TLB entries must be invalidated on all processors, which - at least on TLB entries must be invalidated on all processors, which - at least on
ARM systems - can be done processor-local only. Thus we needed a protocol to ARM systems - can be done processor-local only. Thus we needed a protocol to
broadcast the operation. First, we decided to leave it to the user land to broadcast the operation. First, we decided to leave it to the user land to
reserve a worker thread at each processor and synchronize between them. This reserve a worker thread at each processor and synchronize between them. This
way, we didn't have to modify the kernel back end that was responsible for way, we didn't have to modify the kernel back end that was responsible for
updating the caches back in uniprocessor mode. Unfortunately, the revised memory updating the caches back in uniprocessor mode. Unfortunately, the revised
management explained in Section [Sparsely populated core address space] memory management explained in Section [Sparsely populated core address space]
relies on unmap operations at the startup of user threads, which led us into a relies on unmap operations at the startup of user threads, which led us into a
chicken-and-egg situation. Therefore, the broadcasting was moved from the user land chicken-and-egg situation. Therefore, the broadcasting was moved from the
into the kernel. If a user thread now asks the kernel to update TLB caches, the userland into the kernel. If a user thread now asks the kernel to update the
kernel blocks the thread and informs all processors. The last processor that TLBs, the kernel blocks the thread and informs all processors. The last
completes the operation unblocks the user thread. If this unblocking processor that completes the operation unblocks the user thread. If this
happens remotely, the kernel acts exactly the same as described above in the unblocking happens remotely, the kernel acts exactly the same as described
user-communication model. This way, the kernel never blocks itself but only the above in the user-communication model. This way, the kernel never blocks itself
thread that requests an MMU update. but only the thread that requests a TLB update.
Given that all kernel operations are lightweight non-blocking operations, we Given that all kernel operations are lightweight non-blocking operations, we
assume that there is little contention for the global kernel lock. So we hope assume that there is little contention for the global kernel lock. So we hope
that the simple SMP model will perform well for the foreseeable future where that the simple SMP model will perform well for the foreseeable future where
we will have to accommodate a handful of processors. If this assumption turns we will have to accommodate only a handful of processors. If this assumption
out to be wrong, or if the kernel should scale to large-scale SMP systems one turns out to be wrong, or if the kernel should scale to large-scale SMP
day, we still have the choice to advance to a more sophisticated approach systems one day, we still have the choice to advance to a more sophisticated
without much backpedaling. approach without much backpedaling.
Sparsely populated core address space Sparsely populated core address space
@ -1048,7 +1049,7 @@ On reschedule, the context implicitly returns to the lending thread.
Additionally, a thread may request an explicit reschedule in order to Additionally, a thread may request an explicit reschedule in order to
return a lent scheduling context obtained from another thread. return a lent scheduling context obtained from another thread.
The current solution enables Genode to use of NOVA's static priorities. The current solution enables Genode to make use of NOVA's static priorities.
Another unrelated NOVA extension is the ability for a thread to yield Another unrelated NOVA extension is the ability for a thread to yield
the CPU. The context gets enqueued at the end of the run queue without the CPU. The context gets enqueued at the end of the run queue without
@ -1098,7 +1099,7 @@ the 'LIBS' declaration. The tool can be invoked by referring to
For an example of using custom host tools, please refer to the mupdf package For an example of using custom host tools, please refer to the mupdf package
found within the libports repository. During the build of the mupdf library, found within the libports repository. During the build of the mupdf library,
two custom tools fontdump and cmapdump are invoked. The tools are built via two custom tools fontdump and cmapdump are invoked. The tools are built via
the _lib/mk/mupdf_host_tools.mk_ library description file. The actual mupdf the _lib/mk/mupdf_host_tools.mk_ library-description file. The actual mupdf
library (_lib/mk/mupdf.mk_) has the pseudo library 'mupdf_host_tools' listed library (_lib/mk/mupdf.mk_) has the pseudo library 'mupdf_host_tools' listed
in its 'LIBS' declaration and refers to the tools relative to in its 'LIBS' declaration and refers to the tools relative to
'$(BUILD_BASE_DIR)'. '$(BUILD_BASE_DIR)'.
@ -1110,7 +1111,7 @@ Rump-kernel tools
During our work on porting the cryptographic-device driver to Genode, During our work on porting the cryptographic-device driver to Genode,
we identified the need for tools to process block-device and we identified the need for tools to process block-device and
file-system images on our development machines. For this purpose, we file-system images on our development machines. For this purpose, we
added the rumpkernel-based tools, which are used for preparing and added the rump-kernel-based tools, which are used for preparing and
populating disk images as well as creating cgd(4)-based cryptographic populating disk images as well as creating cgd(4)-based cryptographic
disk devices. disk devices.
@ -1120,5 +1121,3 @@ be installed via _tool/tool_chain_rump install_ to the default install
location _/usr/local/genode-rump_. As mentioned in location _/usr/local/genode-rump_. As mentioned in
[Block-level encryption using CGD], instead of using the tools [Block-level encryption using CGD], instead of using the tools
directly, we added the wrapper shell script _tool/rump_. directly, we added the wrapper shell script _tool/rump_.