ehmry/genode
ehmry/genode
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

depot: move versions to subdirectory

Browse Source 
This patch changes the depot layout such that each archive is
represented as a directory that contains the versions of the archive as
subdirectories.

Issue #2610
This commit is contained in:
Norman Feske 2017-12-11 17:19:03 +01:00 committed by Christian Helmuth
parent 2d041f0e9c
commit a52541de18
14 changed files with 218 additions and 188 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

78
doc/depot.txt
View File

@ -88,10 +88,10 @@ different kinds of content, each in a tailored and simple form. To avoid the
clash of the notions of the common meaning of a "package", we speak of clash of the notions of the common meaning of a "package", we speak of
"archives" as the basic unit of delivery. The following subsections introduce "archives" as the basic unit of delivery. The following subsections introduce
the different categories. the different categories.
Archives are named with their version as suffix, appended via a dash. The Archives are named with their version as suffix, appended via a slash. The
suffix is maintained by the author of the archive. The recommended naming suffix is maintained by the author of the archive. The recommended naming
scheme is the use of the release date as version suffix, e.g., scheme is the use of the release date as version suffix, e.g.,
'report_rom-2017-05-14'. 'report_rom/2017-05-14'.
Raw-data archives Raw-data archives
@ -141,9 +141,9 @@ called 'used_apis', which contains a list of API-archive names with each
name on a separate line. For example, the 'used_apis' file of the 'report_rom' name on a separate line. For example, the 'used_apis' file of the 'report_rom'
source archive looks as follows: source archive looks as follows:
! base-2017-05-14 ! base/2017-05-14
! os-2017-05-13 ! os/2017-05-13
! report_session-2017-05-13 ! report_session/2017-05-13
The 'used_apis' file declares the APIs needed to incorporate into the build The 'used_apis' file declares the APIs needed to incorporate into the build
process when building the source archive. Hence, they represent _build-time_ process when building the source archive. Hence, they represent _build-time_
@ -151,9 +151,9 @@ _dependencies_ on the specific API versions.
A source archive may be equipped with a top-level file called 'api' containing A source archive may be equipped with a top-level file called 'api' containing
the name of exactly one API archive. If present, it declares that the source the name of exactly one API archive. If present, it declares that the source
archive _implements_ the specified API. For example, the 'libc-2017-05-14' archive _implements_ the specified API. For example, the 'libc/2017-05-14'
source archive contains the actual source code of the libc and libm as well as source archive contains the actual source code of the libc and libm as well as
an 'api' file with the content 'libc-2017-04-13'. The latter refers to the API an 'api' file with the content 'libc/2017-04-13'. The latter refers to the API
implemented by this version of the libc source package (note the differing implemented by this version of the libc source package (note the differing
versions of the API and source archives) versions of the API and source archives)
@ -182,13 +182,13 @@ Package archive
A package archive contains an 'archives' file with a list of archive names A package archive contains an 'archives' file with a list of archive names
that belong together at runtime. Each listed archive appears on a separate line. that belong together at runtime. Each listed archive appears on a separate line.
For example, the 'archives' file of the package archive for the window For example, the 'archives' file of the package archive for the window
manager 'wm-2017-05-31' looks as follows: manager 'wm/2017-05-31' looks as follows:
! genodelabs/raw/wm-2017-05-31 ! genodelabs/raw/wm/2017-05-31
! genodelabs/src/wm-2017-05-31 ! genodelabs/src/wm/2017-05-31
! genodelabs/src/report_rom-2017-05-31 ! genodelabs/src/report_rom/2017-05-31
! genodelabs/src/decorator-2017-05-31 ! genodelabs/src/decorator/2017-05-31
! genodelabs/src/floating_window_layouter-2017-05-31 ! genodelabs/src/floating_window_layouter/2017-05-31
In contrast to the list of 'used_apis' of a source archive, the content of In contrast to the list of 'used_apis' of a source archive, the content of
the 'archives' file denotes the origin of the respective archives the 'archives' file denotes the origin of the respective archives
@ -216,11 +216,11 @@ is structured as follows:
! <user>/pubkey ! <user>/pubkey
! <user>/download ! <user>/download
! <user>/src/<name>-<version>/ ! <user>/src/<name>/<version>/
! <user>/api/<name>-<version>/ ! <user>/api/<name>/<version>/
! <user>/raw/<name>-<version>/ ! <user>/raw/<name>/<version>/
! <user>/pkg/<name>-<version>/ ! <user>/pkg/<name>/<version>/
! <user>/bin/<arch>/<src-name>-<src-version>/ ! <user>/bin/<arch>/<src-name>/<src-version>/
The <user> stands for the origin of the contained archives. For example, the The <user> stands for the origin of the contained archives. For example, the
official archives provided by Genode Labs reside in a _genodelabs/_ official archives provided by Genode Labs reside in a _genodelabs/_
@ -231,7 +231,7 @@ from the user. The file 'download' specifies the download location as an URL.
Subsuming archives in a subdirectory that correspond to their the origin Subsuming archives in a subdirectory that correspond to their the origin
(user) serves two purposes. First, it provides a user-local name space for (user) serves two purposes. First, it provides a user-local name space for
versioning archives. E.g., there might be two versions of a versioning archives. E.g., there might be two versions of a
'nitpicker-2017-04-15' source archive, one by "genodelabs" and one by 'nitpicker/2017-04-15' source archive, one by "genodelabs" and one by
"nfeske". However, since each version resides under its origin's subdirectory, "nfeske". However, since each version resides under its origin's subdirectory,
version-naming conflicts between different origins cannot happen. Second, by version-naming conflicts between different origins cannot happen. Second, by
allowing multiple archive origins in the depot side-by-side, package archives allowing multiple archive origins in the depot side-by-side, package archives
@ -282,7 +282,7 @@ corresponding user subdirectory must contain two files:
If both the public key and the download locations are defined, the download If both the public key and the download locations are defined, the download
tool can be used as follows: tool can be used as follows:
! ./tool/depot/download genodelabs/src/zlib-2017-05-31 ! ./tool/depot/download genodelabs/src/zlib/2017-05-31
The tool automatically downloads the specified archives and their The tool automatically downloads the specified archives and their
dependencies. For example, as the zlib depends on the libc API, the libc API dependencies. For example, as the zlib depends on the libc API, the libc API
@ -294,7 +294,7 @@ all binary archives for the 32-bit x86 architecture. Downloaded binary
archives are always accompanied with their corresponding source and used API archives are always accompanied with their corresponding source and used API
archives. archives.
! ./tool/depot/download genodelabs/pkg/x86_32/wm-2017-05-31 ! ./tool/depot/download genodelabs/pkg/x86_32/wm/2017-05-31
Archive content is not downloaded directly to the depot. Instead, the Archive content is not downloaded directly to the depot. Instead, the
individual archives and signature files are downloaded to a quarantine area in individual archives and signature files are downloaded to a quarantine area in
@ -321,14 +321,14 @@ CPU architecture. For example, the following command builds the 'zlib'
library for the 64-bit x86 architecture. It executes four concurrent jobs library for the 64-bit x86 architecture. It executes four concurrent jobs
during the build process. during the build process.
! ./tool/depot/build genodelabs/bin/x86_64/zlib-2017-05-31 -j4 ! ./tool/depot/build genodelabs/bin/x86_64/zlib/2017-05-31 -j4
Note that the command expects a specific version of the source archive as Note that the command expects a specific version of the source archive as
argument. The depot may contain several versions. So the user has to decide, argument. The depot may contain several versions. So the user has to decide,
which one to build. which one to build.
After the tool is finished, the freshly built binary archive can be found in After the tool is finished, the freshly built binary archive can be found in
the depot within the _genodelabs/bin/<arch>/<src>-<version>/_ subdirectory. the depot within the _genodelabs/bin/<arch>/<src>/<version>/_ subdirectory.
Only the final result of the built process is preserved. In the example above, Only the final result of the built process is preserved. In the example above,
that would be the _zlib.lib.so_ library. that would be the _zlib.lib.so_ library.
@ -358,28 +358,28 @@ be present in the key ring of your GNU privacy guard.
To publish archives, one needs to specify the specific version to publish. To publish archives, one needs to specify the specific version to publish.
For example: For example:
! ./tool/depot/publish <you>/pkg/wm-2017-05-31 ! ./tool/depot/publish <you>/pkg/wm/2017-05-31
The command checks that the specified archive and all dependencies are present The command checks that the specified archive and all dependencies are present
in the depot. It then proceeds with the archiving and signing operations. For in the depot. It then proceeds with the archiving and signing operations. For
the latter, the pass phrase for your private key will be requested. The the latter, the pass phrase for your private key will be requested. The
publish tool prints the information about the processed archives, e.g.: publish tool prints the information about the processed archives, e.g.:
! publish /.../genode/public/<you>/pkg/wm-2017-05-30.tgz ! publish /.../public/<you>/pkg/wm/2017-05-30.tar.xz
! publish /.../genode/public/<you>/src/decorator-2017-05-30.tgz ! publish /.../public/<you>/src/decorator/2017-05-30.tar.xz
! publish /.../genode/public/<you>/src/floating_window_layouter-2017-05-30.tgz ! publish /.../public/<you>/src/floating_window_layouter/2017-05-30.tar.xz
! publish /.../genode/public/<you>/src/report_rom-2017-05-30.tgz ! publish /.../public/<you>/src/report_rom/2017-05-30.tar.xz
! publish /.../genode/public/<you>/src/wm-2017-05-30.tgz ! publish /.../public/<you>/src/wm/2017-05-30.tar.xz
! publish /.../genode/public/<you>/raw/wm-2017-05-30.tgz ! publish /.../public/<you>/raw/wm/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/base-2017-05-30.tgz ! publish /.../public/<you>/api/base/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/framebuffer_session-2017-05-30.tgz ! publish /.../public/<you>/api/framebuffer_session/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/gems-2017-05-30.tgz ! publish /.../public/<you>/api/gems/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/input_session-2017-05-30.tgz ! publish /.../public/<you>/api/input_session/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/nitpicker_gfx-2017-04-24.tgz ! publish /.../public/<you>/api/nitpicker_gfx/2017-04-24.tar.xz
! publish /.../genode/public/<you>/api/nitpicker_session-2017-05-30.tgz ! publish /.../public/<you>/api/nitpicker_session/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/os-2017-05-30.tgz ! publish /.../public/<you>/api/os/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/report_session-2017-05-30.tgz ! publish /.../public/<you>/api/report_session/2017-05-30.tar.xz
! publish /.../genode/public/<you>/api/scout_gfx-2017-04-24.tgz ! publish /.../public/<you>/api/scout_gfx/2017-04-24.tar.xz
According to the output, the tool populates a directory called _public/_ According to the output, the tool populates a directory called _public/_
at the root of the Genode source tree with the to-be-published archives. at the root of the Genode source tree with the to-be-published archives.

2
repos/gems/run/depot_query.run
View File

@ -12,7 +12,7 @@ create_tar_from_depot_binaries [run_dir]/genode/depot.tar \
genodelabs/pkg/test-fs_report genodelabs/pkg/test-fs_report
proc query_pkg {} { proc query_pkg {} {
return [_versioned_depot_archive_name genodelabs pkg test-fs_report] } return test-fs_report/[_current_depot_archive_version pkg test-fs_report] }
install_config { install_config {
<config> <config>

11
repos/gems/src/app/depot_query/main.cc
View File

@ -30,6 +30,7 @@ struct Depot_query::Archive
typedef String<100> Path; typedef String<100> Path;
typedef String<64> User; typedef String<64> User;
typedef String<80> Name; typedef String<80> Name;
typedef String<40> Version;
enum Type { PKG, RAW, SRC }; enum Type { PKG, RAW, SRC };
@ -89,7 +90,8 @@ struct Depot_query::Archive
throw Unknown_archive_type(); throw Unknown_archive_type();
} }
static Name name(Path const &path) { return _path_element<Name>(path, 2); } static Name name (Path const &path) { return _path_element<Name>(path, 2); }
static Version version(Path const &path) { return _path_element<Name>(path, 3); }
}; };
@ -210,9 +212,10 @@ Depot_query::Main::_find_rom_in_pkg(Directory::Path const &pkg_path,
case Archive::SRC: case Archive::SRC:
{ {
Archive::Path const Archive::Path const
rom_path(Archive::user(archive_path), "/bin/", rom_path(Archive::user(archive_path), "/bin/",
_architecture, "/", _architecture, "/",
Archive::name(archive_path), "/", rom_label); Archive::name(archive_path), "/",
Archive::version(archive_path), "/", rom_label);
if (depot_dir.file_exists(rom_path)) if (depot_dir.file_exists(rom_path))
result = rom_path; result = rom_path;

9
tool/depot/build
View File

@ -23,12 +23,12 @@ define HELP_MESSAGE
E.g., the user 'alan' may build the following archives: E.g., the user 'alan' may build the following archives:
alan/bin/x86_64/zlib-<version> - a binary archive of the zlib alan/bin/x86_64/zlib/<version> - a binary archive of the zlib
library with the specified library with the specified
version, built for the 64-bit version, built for the 64-bit
x86 architecture x86 architecture
alan/pkg/x86_32/wm-<version> - all binary archives needed by alan/pkg/x86_32/wm/<version> - all binary archives needed by
the 'wm' package archive, built the 'wm' package archive, built
for the 32-bit x86 architecture for the 32-bit x86 architecture
@ -95,7 +95,7 @@ endif
# determine binary-archive path within the depot # determine binary-archive path within the depot
_dst_bin_spec_path = $(call archive_user,$1)/bin/$(call bin_archive_spec,$1)/ _dst_bin_spec_path = $(call archive_user,$1)/bin/$(call bin_archive_spec,$1)/
dst_archive_path = $(call _dst_bin_spec_path,$1)$(call bin_archive_recipe,$1) dst_archive_path = $(call _dst_bin_spec_path,$1)$(call bin_archive_recipe,$1)/$(call bin_archive_version,$1)
BUILD_MK_FILE := $(DEPOT_DIR)/var/build.mk BUILD_MK_FILE := $(DEPOT_DIR)/var/build.mk
@ -113,11 +113,12 @@ $(BUILD_MK_FILE): checked_source_archives_exist checked_no_uncategorized
target=$(call dst_archive_path,$A); \ target=$(call dst_archive_path,$A); \
user=$(call archive_user,$A); \ user=$(call archive_user,$A); \
recipe=$(call bin_archive_recipe,$A); \ recipe=$(call bin_archive_recipe,$A); \
version=$(call bin_archive_version,$A); \
spec=$(call bin_archive_spec,$A); \ spec=$(call bin_archive_spec,$A); \
echo ""; \ echo ""; \
echo "TARGETS += $$target"; \ echo "TARGETS += $$target"; \
echo "TOOL($$target) := build_bin_archive"; \ echo "TOOL($$target) := build_bin_archive"; \
echo "ARGS($$target) := $$recipe USER=$$user SPEC=$$spec"; \ echo "ARGS($$target) := $$recipe/$$version USER=$$user SPEC=$$spec"; \
) \ ) \
echo -e "\nall: \$$(TARGETS)"; \ echo -e "\nall: \$$(TARGETS)"; \
echo -e "\n\$$(TARGETS):"; \ echo -e "\n\$$(TARGETS):"; \

4
tool/depot/create
View File

@ -44,8 +44,8 @@ extract:
$(MAKECMDGOALS): extract $(MAKECMDGOALS): extract
_versioned_src_of_bin = $1-$(call recipe_version,src/$(call bin_archive_recipe,$1)) _versioned_src_of_bin = $1/$(call recipe_version,src/$(call bin_archive_recipe,$1))
_versioned_pkg = $1-$(call recipe_version,pkg/$(call bin_archive_recipe,$1)) _versioned_pkg = $1/$(call recipe_version,pkg/$(call bin_archive_recipe,$1))
versioned_archive = $(if $(call archive_has_type,$1,bin),$(call _versioned_src_of_bin,$1),\ versioned_archive = $(if $(call archive_has_type,$1,bin),$(call _versioned_src_of_bin,$1),\
$(if $(call archive_has_type,$1,pkg),$(call _versioned_pkg,$1))) $(if $(call archive_has_type,$1,pkg),$(call _versioned_pkg,$1)))

13
tool/depot/extract
View File

@ -47,6 +47,9 @@ endef
export GENODE_DIR := $(realpath $(dir $(MAKEFILE_LIST))/../..) export GENODE_DIR := $(realpath $(dir $(MAKEFILE_LIST))/../..)
# the extract tool expects archive paths given without the version element
BIN_PKG_PATH_ELEMS := 4
include $(GENODE_DIR)/tool/depot/mk/front_end.inc include $(GENODE_DIR)/tool/depot/mk/front_end.inc
include $(GENODE_DIR)/tool/depot/mk/categorize_args.inc include $(GENODE_DIR)/tool/depot/mk/categorize_args.inc
@ -86,9 +89,9 @@ include $(GENODE_DIR)/tool/depot/mk/dependencies.inc
# #
# Obtain version information from recipes # Obtain version information from recipes
# #
# The 'archive_version' function takes the archive type and name as arguments # The 'archive_curr_version' function takes the archive type and name as
# and returns the version identifier as present in the corresponding recipe. # arguments and returns the version identifier as present in the corresponding
# The nested foreach loop populates 'ARCHIVE_VERSION' with the version # recipe. The nested foreach loop populates 'ARCHIVE_VERSION' with the version
# identifier for each archive. # identifier for each archive.
# #
# If an archive is given with a complete (versioned) name, we don't need to # If an archive is given with a complete (versioned) name, we don't need to
@ -99,7 +102,7 @@ include $(GENODE_DIR)/tool/depot/mk/dependencies.inc
$(foreach TYPE,api src raw pkg,\ $(foreach TYPE,api src raw pkg,\
$(foreach PATH,${ARCHIVES(${TYPE})},\ $(foreach PATH,${ARCHIVES(${TYPE})},\
$(eval ARCHIVE_VERSION(${PATH}) := $(call archive_version,$(PATH))))) $(eval ARCHIVE_VERSION(${PATH}) := $(call archive_curr_version,$(PATH)))))
archive_exists_in_depot = $(wildcard $(DEPOT_DIR)/$1) archive_exists_in_depot = $(wildcard $(DEPOT_DIR)/$1)
@ -121,7 +124,7 @@ endif
# return versioned archive path, if 'ARCHIVE_VERSION' is undefined, assume # return versioned archive path, if 'ARCHIVE_VERSION' is undefined, assume
# that the argument is already a versiond path # that the argument is already a versiond path
versioned_archive = $(if $(ARCHIVE_VERSION($1)),$(addsuffix -${ARCHIVE_VERSION($1)},$1),$1) versioned_archive = $(if $(ARCHIVE_VERSION($1)),$(addsuffix /${ARCHIVE_VERSION($1)},$1),$1)
EXTRACT_MK_FILE := $(DEPOT_DIR)/var/extract.mk EXTRACT_MK_FILE := $(DEPOT_DIR)/var/extract.mk

2
tool/depot/mk/build_bin_archive
View File

@ -76,7 +76,7 @@ checked_src_archive:
@true @true
ifeq ($(SRC_DIR),) ifeq ($(SRC_DIR),)
VERSIONED_ARCHIVE := $(ARCHIVE)-$(SRC_VERSION) VERSIONED_ARCHIVE := $(ARCHIVE)/$(SRC_VERSION)
SRC_DIR := $(DEPOT_SRC_DIR)/$(VERSIONED_ARCHIVE) SRC_DIR := $(DEPOT_SRC_DIR)/$(VERSIONED_ARCHIVE)
checked_src_archive: checked_src_hash_file checked_src_archive: checked_src_hash_file
endif endif

21
tool/depot/mk/categorize_args.inc
View File

@ -36,16 +36,23 @@ endif
# #
# Sub-categorize source-pkg archives (<user>/pkg/<name>) from binary-pkg # Sub-categorize source-pkg archives (<user>/pkg/<name>[/<version>]) from
# archives (<user>/pkg/<spec>/<name>) so that 'ARCHIVES(pkg)' contains source # binary-pkg archives (<user>/pkg/<spec>/<name>,[<version>]) so that
# pkgs only, and 'ARCHIVES(binpkg)' contains binary pkgs. # 'ARCHIVES(pkg)' contains source pkgs only, and 'ARCHIVES(binpkg)' contains
# binary pkgs.
# #
# If the path contains 4 elements, it refers to a binary pkg where the third # If the path contains 'BIN_PKG_PATH_ELEMS' elements, it refers to a binary pkg
# element is the build spec. Otherwise, the path refers to a source pkg. # where the third element is the build spec. Otherwise, the path refers to a
# source pkg. By default, the distinction assumes versioned archive paths.
# For the 'extract' tool where the version elements are omitted because they
# refer to the current version as present in the repository, the value is
# customized to '4'.
# #
_src_pkg = $(if $(word 4,$(subst /, ,$1)),,$1) BIN_PKG_PATH_ELEMS ?= 5
_bin_pkg = $(if $(word 4,$(subst /, ,$1)),$1,)
_src_pkg = $(if $(word $(BIN_PKG_PATH_ELEMS),$(subst /, ,$1)),,$1)
_bin_pkg = $(if $(word $(BIN_PKG_PATH_ELEMS),$(subst /, ,$1)),$1,)
ARCHIVES(binpkg) := $(strip $(foreach PKG,${ARCHIVES(pkg)},$(call _bin_pkg,$(PKG)))) ARCHIVES(binpkg) := $(strip $(foreach PKG,${ARCHIVES(pkg)},$(call _bin_pkg,$(PKG))))
ARCHIVES(pkg) := $(strip $(foreach PKG,${ARCHIVES(pkg)},$(call _src_pkg,$(PKG)))) ARCHIVES(pkg) := $(strip $(foreach PKG,${ARCHIVES(pkg)},$(call _src_pkg,$(PKG))))

2
tool/depot/mk/dependencies.inc
View File

@ -66,7 +66,7 @@ _src_exists = $(wildcard $(dir $(call used_apis_file,$1)))
# return binary-archive path for architecture $1 and source archive $2 # return binary-archive path for architecture $1 and source archive $2
_bin_for_src = $(strip \ _bin_for_src = $(strip \
$(if $(call _src_exists,$2),\ $(if $(call _src_exists,$2),\
$(call archive_user,$2)/bin/$1/$(call archive_recipe,$2))) $(call archive_user,$2)/bin/$1/$(call archive_recipe,$2)/$(call archive_version,$2)))
# return list of binary archives contained in a binary package # return list of binary archives contained in a binary package
_binpkg_bin_archives = $(foreach S,$(call _binpkg_src_archives,$1),\ _binpkg_bin_archives = $(foreach S,$(call _binpkg_src_archives,$1),\

22
tool/depot/mk/extract.inc
View File

@ -85,7 +85,7 @@ ORIG_RECIPE_HASH_VALUE := $(RECIPE_HASH_VALUE)
# archive hash. # archive hash.
# #
DEPOT_ARCHIVE_DIR := $(DEPOT_SUB_DIR)/$(ARCHIVE).incomplete DEPOT_ARCHIVE_DIR := $(DEPOT_SUB_DIR)/$(ARCHIVE)/incomplete
reset_stale_temporary_archive_dir: reset_stale_temporary_archive_dir:
ifneq ($(wildcard $(DEPOT_ARCHIVE_DIR)),) ifneq ($(wildcard $(DEPOT_ARCHIVE_DIR)),)
@ -110,16 +110,16 @@ $(ARCHIVE): _rename_to_final_archive
# discard the just-built archive. # discard the just-built archive.
# #
_rename_to_final_archive: _check_hash _rename_to_final_archive: _check_hash
@$(VERBOSE)final_name=$(ARCHIVE)-$(RECIPE_VERSION); \ $(VERBOSE)final_name=$(ARCHIVE)/$(RECIPE_VERSION); \
rm -rf $(DEPOT_SUB_DIR)/$$final_name; \ rm -rf $(DEPOT_SUB_DIR)/$$final_name; \
mv $(DEPOT_ARCHIVE_DIR) $(DEPOT_SUB_DIR)/$$final_name; \ mv $(DEPOT_ARCHIVE_DIR) $(DEPOT_SUB_DIR)/$$final_name; \
hash=$$(< $(DEPOT_ARCHIVE_DIR).hash); hint=""; \ hash=$$(< $(DEPOT_ARCHIVE_DIR).hash); hint=""; \
test $$hash = $(ORIG_RECIPE_HASH_VALUE) ||\ test $$hash = $(ORIG_RECIPE_HASH_VALUE) ||\
hint=" $(BRIGHT_COL)(new version)$(DEFAULT_COL)"; \ hint=" $(BRIGHT_COL)(new version)$(DEFAULT_COL)"; \
rm -f $(DEPOT_ARCHIVE_DIR).hash; \ rm -f $(DEPOT_ARCHIVE_DIR).hash; \
$(ECHO) "$(DARK_COL)created$(DEFAULT_COL)" \ $(ECHO) "$(DARK_COL)created$(DEFAULT_COL)" \
"$(USER)/$(notdir $(DEPOT_SUB_DIR))/$$final_name$$hint"; \ "$(USER)/$(notdir $(DEPOT_SUB_DIR))/$$final_name$$hint"; \
true; true;
# #
# Generate suggested version name for 'HASH_OUT_OF_DATE_MESSAGE' # Generate suggested version name for 'HASH_OUT_OF_DATE_MESSAGE'

2
tool/depot/mk/extract_pkg_archive
View File

@ -45,7 +45,7 @@ include $(GENODE_DIR)/tool/depot/mk/extract.inc
# #
_version = $(call recipe_version,$(call archive_type,$1)/$(call archive_recipe,$1)) _version = $(call recipe_version,$(call archive_type,$1)/$(call archive_recipe,$1))
_versioned_entry = _/$(call archive_type,$1)/$(call archive_recipe,$1)-$(call _version,$1) _versioned_entry = _/$(call archive_type,$1)/$(call archive_recipe,$1)/$(call _version,$1)
VERSIONED_ARCHIVES := $(foreach A,$(call file_content,$(RECIPE_DIR)/archives),\ VERSIONED_ARCHIVES := $(foreach A,$(call file_content,$(RECIPE_DIR)/archives),\
$(if $(call archive_has_user,$A,_),$(call _versioned_entry,$A),$A)) $(if $(call archive_has_user,$A,_),$(call _versioned_entry,$A),$A))

4
tool/depot/mk/extract_src_archive
View File

@ -67,7 +67,7 @@ $(DEPOT_ARCHIVE_DIR)/used_apis: $(RECIPE_DIR)/used_apis
fi; \ fi; \
hash_file_content=$$(< $$hash_file); \ hash_file_content=$$(< $$hash_file); \
version=$${hash_file_content%% *}; \ version=$${hash_file_content%% *}; \
echo "$$api-$$version" >> $@; \ echo "$$api/$$version" >> $@; \
done; $$result done; $$result
# #
@ -89,5 +89,5 @@ $(DEPOT_ARCHIVE_DIR)/api: $(RECIPE_DIR)/api
fi; \ fi; \
hash_file_content=$$(< $$hash_file); \ hash_file_content=$$(< $$hash_file); \
version=$${hash_file_content%% *}; \ version=$${hash_file_content%% *}; \
echo "$$api-$$version" >> $@; echo "$$api/$$version" >> $@;

24
tool/depot/mk/front_end.inc
View File

@ -42,24 +42,34 @@ last_path_element = $(call sanitized,$(lastword $(subst /, ,$1)))
archive_user = $(call path_element,1,$1) archive_user = $(call path_element,1,$1)
archive_type = $(call path_element,2,$1) archive_type = $(call path_element,2,$1)
archive_recipe = $(call path_element,3,$1) archive_recipe = $(call path_element,3,$1)
archive_version = $(call path_element,4,$1)
archive_has_type = $(filter $(call archive_type,$1),$2) archive_has_type = $(filter $(call archive_type,$1),$2)
archive_has_user = $(filter $(call archive_user,$1),$2) archive_has_user = $(filter $(call archive_user,$1),$2)
archive_version = $(call recipe_version,$(addprefix $(call archive_type,$1)/,$(call archive_recipe,$1)))
# binary archives have the form <user>/bin/<spec>/<name>{-<version>} archive_curr_version = $(call recipe_version,$(addprefix $(call archive_type,$1)/,$(call archive_recipe,$1)))
# binary archives have the form <user>/bin/<spec>/<name>/<version>
bin_archive_spec = $(call path_element,3,$1) bin_archive_spec = $(call path_element,3,$1)
bin_archive_recipe = $(call last_path_element,$1) bin_archive_recipe = $(call path_element,4,$1)
bin_archive_version = $(call recipe_version,src/$(call bin_archive_recipe,$1)) bin_archive_version = $(call path_element,5,$1)
grep_archive_type = $(foreach A,$2,$(if $(call archive_has_type,$A,$1),$A,)) grep_archive_type = $(foreach A,$2,$(if $(call archive_has_type,$A,$1),$A,))
grep_archive_user = $(foreach A,$2,$(if $(call archive_has_user,$A,$1),$A,)) grep_archive_user = $(foreach A,$2,$(if $(call archive_has_user,$A,$1),$A,))
#
# The following functions can be called for archive paths with or without
# the version part. In the latter case, 'bin_archive_version' is empty.
# The 'addprefix' is needed to omit the trailing '/' in the result if a
# path without version is specified.
#
# return pkg-archive path of given binary-pkg archive path # return pkg-archive path of given binary-pkg archive path
pkg_of_binpkg = $(call archive_user,$1)/pkg/$(call bin_archive_recipe,$1) pkg_of_binpkg = $(call archive_user,$1)/pkg/$(call bin_archive_recipe,$1)$(addprefix /,$(call bin_archive_version,$1))
# return source-archive path for given binary-archive path # return source-archive path for given binary-archive path
src_of_bin = $(call archive_user,$1)/src/$(call bin_archive_recipe,$1) src_of_bin = $(call archive_user,$1)/src/$(call bin_archive_recipe,$1)$(addprefix /,$(call bin_archive_version,$1))
# return binary-package archive path for architecture $1 and package archive $2 # return binary-package archive path for architecture $1 and package archive $2
binpkg_for_pkg = $(call archive_user,$2)/bin/$1/$(call archive_recipe,$2) binpkg_for_pkg = $(call archive_user,$2)/bin/$1/$(call archive_recipe,$2)$(addprefix /,$(call archive_version,$2))

212
tool/run/depot.inc
View File

@ -22,28 +22,40 @@ proc depot_spec { } {
# depot. The list is populated by calls of 'import_from_depot' and evaluated # depot. The list is populated by calls of 'import_from_depot' and evaluated
# at the boot-image-creation stage via 'check_for_missing_depot_archives'. # at the boot-image-creation stage via 'check_for_missing_depot_archives'.
# #
# Each list element is a list of <user>, <type>, <spec>, <name>, and <version>.
#
set _missing_depot_archives {} set _missing_depot_archives {}
# #
# Pattern to parse an archive path into <user>, <type>, <name> # Pattern to parse an version-less archive path into <user>, <type>, <name>
# #
proc _depot_archive_path_pattern { } { return {^([\w\d]+)/([\w]+)/([\w\d\-_]+)$} } proc _depot_archive_path_pattern { } {
return {^([\w\d]+)/([\w]+)/([\w\d\-_]+)$} }
#
# Pattern to parse an versioned archive path into <user>, <type>, <name>, <version>
#
proc _depot_archive_versioned_path_pattern { } {
return {^([\w\d]+)/([\w]+)/([\w\d\-_]+)/([\w\d\-_]+)$} }
# #
# Pattern to parse an binary archive path into <user>, <spec>, <name>. # Pattern to parse an binary archive path into <user>, <spec>, <name>.
# #
proc _depot_bin_archive_path_pattern { } { return {^([\w\d]+)/bin/([\w\d]+)/([\w\d\-_]+)$} } proc _depot_bin_archive_path_pattern { } {
return {^([\w\d]+)/bin/([\w\d]+)/([\w\d\-_]+)$} }
## ##
# Determine content of a pkg archive and its dependencies # Determine content of a pkg archive and its dependencies
# #
proc _collect_pkg_archive_from_depot { user name } { proc _collect_pkg_archive_from_depot { user name version } {
global _missing_depot_archives global _missing_depot_archives
set archive_dir "$user/pkg/$name" set archive_dir "$user/pkg/$name/$version"
set archives_file "[depot_dir]/$archive_dir/archives" set archives_file "[depot_dir]/$archive_dir/archives"
if {![file exists $archives_file]} { if {![file exists $archives_file]} {
@ -57,7 +69,7 @@ proc _collect_pkg_archive_from_depot { user name } {
set content "$archive_dir" set content "$archive_dir"
foreach archive $archives { foreach archive $archives {
if {[regexp [_depot_archive_path_pattern] $archive dummy user type name]} { if {[regexp [_depot_archive_versioned_path_pattern] $archive dummy user type name version]} {
if {($type == "pkg") || ($type == "src") || ($type == "raw")} { if {($type == "pkg") || ($type == "src") || ($type == "raw")} {
set content [concat $content [_collect_from_depot $archive]] set content [concat $content [_collect_from_depot $archive]]
} }
@ -79,23 +91,24 @@ proc _copy_directory_content_to_run_dir { dir } {
} }
proc _collect_raw_archive_from_depot { user name } { return "$user/raw/$name" } proc _collect_raw_archive_from_depot { user name version } {
return "$user/raw/$name/$version" }
## ##
# Determine binary content for a given source archive # Determine binary content for a given source archive
# #
proc _collect_src_archive_from_depot { user name } { proc _collect_src_archive_from_depot { user name version } {
global _missing_depot_archives; global _missing_depot_archives;
set src_archive_dir "$user/src/$name" set src_archive_dir "$user/src/$name/$version"
set bin_archive_dir "$user/bin/[depot_spec]/$name" set bin_archive_dir "$user/bin/[depot_spec]/$name/$version"
if {[file exists [depot_dir]/$bin_archive_dir]} { if {[file exists [depot_dir]/$bin_archive_dir]} {
return [list $src_archive_dir $bin_archive_dir] return [list $src_archive_dir $bin_archive_dir]
} else { } else {
lappend _missing_depot_archives $bin_archive_dir lappend _missing_depot_archives [list $user bin [depot_spec] $name $version]
} }
return {} return {}
@ -103,35 +116,30 @@ proc _collect_src_archive_from_depot { user name } {
## ##
# Determine the version-suffixed name of an archive # Determine the current version for the given archive
# #
# This function return an empty string if the archive is missing from the # This function tries to determine the version information from the Genode
# source tree. It returns an empty string if the archive is missing from the
# depot. # depot.
# #
proc _versioned_depot_archive_name { user type name } { proc _current_depot_archive_version { type name } {
# if correctly versioned archive is specified, use it
if {[file exists [depot_dir]/$user/$type/$name]} { return $name }
#
# The given archive name may lack the version identifier if it refers
# to an archive generated locally from the Genode repository. In this case,
# we try to determine the version information from the Genode source tree.
#
set hash_rel_path "recipes/$type/$name/hash" set hash_rel_path "recipes/$type/$name/hash"
set repo [repository_contains $hash_rel_path] set repo [repository_contains $hash_rel_path]
set version ""
if {$repo != ""} { if {$repo != ""} {
set fh [open "$repo/$hash_rel_path" "RDONLY"] set fh [open "$repo/$hash_rel_path" "RDONLY"]
set version [lindex [gets $fh] 0] set version [lindex [gets $fh] 0]
close $fh close $fh
append name "-" $version
if {[file exists [depot_dir]/$user/$type/$name]} {
return $name }
} }
return $version
}
return ""
proc _depot_contains_archive { user type name version } {
return [file exists [depot_dir]/$user/$type/$name/$version]
} }
@ -143,37 +151,49 @@ proc _collect_from_depot { archives } {
foreach archive $archives { foreach archive $archives {
if {[regexp [_depot_archive_path_pattern] $archive dummy user type name]} { set version ""
set versioned_name [_versioned_depot_archive_name $user $type $name] #
if {$versioned_name == ""} { # Try to parse versioned archive path. If no version is specified, use
lappend _missing_depot_archives $archive # the current version as present in the source tree.
#
if {![regexp [_depot_archive_versioned_path_pattern] $archive dummy user type name version]} {
if {[regexp [_depot_archive_path_pattern] $archive dummy user type name]} {
set version [_current_depot_archive_version $type $name]
} else { } else {
puts stderr "Error: malformed depot-archive path '$archive',"
set content {} puts stderr " expected '<user>/<type>/<name>'"
exit 1
switch $type {
"pkg" { set content [_collect_pkg_archive_from_depot $user $versioned_name] }
"src" { set content [_collect_src_archive_from_depot $user $versioned_name] }
"raw" { set content [_collect_raw_archive_from_depot $user $versioned_name] }
default {
puts stderr "Error: unknown depot-archive type '$type'"
exit 1
}
}
set all_content [concat $all_content $content]
} }
} else { }
puts stderr "Error: malformed depot-archive path '$archive',"
puts stderr " expected '<user>/<type>/<name>'" if {$version == ""} {
puts stderr "Error: unable to guess version of '$type/$name' archive"
exit 1 exit 1
} }
}
if {![_depot_contains_archive $user $type $name $version]} {
lappend _missing_depot_archives [list $user $type "" $name $version]
continue
}
set content {}
switch $type {
"pkg" { set content [_collect_pkg_archive_from_depot $user $name $version] }
"src" { set content [_collect_src_archive_from_depot $user $name $version] }
"raw" { set content [_collect_raw_archive_from_depot $user $name $version] }
default {
puts stderr "Error: unknown depot-archive type '$type'"
exit 1
}
}
set all_content [concat $all_content $content]
}
return $all_content return $all_content
} }
@ -187,7 +207,7 @@ proc import_from_depot { args } {
foreach subdir [_collect_from_depot $args] { foreach subdir [_collect_from_depot $args] {
# prevent src, api, and pkg archives from inflating the boot image # prevent src, api, and pkg archives from inflating the boot image
if {[regexp [_depot_archive_path_pattern] $subdir dummy user type]} { if {[regexp [_depot_archive_versioned_path_pattern] $subdir dummy user type]} {
if {$type == "src"} continue; if {$type == "src"} continue;
if {$type == "api"} continue; if {$type == "api"} continue;
if {$type == "pkg"} continue; if {$type == "pkg"} continue;
@ -206,7 +226,7 @@ proc create_tar_from_depot_binaries { archive_path args } {
# filter out api and src archives from requested depot content # filter out api and src archives from requested depot content
set content {} set content {}
foreach subdir [_collect_from_depot $args] { foreach subdir [_collect_from_depot $args] {
if {[regexp [_depot_archive_path_pattern] $subdir dummy user type]} { if {[regexp [_depot_archive_versioned_path_pattern] $subdir dummy user type]} {
if {$type == "src"} continue; if {$type == "src"} continue;
if {$type == "api"} continue; if {$type == "api"} continue;
} }
@ -219,31 +239,17 @@ proc create_tar_from_depot_binaries { archive_path args } {
} }
proc _locally_available_recipe { user type name } { proc _locally_available_recipe { user type name version } {
if {[repository_contains "recipes/$type/$name/hash"] != ""} { if {$type == "bin"} { set type "src" }
return $name }
# if {[repository_contains "recipes/$type/$name/hash"] == ""} {
# If the supplied 'name' is a versioned name (as obtained from a pkg return 0 }
# 'archives' filed already stored in the depot, we try to find a recipe
# in the source tree that matches the specified name and version. If
# we find the matching recipe, we can build the archive locally using
# the version-less recipe name.
#
set version_suffix_pattern {\-[^\-/]*$}
set versioned_name $name
while {[regexp -- $version_suffix_pattern $name dummy]} { if {$version != [_current_depot_archive_version $type $name]} {
return 0 }
# strip last part of version suffix return 1
regsub -- $version_suffix_pattern $name "" name
if {[repository_contains "recipes/$type/$name/hash"] != ""} {
if {[_versioned_depot_archive_name $user $type $name] == $versioned_name} {
return $name } }
}
return ""
} }
@ -280,45 +286,45 @@ proc check_for_missing_depot_archives { } {
set foreign_archives {} set foreign_archives {}
foreach archive $_missing_depot_archives { foreach archive $_missing_depot_archives {
puts stderr " $archive"
if {[regexp [_depot_archive_path_pattern] $archive dummy user type name]} {
# set user [lindex $archive 0]
# If a pkg archive is missing, suggest to obtain the binary-pkg set type [lindex $archive 1]
# archive (matching the build directory) immediately, which implies set spec [lindex $archive 2]
# the pkg archive. Otherwise, the user would first obtain the pkg set name [lindex $archive 3]
# archive and its source dependencies, and then get an error for set version [lindex $archive 4]
# the missing binary archives on the next attempt to execute the
# run script.
#
if {$type == "pkg"} { set archive "$user/pkg/[depot_spec]/$name" }
if {$type == "src"} { set archive "$user/bin/[depot_spec]/$name" }
if {[_locally_available_recipe $user $type $name] != ""} { #
lappend local_user_archives $archive # If a pkg archive is missing, suggest to obtain the binary-pkg
} else { # archive (matching the build directory) immediately, which implies
lappend foreign_archives $archive # the pkg archive. Otherwise, the user would first obtain the pkg
} # archive and its source dependencies, and then get an error for
} elseif {[regexp [_depot_bin_archive_path_pattern] $archive dummy user spec name]} { # the missing binary archives on the next attempt to execute the
# run script.
#
if {$type == "pkg"} { set spec "[depot_spec]" }
if {$type == "src"} {
set type "bin"
set spec "[depot_spec]"
}
# source code is present in the current source tree set path "$user/$type/$name"
set recipe [_locally_available_recipe $user src $name] if {$type == "bin"} {
if {$recipe != ""} { set path "$user/bin/$spec/$name" }
lappend local_user_archives $user/bin/$spec/$recipe
# source code is present in the depot puts stderr " $path/$version"
} elseif {[file isdirectory [depot_dir]/$user/src/$name]} {
lappend local_user_archives $user/bin/$spec/$name
} else { if {[_locally_available_recipe $user $type $name $version]} {
lappend foreign_archives $archive lappend local_user_archives $path
} } else {
lappend foreign_archives $path/$version
} }
} }
append create_args " CROSS_DEV_PREFIX=[cross_dev_prefix]" if {[llength $local_user_archives] || [llength $foreign_archives]} {
puts stderr "" }
if {[llength $local_user_archives]} { if {[llength $local_user_archives]} {
append create_args " CROSS_DEV_PREFIX=[cross_dev_prefix]"
puts stderr "You may create the following archives locally:\n" puts stderr "You may create the following archives locally:\n"
puts stderr " [genode_dir]/tool/depot/create $local_user_archives$create_args\n" puts stderr " [genode_dir]/tool/depot/create $local_user_archives$create_args\n"
} }