pkg-infra: don't use DL_DIR as scratchpad for temporary downloads

DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.

We do not want to trash this location with our temporary downloads (e.g.
git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).

Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.

Also, we create the temporary tarballs with mktemp using the final tarball,
as template, since we want the temporary to be on the same filesystem as
the final location, so the 'mv' is just a plain, atomic rename(2), and we
are not left with a half-copied file as the final location.

Using mktemp ensures all temp file names are unique, so it allows for
parallel downloads from different build dirs at the same time, without
cloberring each downloads.

Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail: two separate builds
using a shared DL_DIR, ccache enabled, so that they run almost
synchronously. These would download the same file at the same time,
corrupting each other. With the patches in this series, all works
fine.]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit is contained in:
Yann E. MORIN 2014-07-03 21:36:20 +02:00 committed by Peter Korsgaard
parent 284baca15a
commit 19afad50f5
8 changed files with 173 additions and 39 deletions

View File

@ -9,11 +9,30 @@ set -e
# $2: bzr revision # $2: bzr revision
# $3: output file # $3: output file
# And this environment: # And this environment:
# BZR : the bzr command to call # BZR : the bzr command to call
# BR2_DL_DIR: path to Buildroot's download dir # BUILD_DIR: path to Buildroot's build dir
repo="${1}" repo="${1}"
rev="${2}" rev="${2}"
output="${3}" output="${3}"
${BZR} export "${output}" "${repo}" -r "${rev}" tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
tmp_output="$( mktemp "${output}.XXXXXX" )"
# Play tic-tac-toe with temp files
# - first, we download to a trashable location (the build-dir)
# - the we move to a temp file in the final location, so it is
# on the same filesystem as the final file
# - finally, we atomically rename to the final file
ret=1
if ${BZR} export "${tmp_dl}" "${repo}" -r "${rev}"; then
if mv "${tmp_dl}" "${tmp_output}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
fi
# Cleanup
rm -f "${tmp_dl}" "${tmp_output}"
exit ${ret}

View File

@ -13,4 +13,14 @@ set -e
source="${1}" source="${1}"
output="${2}" output="${2}"
${LOCALFILES} "${source}" "${output}" tmp_output="$( mktemp "${output}.XXXXXX" )"
ret=1
if ${LOCALFILES} "${source}" "${tmp_output}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
# Cleanup
rm -f "${tmp_output}"
exit ${ret}

View File

@ -11,8 +11,8 @@ set -e
# $4: package's basename (eg. foobar-1.2.3) # $4: package's basename (eg. foobar-1.2.3)
# $5: output file # $5: output file
# And this environment: # And this environment:
# CVS : the cvs command to call # CVS : the cvs command to call
# BR2_DL_DIR: path to Buildroot's download dir # BUILD_DIR: path to Buildroot's build dir
repo="${1}" repo="${1}"
rev="${2}" rev="${2}"
@ -20,8 +20,28 @@ rawname="${3}"
basename="${4}" basename="${4}"
output="${5}" output="${5}"
cd "${BR2_DL_DIR}" repodir="${basename}.tmp-cvs-checkout"
${CVS} -z3 -d":pserver:anonymous@${repo}" \ tmp_output="$( mktemp "${output}.XXXXXX" )"
co -d "${basename}" -r ":${rev}" -P "${rawname}"
tar czf "${output}" "${basename}" cd "${BUILD_DIR}"
rm -rf "${basename}" # Remove leftovers from a previous failed run
rm -rf "${repodir}"
# Play tic-tac-toe with temp files
# - first, we download to a trashable location (the build-dir)
# - then we create a temporary tarball in the final location, so it is
# on the same filesystem as the final file
# - finally, we atomically rename to the final file
ret=1
if ${CVS} -z3 -d":pserver:anonymous@${repo}" \
co -d "${repodir}" -r ":${rev}" -P "${rawname}"; then
if tar czf "${tmp_output}" "${repodir}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
fi
# Cleanup
rm -rf "${repodir}" "${tmp_output}"
exit ${ret}

View File

@ -10,15 +10,28 @@ set -e
# $3: package's basename (eg. foobar-1.2.3) # $3: package's basename (eg. foobar-1.2.3)
# $4: output file # $4: output file
# And this environment: # And this environment:
# BR2_DL_DIR: path to Buildroot's download dir # GIT : the git command to call
# GIT : the git command to call # BUILD_DIR: path to Buildroot's build dir
repo="${1}" repo="${1}"
cset="${2}" cset="${2}"
basename="${3}" basename="${3}"
output="${4}" output="${4}"
repodir="${BR2_DL_DIR}/${basename}" repodir="${basename}.tmp-git-checkout"
tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
tmp_output="$( mktemp "${output}.XXXXXX" )"
# Play tic-tac-toe with temp files
# - first, we download to a trashable location (the build-dir)
# - then we create the uncomporessed tarball in tht same trashable location
# - then we create a temporary compressed tarball in the final location, so
# it is on the same filesystem as the final file
# - finally, we atomically rename to the final file
cd "${BUILD_DIR}"
# Remove leftovers from a previous failed run
rm -rf "${repodir}"
if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
printf "Doing shallow clone\n" printf "Doing shallow clone\n"
@ -28,10 +41,16 @@ else
${GIT} clone --bare "${repo}" "${repodir}" ${GIT} clone --bare "${repo}" "${repodir}"
fi fi
ret=1
pushd "${repodir}" >/dev/null pushd "${repodir}" >/dev/null
${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}" if ${GIT} archive --prefix="${basename}/" -o "${tmp_tar}" \
gzip -c "${output}.tmp" >"${output}" --format=tar "${cset}"; then
rm -f "${output}.tmp" if gzip -c "${tmp_tar}" >"${tmp_output}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
fi
popd >/dev/null popd >/dev/null
rm -rf "${repodir}" rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
exit ${ret}

View File

@ -10,16 +10,37 @@ set -e
# $3: package's basename (eg. foobar-1.2.3) # $3: package's basename (eg. foobar-1.2.3)
# $4: output file # $4: output file
# And this environment: # And this environment:
# HG : the hg command to call # HG : the hg command to call
# BR2_DL_DIR: path to Buildroot's download dir # BUILD_DIR: path to Buildroot's build dir
repo="${1}" repo="${1}"
cset="${2}" cset="${2}"
basename="${3}" basename="${3}"
output="${4}" output="${4}"
cd "${BR2_DL_DIR}" repodir="${basename}.tmp-hg-checkout"
${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}" tmp_output="$( mktemp "${output}.XXXXXX" )"
${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
--rev "${cset}" "${output}" cd "${BUILD_DIR}"
rm -rf "${basename}" # Remove leftovers from a previous failed run
rm -rf "${repodir}"
# Play tic-tac-toe with temp files
# - first, we download to a trashable location (the build-dir)
# - then we create a temporary tarball in the final location, so it is
# on the same filesystem as the final file
# - finally, we atomically rename to the final file
ret=1
if ${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}"; then
if ${HG} archive --repository "${repodir}" --type tgz \
--prefix "${basename}" --rev "${cset}" \
"${tmp_output}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
fi
# Cleanup
rm -rf "${repodir}" "${tmp_output}"
exit ${ret}

View File

@ -12,5 +12,17 @@ set -e
url="${1}" url="${1}"
output="${2}" output="${2}"
tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
tmp_output="$( mktemp "${output}.XXXXXX" )"
${SCP} "${url}" "${output}" ret=1
if ${SCP} "${url}" "${tmp_dl}"; then
if mv "${tmp_dl}" "${tmp_output}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
fi
# Cleanup
rm -f "${tmp_dl}" "${tmp_output}"
exit ${ret}

View File

@ -10,16 +10,35 @@ set -e
# $3: package's basename (eg. foobar-1.2.3) # $3: package's basename (eg. foobar-1.2.3)
# $4: output file # $4: output file
# And this environment: # And this environment:
# SVN : the svn command to call # SVN : the svn command to call
# BR2_DL_DIR: path to Buildroot's download dir # BUILD_DIR: path to Buildroot's build dir
repo="${1}" repo="${1}"
rev="${2}" rev="${2}"
basename="${3}" basename="${3}"
output="${4}" output="${4}"
pushd "${BR2_DL_DIR}" >/dev/null repodir="${basename}.tmp-svn-checkout"
${SVN} export "${repo}@${rev}" "${basename}" tmp_output="$( mktemp "${output}.XXXXXX" )"
tar czf "${output}" "${basename}"
rm -rf "${basename}" cd "${BUILD_DIR}"
popd >/dev/null # Remove leftovers from a previous failed run
rm -rf "${repodir}"
# Play tic-tac-toe with temp files
# - first, we download to a trashable location (the build-dir)
# - then we create a temporary tarball in the final location, so it is
# on the same filesystem as the final file
# - finally, we atomically rename to the final file
ret=1
if ${SVN} export "${repo}@${rev}" "${repodir}"; then
if tar czf "${tmp_output}" "${repodir}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
fi
# Cleanup
rm -rf "${repodir}" "${tmp_output}"
exit ${ret}

View File

@ -8,14 +8,28 @@ set -e
# $1: URL # $1: URL
# $2: output file # $2: output file
# And this environment: # And this environment:
# WGET : the wget command to call # WGET : the wget command to call
# BUILD_DIR: path to Buildroot's build dir
url="${1}" url="${1}"
output="${2}" output="${2}"
if ${WGET} -O "${output}.tmp" "${url}"; then tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
mv "${output}.tmp" "${output}" tmp_output="$( mktemp "${output}.XXXXXX" )"
else
rm -f "${output}.tmp" # Play tic-tac-toe with temp files
exit 1 # - first, we download to a trashable location (the build-dir)
# - then we copy to a temporary tarball in the final location, so it is
# on the same filesystem as the final file
# - finally, we atomically rename to the final file
ret=1
if ${WGET} -O "${tmp_dl}" "${url}"; then
if mv "${tmp_dl}" "${tmp_output}"; then
mv "${tmp_output}" "${output}"
ret=0
fi
fi fi
rm -f "${tmp_dl}" "${tmp_output}"
exit ${ret}