Minix/pkgsrc-ng
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Import of pkgsrc-2013Q2

Browse Source
This commit is contained in:
Lionel Sambuc
2013-09-26 17:14:40 +02:00
commit 785076ae39
74991 changed files with 4380255 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
pkgtools/cdpack/DESCR Normal file
View File

@@ -0,0 +1,17 @@
cdpack is a small utility for creating ISO 9660 images for
a multi-CD/DVD binary package collection. The utility creates
ISO 9660 images for all the binary packages in a specified
directory.
A choice of two algorithms is available for how the packages
are grouped. The "no duplication" algorithm arranges the
packages so any package on CD/DVD number `n' will have all of
its dependencies on CD/DVD numbers 1 through `n'.
The "no inter-CD/DVD depends" algorithm will place certain
packages on more than one CD/DVD to ensure that each CD/DVD is
self contained (all package dependencies are satisfied within
the single CD/DVD).
The cddist utility in this package is used for creating an archive
set of distfiles.

62
pkgtools/cdpack/Makefile Normal file
View File

@@ -0,0 +1,62 @@
# $NetBSD: Makefile,v 1.31 2012/09/11 23:19:34 asau Exp $
#
DISTNAME= cdpack-1.9
CATEGORIES= pkgtools
MASTER_SITES= # empty
DISTFILES= # empty
MAINTAINER= dmcmahill@NetBSD.org
HOMEPAGE= http://www.NetBSD.org/
COMMENT= Utility to create multi-cd/dvd binary package collections
# for mkisofs
DEPENDS+= cdrtools-[0-9]*:../../sysutils/cdrtools
WRKSRC= ${WRKDIR}
NO_CONFIGURE= yes
USE_TOOLS+= awk:run
USE_TOOLS+= bzcat:run
USE_TOOLS+= expr:run
USE_TOOLS+= gzcat:run
USE_TOOLS+= printf:run
USE_TOOLS+= sort:run
USE_TOOLS+= tar:run
USE_TOOLS+= tsort:run
INSTALLATION_DIRS= bin libexec ${PKGMANDIR}/man1
do-build:
.for f in cdpack cddist
${SED} -e 's;@prefix@;${PREFIX};g' \
-e 's;@progver@;${PKGVERSION};g' \
-e 's;@AWK@;${AWK};g' \
-e 's;@BZCAT@;${BZCAT};g' \
-e 's;@EXPR@;${EXPR};g' \
-e 's;@GZCAT@;${GZCAT};g' \
-e 's;@PRINTF@;${PRINTF};g' \
-e 's;@SORT@;${SORT};g' \
-e 's;@TAR@;${TAR};g' \
-e 's;@TSORT@;${TSORT};g' \
${FILESDIR}/${f}.sh > \
${WRKSRC}/${f}
${CHMOD} 755 ${WRKDIR}/${f}
${SED} -e 's;@AWK@;${AWK};g' \
-e 's;@BZCAT@;${BZCAT};g' \
-e 's;@GZCAT@;${GZCAT};g' \
-e 's;@TAR@;${TAR};g' \
${FILESDIR}/${f}.1 > \
${WRKSRC}/${f}.1
.endfor
${SED} -e 's;@AWK@;${AWK};g' ${FILESDIR}/cdgen.awk > \
${WRKSRC}/cdgen.awk
do-install:
.for f in cdpack cddist
${INSTALL_SCRIPT} ${WRKSRC}/${f} ${DESTDIR}${PREFIX}/bin
${INSTALL_MAN} ${WRKSRC}/${f}.1 ${DESTDIR}${PREFIX}/${PKGMANDIR}/man1
.endfor
${INSTALL_SCRIPT} ${WRKSRC}/cdgen.awk ${DESTDIR}${PREFIX}/libexec
.include "../../mk/bsd.pkg.mk"

6
pkgtools/cdpack/PLIST Normal file
View File

@@ -0,0 +1,6 @@
@comment $NetBSD: PLIST,v 1.3 2008/01/28 23:06:13 dmcmahill Exp $
bin/cddist
bin/cdpack
libexec/cdgen.awk
man/man1/cddist.1
man/man1/cdpack.1

34
pkgtools/cdpack/files/TODO Normal file
View File

@@ -0,0 +1,34 @@
$NetBSD: TODO,v 1.5 2002/09/20 11:48:12 dmcmahill Exp $
To-Do List for cdpack
---------------------
- extract COMMENT and DESCR to generate a browseable index of all the
packages on the CD set. This would be similar to the README.html's
in pkgsrc, but with some modifications. The changes would be to
list which CD certain pkgs are on. Also, if a DEPENDS is not on the
CD set (due to redistribution restrictions for example), perhaps
list that so the user knows s/he must install the restricted DEPENDS
by hand first.
- come up with some sort of install tool around pkg_add. Perhaps this
could be done using SUSHI. The idea would be to let the user browse
the selection of packages, select some subset of them and then do
a batch install.
- add a -pipe option to pipe the output of mkisofs directly to
cdrecord (or other program) to allow direct writing to CD avoiding
the need for a potentially large amount of disk space for the
generated images.
- add an option to go through pkgsrc and find all the packages
which can't be packaged (NO_PACKAGE or INTERACTIVE) and copy
the distfiles into a single directory. This way you can create
a CD set where you include distfiles for the packages which
couldn't be made into binary pkgs.
- add an options that would search in multiple directories for
binary packages and only pick the most recent versions of any
given package. If an older version is required by some other
pkg, also pick that one up too.

154
pkgtools/cdpack/files/cddist.1 Normal file
View File

@@ -0,0 +1,154 @@
.\" $NetBSD: cddist.1,v 1.3 2009/05/13 10:43:19 wiz Exp $
.\"
.\" Copyright (c) 2008 Dan McMahill, All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" 3. All advertising materials mentioning features or use of this software
.\" must display the following acknowledgement:
.\" This product includes software developed by Dan McMahill
.\" 4. The name of the author may not be used to endorse or promote
.\" products derived from this software without specific prior written
.\" permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY DAN MCMAHILL
.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd January 28, 2008
.Dt CDDIST 1
.Os
.Sh NAME
.Nm cddist
.Nd produce multi-CD/DVD distfile archive images
.Sh SYNOPSIS
.Nm
.Op Fl dRVv
.Op Fl dvd
.Op Fl h | Fl Fl help
.Op Fl l Ar logfile
.Op Fl X Ar extras_directory
.Op Fl x Ar common_directory
.Ar distfiles_directory
.Ar image_directory
.Sh DESCRIPTION
The
.Nm
utility produces ISO 9660 image files for
a multiple CD/DVD archive of pkgsrc distfiles.
.Pp
The
.Nm
utility archives all files and directories found in the directory
.Ar distfiles_directory
specified on the command line.
The directory
.Ar cd_image_directory
is used as a staging area and the final ISO 9660 images are left in
that directory.
A file,
.Pa .index ,
which lists the contents of each CD/DVD will be left in the root
directory of each CD/DVD image.
.Bl -tag -width "-x directory "
.It Fl d
Enable debugging by not deleting temporary files at the end of the
run.
These files are left in
.Ev TMPDIR .
.It Fl dvd
Create DVD sized images rather than the default CD-ROM sized images.
.It Fl h
.It Fl Fl help
Displays a short usage message.
.It Fl l Ar logfile
Log the output of
.Ic mkisofs Fl v Fl v
to the file
.Ar logfile .
.It Fl R
Generate a file,
.Pa README.txt ,
which will be placed in the root directory of each CD/DVD image.
This README file contains a small amount of information about the
contents of the CD/DVD set.
.It Fl V
Print the program version number and exit.
.It Fl v
Be verbose.
Currently does not change things much.
.It Fl X Ar directory
Includes the contents of
.Ar directory
on the last CD/DVD in the set.
This is useful for adding
a pkgsrc.tar.gz tarball, i.e. a file which does not
need to be on each CD/DVD.
.It Fl x Ar directory
Includes the contents of
.Ar directory
on each CD/DVD.
This is useful for README files, etc., which you want
to be present on each CD/DVD in the set.
.El
.Sh EXIT STATUS
.Ex -std cddist
.Sh ENVIRONMENT
.Nm
uses the fillowing environment variables if set.
.Bl -tag -width "TMPDIR"
.It Ev AWK
awk implementation to use.
Defaults to
.Dq Pa @AWK@ .
.It Ev TMPDIR
Directory for holding temporary files during the image creation.
Defaults to
.Dq Pa /tmp .
.El
.Sh EXAMPLES
This example takes all of the distfiles found in
.Pa /usr/pkgsrc/distfiles
and creates cd images in
.Pa /u2/images/disk{1,2,...}.iso .
The directories
.Pa /u2/images/disk{1,2,...}
are used to create the ISO 9660 layouts.
.D1 cddist /usr/pkgsrc/distfiles /u2/images
.Pp
To include a common set of files on all CD/DVD's, place them in a
directory,
.Pa /tmp/foo
in this example, and pass in the directory name with the
.Fl x
option.
.Dl cddist -x /tmp/foo /usr/pkgsrc/distfiles /u2/images
.Sh SEE ALSO
.Xr cdpack 1 ,
.Xr mkisofs 1
.Sh HISTORY
The
.Nm
utility first appeared in the
.Nx
packages collection in January, 2008.
.Sh AUTHORS
The
.Nm
utility was written by
.An Dan McMahill
.Aq dmcmahill@NetBSD.org .

533
pkgtools/cdpack/files/cddist.sh Executable file
View File

@@ -0,0 +1,533 @@
#!/bin/sh
# $NetBSD: cddist.sh,v 1.2 2009/02/20 07:50:15 wiz Exp $
#
# Copyright (c) 2008 Dan McMahill, All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. All advertising materials mentioning features or use of this software
# must display the following acknowledgement:
# This product includes software developed by Dan McMahill
# 4. The name of the author may not be used to endorse or promote
# products derived from this software without specific prior written
# permission.
#
# THIS SOFTWARE IS PROVIDED BY DAN MCMAHILL
# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# $0 = program name
# $2 = distfiles directory (/usr/pkgsrc/distfiles)
# $3 = output directory
prog=cddist
progver=@progver@
TMPDIR=${TMPDIR:-/tmp}
TMP=${TMPDIR}/${prog}.$$
AWK=${AWK:-@AWK@}
EXPR="@EXPR@"
PRINTF="@PRINTF@"
SORT="@SORT@"
sizes=$TMP/sizes
sorted=$TMP/sorted
cdlist=$TMP/cdlist
indexf=$TMP/indexf
readme=$TMP/readme
warnings=$TMP/warnings
rm -fr $TMP
mkdir -m 0700 $TMP
if test $? -ne 0 ; then
echo "Could not create temporary directory $TMP"
exit 1
fi
touch $exclude
usage(){
cat << EOF
$prog - generates ISO9660 images for a multi-CD/DVD pkgsrc distfiles archive
Usage: $prog [-dRVv] [-dvd] [-h|--help] [-l logfile]
[-X dir] [-x dir] distfiles_directory cdimage_directory
Example: $prog /usr/pkgsrc/packages/distfiles /images/distfiles
Please refer to the manual page for complete documentation."
EOF
echo " "
}
clean_and_exit(){
if test $# -ge 1 ; then
rc=$1
else
rc=0
fi
if [ "x$DEBUG" = "xno" ]; then
rm -fr $TMP
else
echo "Debugging output left in $TMP"
fi
exit $rc
}
# return the full path name from a path which may
# be a full path name or a relative path name
fullpath(){
local x
x=$1
case $x in
/*) # do nothing, its an absolute path
;;
*) x=`pwd`/$x
;;
esac
echo $x
}
ADD_README=no
DEBUG=no
VERBOSE=no
VERSION=no
USE_XTRA=no
USE_OTHERS=no
mkisofslog=/dev/null
DVD=no
while
test -n "$1"
do
case "$1"
in
# enable debugging
-d) DEBUG=yes
shift
;;
# make a DVD sized image
-dvd) DVD=yes
shift
;;
# help
-h|--help) usage
exit 0
;;
# log file for the output of mkisofs -v -v
-l) mkisofslog=$2
shift 2
;;
# automatically generate a README.txt file for each CD-ROM
-R) ADD_README=yes
shift
;;
# be verbose
-v) VERBOSE=yes
shift
;;
# display version
-V) VERSION=yes
shift
;;
# extra directory to go on each CD.
-x) extra=$2
USE_XTRA=yes
shift 2
;;
# extra directory to go on only 1 CD (pkgsrc.tar.gz for example)
-X) others=$2
USE_OTHERS=yes
shift 2
;;
-*) echo "$prog: ERROR: $s is not a valid option"
usage
clean_and_exit 1
;;
*) ARGS="$ARGS $1"
shift
;;
esac
done
if [ $VERSION = "yes" ]; then
echo "$prog-$progver"
clean_and_exit 0
fi
set -- $ARGS
if [ $# -ne 2 ]; then
echo "$prog: wrong number of arguments"
usage
clean_and_exit 1
fi
distfiles=$1
cddir=$2
if [ "$VERBOSE" = "yes" ]; then
echo "Verbose output is on"
if [ "$DVD" = "yes" ]; then
echo "A DVD sized image will be created"
else
echo "A CD-ROM sized image will be created"
fi
fi
#
# Make sure all required directories and files exist
#
if [ ! -d $distfiles ]; then
echo "$prog: packages directory \"$distfiles\" does not exist"
usage
clean_and_exit 1
else
distfiles=`fullpath $distfiles`
fi
if [ ! -d $cddir ]; then
echo "$prog: cd image directory \"$cddir\" does not exist"
usage
clean_and_exit 1
else
cddir=`fullpath $cddir`
fi
if [ "$USE_XTRA" = "yes" -a ! -d $extra ]; then
echo "$prog: extra directory \"$extra\" specified with -x does not exist"
usage
clean_and_exit 1
fi
if [ "$USE_XTRA" = "yes" ]; then
extra=`fullpath $extra`
XTRA_SIZE=`du -sk $extra | ${AWK} '{print $1 * 1024}'`
if [ "$VERBOSE" = "yes" ]; then
echo "Extra directory full path name is \"$extra\". It contains $XTRA_SIZE bytes."
fi
else
XTRA_SIZE=0
fi
if [ "$USE_OTHERS" = "yes" -a ! -d $others ]; then
echo "$prog: other files directory \"$others\" specified with -X does not exist"
usage
clean_and_exit 1
fi
if [ "$USE_OTHERS" = "yes" ]; then
others=`fullpath $others`
OTHER_SIZE=`du -sk $others | ${AWK} '{print $1 * 1024}'`
if [ "$VERBOSE" = "yes" ]; then
echo "Other files directory full path name is \"$others\". It contains $OTHER_SIZE bytes."
fi
else
OTHER_SIZE=0
fi
echo " "
echo "$prog starting: `date`"
echo " "
find_size() {
d=$1
if test $# -eq 1 ; then
sz=0
else
sz=$2
fi
test "$VERBOSE" = "yes" && echo "find_size $d $sz" >/dev/stderr
szf=0
if test -f $d ; then
szf=`wc -c $d | ${AWK} '{print $1}'`
elif test -d $d ; then
for f in $d/* ; do
szf=`find_size $f $szf`
done
else
echo "WARNING: I do not know what to do with" >/dev/stderr
echo " $d">/dev/stderr
echo "which is neither a file nor a directory" >/dev/stderr
fi
tot=`${EXPR} $sz + $szf`
test "$VERBOSE" = "yes" && echo " return $tot" >/dev/stderr
echo "$tot"
}
ndist=0
echo "Extracting sizes for the distfiles ..."
rm -f $sizes
here="`pwd`"
cd $distfiles
for d in *
do
case $d in
CVS)
echo "Skip CVS directory"
;;
*)
sz=`find_size $d`
echo "$d $sz" >> $sizes
ndist=`${EXPR} $ndist + 1`
esac
done
cd "$here"
echo " "
echo "Finished extracting sizes for $ndist distfiles"
echo " "
#
# sort the distfiles alphabetically
#
${SORT} $sizes > $sorted
#
# Create directories for each image and populate the directories
# with symlinks that will be resolved when we actually create the
# ISO image
#
if [ "$DVD" = "yes" ]; then
what="DVD"
space=" "
capacity=4300000000
else
what="CD-ROM"
space=""
capacity=700000000
fi
new_image() {
cdn=`${EXPR} $cdn + 1`
dest=$cddir/disk$cdn
mkdir -p $dest/distfiles
echo "disk$cdn" >> $cdlist
cds=${XTRA_SIZE}
}
cdn=0
new_image
cds=`${EXPR} $cds + ${OTHER_SIZE}`
cat $sorted | while read f s ; do
newsize=`${EXPR} $cds + $s`
# use awk instead of test because we overflow the 32 bit
# signed values used by test when we're using a DVD image
full=`echo "$newsize $capacity" | \
${AWK} '{if($1 > $2) {print "yes"} else {print "no"}}'`
if test "$full" = "no" ; then
echo "Add $f to image #$cdn to increase the size to $newsize"
cds=$newsize
ln -sf $distfiles/$f $dest/distfiles/$f
else
new_image
cds=`${EXPR} $cds + $s`
echo "Add $f to new image #$cdn with starting size $s"
ln -sf $distfiles/$f $dest/distfiles/$f
fi
done
#
# Generate a README
#
cat <<EOF > $readme
This is @DISKNAME@ of a @NDISKS@-disk ${what} collection
containing an archive of distfiles used by the NetBSD
pkgsrc system.
For information on the NetBSD package collection, please visit
the NetBSD pkgsrc homepage at
http://www.pkgsrc.org
For more information about the NetBSD project, please visit the
project's homepage at
http://www.NetBSD.org
Please do not distribute this ${what} set unless you have verified
that the individual licenses of all the distfiles allows you to.
It is anticipated that the primary use of this ${what} set is
not for distribution, but rather for archiving. Some licenses
require that if you distribute a binary package that you are willing
to provide sources on request. By archiving the distfiles, you are
ensuring that you are able to fulfill such a request.
EOF
#
# cdpack Comment
#
cat <<EOF >> $readme
This file, along with the ${what} layout was created using the
cddist program which is available as part of the cdpack package
in the NetBSD packages collection at
ftp://ftp.NetBSD.org/pub/NetBSD/packages/pkgsrc/pkgtools/cdpack/
EOF
#
# Generate an index file which lists the contents of each CD.
#
echo "Creating Disk Index File"
show_files() {
d="$1"
path=
if test -d "$d" ; then
for fs in $d/* ; do
show_files $fs
done
else
${PRINTF} "%-60s %s\n" $d $cdname >> $indexf
fi
}
numdisks=0
for cdname in `cat ${cdlist}`
do
if [ "$VERBOSE" = "yes" ]; then
echo "Creating index for $cdname"
fi
cd ${cddir}/${cdname}/distfiles && for d in *
do
show_files $d
done
numdisks=`${EXPR} ${numdisks} + 1`
done
mv $indexf ${indexf}.tmp
${SORT} ${indexf}.tmp > $indexf
#
# Populate the cd's with the index file and readme
# actually copy these over since the originals will
# be deleted from their temporary directories
#
# Also make symlinks to the stuff in the extra directory
# specified with the -x flag.
#
if [ "$VERBOSE" = "yes" ]; then
echo "Copying .index file to the image directories."
fi
ncds=0
for cdname in `cat $cdlist`
do
if [ -f $indexf ]; then
(cd ${cddir}/${cdname} && cp $indexf .index)
fi
if [ "$ADD_README" = "yes" ]; then
if [ "$VERBOSE" = "yes" ]; then
echo "Copying README.txt file"
fi
(cd ${cddir}/${cdname} && sed -e "s/@DISKNAME@/$cdname/g" \
-e "s/@NDISKS@/${numdisks}/g" \
< $readme > README.txt)
fi
if [ $USE_XTRA = "yes" ]; then
for f in $extra/*
do
(cd ${cddir}/${cdname} && ln -sf $f)
done
fi
ncds=`${EXPR} $ncds + 1`
done
if [ "$USE_OTHERS" = "yes" ]; then
if [ "$VERBOSE" = "yes" ]; then
echo "Creating symlinks from $others (specified with -X) to ${cddir}/${cdname}"
fi
cdname=`tail -1 $cdlist`
for f in $others/*
do
(cd ${cddir}/${cdname} && ln -sf $f)
done
fi
#
# Create the ISO Images
#
volid=DistDisk
#mkisofs_flags="-f -l -r -J -allow-leading-dots -volset-size $ncds -V $volid "
mkisofs_flags="-v -v -f -l -r -J -allow-leading-dots "
echo "Creating the ISO images"
cdn=1
for cdname in `cat $cdlist`
do
echo "----------- $cdname ----------"
#mkisofs_flags2=" -volset-ID $cdname -volset-seqno $cdn "
mkisofs_flags2=" -V ${volid}$cdn "
echo "( cd ${cddir} && mkisofs $mkisofs_flags $mkisofs_flags2 -o ${cdname}.iso $cdname >> $mkisofslog 2>&1)"
( cd ${cddir} && mkisofs $mkisofs_flags $mkisofs_flags2 -o ${cdname}.iso $cdname >> $mkisofslog 2>&1)
if [ $? != 0 ]; then
echo "mkisofs failed"
clean_and_exit 1
fi
cdn=`${EXPR} $cdn + 1`
done
echo "ISO Images are available in $cddir"
if [ -f $warnings ]; then
echo "There were warnings generated:"
cat $warnings
fi
cat <<EOF
-----------------------------------------------------------------
* Please note: This ${what} set may contain files which may ${space}*
* not be redistributed. Before you distribute this set you *
* must verify that you will not violate any licenses with this *
* ${what} set. ${space}*
-----------------------------------------------------------------
$prog finished: `date`
EOF
clean_and_exit 0

556
pkgtools/cdpack/files/cdgen.awk Executable file
View File

@@ -0,0 +1,556 @@
#!@AWK@ -f
# $NetBSD: cdgen.awk,v 1.7 2009/02/20 05:16:51 dmcmahill Exp $
#
# Copyright (c) 2001, 2002, 2003, 2005, 2009 Dan McMahill, All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. All advertising materials mentioning features or use of this software
# must display the following acknowledgement:
# This product includes software developed by Dan McMahill
# 4. The name of the author may not be used to endorse or promote
# products derived from this software without specific prior written
# permission.
#
# THIS SOFTWARE IS PROVIDED BY DAN MCMAHILL
# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
BEGIN {
# all of the PKG_SUFX values which are currently allowed. This is hardcoded into
# pkg_create(1) and friends.
allsf[1] = ".tgz";
allsf[2] = ".tbz";
# ARGV[0] = program name
# ARGV[1] = packages directory (/usr/pkgsrc/packages/All)
# ARGV[2] = output directory
# ARGV[3] = Depends tree file. Has packages in tsort(1) input format.
# ARGV[4] = Exclude file. Packages listed here are excluded.
# ARGV[5] = Depends order file. Has packages in tsort(1) output format.
# ARGV[6] = CD list. This script leaves a list of the CD directories in this file
# ARGV[7] = dup flag. "dup=yes" for package duplication, "dup=no" for no duplication.
# ARGV[8] = verbose flag. "verbose=yes" for verbose output
# ARGV[9] = xtra_size. How many kB are needed per CD for common files
reqnum = 12;
if (ARGC != reqnum){
printf("cdgen.awk: wrong number of arguments (got %d, wanted %d)\n", ARGC, reqnum);
usage();
exit(1);
}
prog = "cdgen.awk";
packages = ARGV[1];
cddir = ARGV[2];
deptree = ARGV[3];
exclude = ARGV[4];
order = ARGV[5];
cdlist = ARGV[6];
xtra_size= ARGV[10];
other_size= ARGV[11];
if (ARGV[7] ~ "dup=yes"){
dup=1;
}
else{
dup=0;
}
if (ARGV[8] ~ "verbose=yes"){
verbose=1;
}
if (ARGV[9] ~ "dvd=yes"){
dvd=1;
}
else{
dvd=0;
}
#
# Make sure all required directories and files exist
#
cmd="test -d " packages ;
if(system(cmd) != 0){
printf("%s: packages directory \"%s\" does not exist\n",prog,packages);
usage();
exit(1);
}
cmd="test -d " cddir ;
if(system(cmd) != 0){
printf("%s: cd image directory \"%s\" does not exist\n",prog,cddir);
usage();
exit(1);
}
cmd="test -f " deptree ;
if(system(cmd) != 0){
printf("%s: dependstree file \"%s\" does not exist\n",prog,deptree);
usage();
exit(1);
}
cmd="test -f " exclude ;
if(system(cmd) != 0){
printf("%s: exclude file \"%s\" does not exist\n",prog,exclude);
usage();
exit(1);
}
cmd="test -f " order ;
if(system(cmd) != 0){
printf("%s: build order file \"%s\" does not exist\n",prog,order);
usage();
exit(1);
}
"date" | getline now;
printf("%s starting %28s\n",prog,now);
#
# Read in the build order. This gives the list of all possible
# packages (note that some may actually not be available as binary
# packages). The order is such that a package earlier in the list
# will never depend on a package lower on the list.
#
printf("Reading the depends order from \"%s\".\n",order);
n=1;
while(getline < order > 0){
pkgorder[n] = $1;
n++;
}
close(order);
npkgs = n-1;
printf("%d packages to go on CD-ROM/DVD!\n",npkgs);
#
# Read in the list of excluded packages
#
printf("Reading list of packages to be excluded\n");
while(getline < exclude > 0){
excludes[$1] = 1 ;
}
close(exclude);
#
# Read in the depends tree and flatten it.
#
printf("Reading depends tree\n");
while(getline < deptree > 0){
if ($2 in topdepends)
topdepends[$2] = topdepends[$2] " " $1 " " ;
else
topdepends[$2] = " " $1 " " ;
}
close(deptree);
# Now recurse the tree to give a flattened depends list for each pkg
printf("Flattening depends tree\n");
for (toppkg in topdepends){
find_all_depends(toppkg);
}
#
# Next get the sizes of each of the binary pacakges if they
# exist. For those which don't exist, make a note of that.
# getsize() returns size in kB if the file exists, 0 otherwise.
#
printf("Loading binary package sizes\n");
nonzero=0;
totsize=0;
for (n in pkgorder){
pkgsize[pkgorder[n]] = getsize(pkgorder[n]);
if(pkgsize[pkgorder[n]] > 0){
nonzero++;
}
else{
printf("WARNING: binary package \"%s\" has zero size\n",pkgorder[n]);
if(!verbose) {
verbose = 1;
getsize(pkgorder[n]);
verbose = 0;
}
}
totsize = totsize + pkgsize[pkgorder[n]];
}
printf("%d binary packages are available (%g Mb)!\n",nonzero,totsize/1024);
#
# Now we need to figure out which binary packages go on each
# CD. The way we'll do this is start taking packages in the order
# specified by 'pkgorder' until the first CD is full, then move
# to the second, etc. This way, we never have to reinsert a CD
# while installing pkgs.
#
printf("Figuring out which packages go on each CD/DVD\n");
# maximum kB for binary pkgs per CD.
if( dvd )
maxcd = 1024 * 4300;
else
maxcd = 1024 * 620;
if( verbose ) printf("Maximum image size = %g Mb\n", maxcd/1024);
#
# no package duplication
#
if (!dup){
cdn=1;
pkgn=0;
cdtot[cdn]=xtra_size;
cdpkgs[cdn]=0;
for (n=1; n<=npkgs ; n=n+1){
if (verbose) printf("Processing: %s\n",pkgorder[n]);
if (pkgorder[n] in excludes) {
if (verbose) printf("Skipping excluded package: %s\n",pkgorder[n]);
}
else {
# only process the package if it exists.
if (pkgsize[pkgorder[n]] > 0){
if (cdtot[cdn] < (maxcd-pkgsize[pkgorder[n]]) ){
cdtot[cdn] = cdtot[cdn] +pkgsize[pkgorder[n]];
cdcontents[cdn":"pkgorder[n]] = 1;
pkgn = pkgn + 1;
}
else{
# the CD is full
printf("disk number %d is full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
cdpkgs[cdn] = pkgn;
# increment the CD counter
cdn = cdn + 1;
pkgn = 1;
cdtot[cdn] = xtra_size + pkgsize[pkgorder[n]];
cdcontents[cdn":"pkgorder[n]] = 1;
}
}
}
}
cdpkgs[cdn] = pkgn;
# see if the extra files will fit on the last CD
if ( (cdtot[cdn] + other_size) < maxcd ){
printf("disk number %d is partially full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
}
else{
printf("disk number %d is full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
cdn = cdn + 1;
cdtot[cdn] = other_size;
cdpkgs[cdn] = 0;
printf("disk number %d is partially full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
}
}
#
# We will duplicate some packages to eliminate inter-CD dependencies.
#
else{
cdn=1;
pkgn=0;
# initialize the size count for the current CD with the extras that we
# are putting on all CD's
cdtot[cdn]=xtra_size;
cdpkgs[cdn]=0;
n=npkgs;
while (n > 0){
if (verbose) printf("Begin processing %s\n",pkgorder[n]);
if ( !pkg_done[pkgorder[n]]){
size_needed = pkgsize[pkgorder[n]];
deps_needed = 0;
split(alldepends[pkgorder[n]],pkgdeps);
for (dep in pkgdeps){
if (verbose) printf(" Examining dependency: %s\n",pkgdeps[dep]);
if (pkgdeps[dep] in excludes) {
if (verbose) printf(" Skipping excluded dependency in count: %s\n",pkgdeps[dep]);
}
else {
if(!cdcontents[cdn":"pkgdeps[dep]]){
size_needed = size_needed + pkgsize[pkgdeps[dep]];
deps_needed++;
}
}
}
if (cdtot[cdn] + size_needed < maxcd){
if (verbose) printf(" Processing %s\n",pkgorder[n]);
if (pkgorder[n] in excludes) {
if (verbose) printf(" Skipping excluded package in packing: %s\n",pkgorder[n]);
}
else {
cdcontents[cdn":"pkgorder[n]] = 1;
}
pkg_done[pkgorder[n]] = 1;
if (verbose) printf(" Marked %s as processed\n",pkgorder[n]);
for (dep in pkgdeps){
if (pkgdeps[dep] in excludes) {
if (verbose) printf(" Skipping excluded dependency in packing: %s\n",pkgdeps[dep]);
}
else {
cdcontents[cdn":"pkgdeps[dep]] = 1;
pkg_done[pkgdeps[dep]] = 1;
if (verbose) printf(" Marked dependency pkg %s as processed\n",pkgdeps[dep]);
}
}
cdtot[cdn] = cdtot[cdn] + size_needed;
if (pkgorder[n] in excludes) {
# don't include this one in the count if its excluded
if (verbose) printf(" Added %d dependencies to the image\n",deps_needed);
pkgn = pkgn + deps_needed;
}
else {
if (verbose) printf(" Added %s plus %d dependencies to the image\n",pkgorder[n],deps_needed);
pkgn = pkgn + 1 + deps_needed;
}
n--;
}
else{
# the CD is full
printf("disk number %d is full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
cdpkgs[cdn] = pkgn;
cdn++;
cdtot[cdn] = xtra_size;
pkgn = 0;
}
}
else{
# we've already done this pkg
if (verbose) printf(" %s has already been processed\n",pkgorder[n]);
n--;
}
}
cdpkgs[cdn] = pkgn;
# see if the extra files will fit on the last CD
if ( (cdtot[cdn] + other_size) < maxcd ){
printf("disk number %d is partially full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
}
else{
printf("disk number %d is full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
cdn = cdn + 1;
cdtot[cdn] = other_size;
cdpkgs[cdn] = 0;
printf("disk number %d is partially full (%g Mb)\n",cdn,
cdtot[cdn]/1024);
}
}
# remember how many cd's there are
ncd=cdn;
# print some stats
tot_ex=0;
for (ex in excludes){
tot_ex++;
}
if (dup){
tot_cdpkgs=0;
for (cdn=1; cdn<=ncd; cdn=cdn+1){
tot_cdpkgs = tot_cdpkgs + cdpkgs[cdn];
}
printf("CD/DVD images with package duplication resulted in %d packages total\n",tot_cdpkgs);
printf("This is an increase of %d over the base %d packages\n",tot_cdpkgs-(npkgs-tot_ex),npkgs-tot_ex);
}
printf("%d packages out of %d have been excluded due to redistribution restrictions\n",tot_ex,npkgs);
#
# Next, create a subdirectory for each CD and populate the directory
# with links to the actual binary pkgs
#
printf("Creating subdirectories for each CD/DVD and populating it with links\n");
printf("to the binary packages and other required files.\n");
for (cdn=1; cdn<=ncd; cdn=cdn+1){
printf("----------- CD/DVD #%d ----------\n",cdn);
printf(" %3d binary packages\n",cdpkgs[cdn]);
outdir=cddir "/disk" cdn "/packages/All";
cmd="test -d " outdir;
if(system(cmd) == 0){
printf("%10s: output dir \"%s\" already exists.\n",prog,outdir);
printf("%10s please remove it.\n","");
exit(1);
}
# create output directory
cmd = "mkdir -p " outdir;
do_system(cmd);
# populate it with the symlinks to the binary packages
for (n=1; n<=npkgs; n++){
if (cdcontents[cdn":"pkgorder[n]]){
for(i in allsf) {
cmd="test -f " packages "/" pkgorder[n] allsf[i];
if( (r = system(cmd)) == 0){
cmd = "cd " outdir " && ln -s " packages "/" ;
cmd = cmd pkgorder[n] allsf[i];
do_system(cmd);
}
}
}
}
# add it to the cd list
printf("disk%d\n", cdn) > cdlist ;
}
close(cdlist);
"date" | getline now;
printf("%s finished on %28s\n",prog,now);
printf("\n");
exit 0
} # BEGIN
# This is a bit of a hack, but in the event that we have both a
# foo-1.3.tgz and a foo-1.3.tbz, we currently take them both.
# I'm not sure how we avoid this. One option is a flag to
# cdpack that says to prefer the .tbz over the .tgz for example.
# To implement that, the cdpack.sh shell script would need some
# updating as well as this function. Grep for ".tgz" in
# pkgsrc/pkgutils/cdpack/files/* and also grep for allsf in this file
function getsize(name,cmd,sz,i,pkgsfx,r){
sz=0;
for(i in allsf) {
pkgsfx=allsf[i];
cmd="test -f " packages "/" name pkgsfx;
if( verbose ) {
printf("getsize(%s): Checking for suffix \"%s\" with:\n", name, pkgsfx);
printf(" %s\n", cmd);
}
if( (r = system(cmd)) == 0){
cmd="du -k " packages "/" name pkgsfx;
cmd | getline ;
close(cmd);
if( verbose ) {
printf(" \"%s\" gave %s\n", cmd, $0);
}
sz=sz + $1;
} else {
if( verbose ) {
printf(" command returned %d\n", r);
}
}
}
return(sz);
}
function do_system(cmd,rv){
rv = system(cmd);
if (rv != 0){
printf("Error: system(\"%s\") returned %d\n",cmd,rv);
exit(1);
}
return(0);
}
#
# show usage
#
function usage(){
printf("\nUsage:\n\n");
printf("%10s -- Create directories for building binary package ISO images\n",
prog,prog);
printf("%10s packages images dependstree order cdlist\n","");
printf("\n");
printf("\n");
}
function find_all_depends(pkg,pkgreg,i,deps){
# if we find the package already has been fully depended
# then return the depends list
if (pkg in alldepends){
# printf("\t%s is already depended.\n",pkg);
return(alldepends[pkg]);
}
# if we find the package listed in its own depends list, then
# return an empty list if we're going down the depends tree.
# When a package lists itself in the depends tree file, it simply
# is a place holder and means the package has no depends. However
# other packages may still depend upon it, so we need to keep looking.
if ( topdepends[pkg]~reg2str(pkg) ){
# printf("\t%s depends on its self.\n",pkg);
alldepends[pkg] = " ";
return(alldepends[pkg]);
}
# otherwise recursively gather depends that each of the depends
# has
pkgreg=reg2str(pkg);
split(topdepends[pkg],deps);
i=1;
alldepends[pkg] = " ";
while ( i in deps ){
# don't add ourselves to the list (a possibility when going up the tree)
if (" "deps[i]" "!~pkgreg){
alldepends[pkg] = alldepends[pkg] " " deps[i] " " find_all_depends(deps[i]);
}
i=i+1;
}
alldepends[pkg] = uniq(alldepends[pkg]);
return(alldepends[pkg]);
}
#
# take a string which has special characters like '+' in it and
# escape them. Also put a space before and after since that's how
# we'll distinguish things like gnome from gnome-libs
#
function reg2str(reg){
gsub(/\+/,"\\\+",reg);
gsub(/\./,"\\\.",reg);
reg = " "reg" ";
return(reg);
}
#
# take the depends lists and uniq them.
#
function uniq(list,deps,i,ulist){
# split out the depends
split(list,deps);
i=1;
ulist = " ";
while (i in deps){
if (ulist !~reg2str(deps[i])){
ulist = ulist deps[i]" ";
}
i++;
}
return(ulist);
}

242
pkgtools/cdpack/files/cdpack.1 Normal file
View File

@@ -0,0 +1,242 @@
.\" $NetBSD: cdpack.1,v 1.15 2009/02/20 07:41:15 wiz Exp $
.\"
.\" Copyright (c) 2001, 2002, 2003, 2005 Dan McMahill, All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" 3. All advertising materials mentioning features or use of this software
.\" must display the following acknowledgement:
.\" This product includes software developed by Dan McMahill
.\" 4. The name of the author may not be used to endorse or promote
.\" products derived from this software without specific prior written
.\" permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY DAN MCMAHILL
.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd February 20, 2009
.Dt CDPACK 1
.Os
.Sh NAME
.Nm cdpack
.Nd produce multi-CD/DVD binary package images
.Sh SYNOPSIS
.Nm
.Op -dnRVv
.Op -ac | -ec
.Op -af | -ef
.Op -dvd
.Op -h | --help
.Op -l logfile
.Op -m machine_architecture
.Op -o operating_system
.Op -r os_version
.Op -X extras_directory
.Op -x common_directory
.Ar packages_directory
.Ar cd_image_directory
.Sh DESCRIPTION
The
.Nm
utility produces ISO 9660 image files for
a multiple CD/DVD collection of binary packages.
.Pp
The
.Nm
utility extracts the dependency information from the binary
packages in the directory
.Ar packages_directory
specified on the command line.
The dependency information is
used to determine which packages are placed on which CD/DVD.
The directory
.Ar cd_image_directory
is used as a staging area and the final ISO 9660 images are left in
that directory.
A file,
.Pa .index ,
which lists the contents of each CD/DVD will be left in the root
directory of each CD/DVD image.
.Nm
will look for files ending both in
.Pa .tgz
as well as
.Pa .tbz
when scanning the directory
.Ar packages_directory
for binary packages.
.Bl -tag -width "-x directory "
.It Fl ac
Include binary packages marked as
.Dv NO_BIN_ON_CDROM
on the CD/DVD set.
The default is to omit any packages which were marked as
.Dv NO_BIN_ON_CDROM .
This option should
only be used when making a CD/DVD set for personal use and not for
distribution unless you are sure you have not violated any licenses.
.It Fl af
Include binary packages marked as
.Dv NO_BIN_ON_FTP
on the CD/DVD set.
This option is enabled by default.
.It Fl d
Enable debugging by not deleting temporary files at the end of the
run.
These files are left in
.Ev TMPDIR .
.It Fl dvd
Create DVD sized images rather than the default CD-ROM sized images.
.It Fl ec
Exclude packages marked as
.Dv NO_BIN_ON_CDROM
from the CD/DVD set.
This is the default.
.It Fl ef
Exclude packages marked as
.Dv NO_BIN_ON_FTP
from the CD/DVD set.
The default is to allow packages which are
.Dv NO_BIN_ON_FTP .
.It Fl h
.It Fl Fl help
Displays a short usage message.
.It Fl l Ar logfile
Log the output of
.Ic mkisofs Fl v Fl v
to the file
.Ar logfile .
.It Fl m Ar machine_architecture
Specifies
.Ar machine_architecture
as the target architecture for the binary packages in the generated
README file.
.It Fl n
Do not duplicate packages in order to prevent inter-CD/DVD dependencies.
The default without the
.Fl n
option will place a package and all of its dependencies on the same
CD/DVD.
If a particular package is also required by pacakges on other
CD/DVD's, it will be placed there as well.
When the
.Fl n
option is used packages are not duplicated.
This results in slightly smaller ISO 9660 images.
.Nm
places packages which are lower in the dependency tree on lower
numbered CD/DVD's and packages which are higher on the higher numbered
CD/DVD's.
In this fashion, a package on CD/DVD number n
will have all of its dependencies on CD/DVD numbers 1 through n
.It Fl o Ar operating_system
Specifies
.Ar operating_system
as the target operating system for the binary packages in the
generated README file.
.It Fl R
Generate a file,
.Pa README.txt ,
which will be placed in the root directory of each CD/DVD image.
This README file contains a small amount of information about the contents
of the CD/DVD set.
.It Fl r Ar os_version
Specifies
.Ar os_version
as the target operating system version for the binary packages in
the generated README file.
.It Fl V
Print the program version number and exit.
.It Fl v
Be verbose.
Currently does not change things much.
.It Fl X Ar directory
Includes the contents of
.Ar directory
on the last CD/DVD in the set.
This is useful for adding
a pkgsrc.tar.gz tarball, i.e. a file which does not
need to be on each CD/DVD.
.It Fl x Ar directory
Includes the contents of
.Ar directory
on each CD/DVD.
This is useful for README files, etc., which you want
to be present on each CD/DVD in the set.
.Sh EXIT STATUS
.Ex -std cdpack
.Sh ENVIRONMENT
.Nm
uses the fillowing environment variables if set.
.Bl -tag -width "TMPDIR"
.It Ev AWK
awk implementation to use.
Defaults to
.Dq Pa @AWK@ .
.It Ev BZCAT
bzcat implementation to use.
Defaults to
.Dq Pa @BZCAT@ .
.It Ev GZCAT
gzcat implementation to use.
Defaults to
.Dq Pa @GZCAT@ .
.It Ev TMPDIR
Directory for holding temporary files during the image creation.
Defaults to
.Dq Pa /tmp .
.El
.Sh EXAMPLES
This example takes all of the binary packages found in
.Pa /usr/pkgsrc/packages/All
and creates cd images in
.Pa /u2/cdimages/cd{1,2,...}.iso .
The directories
.Pa /u2/cdimages/cd{1,2,...}
are used to create the ISO 9660 layouts.
.D1 cdpack /usr/pkgsrc/packages/All /u2/cdimages
.Pp
To include a common set of files on all CD/DVD's, place them in a
directory,
.Pa /tmp/foo
in this example, and pass in the directory name
with the
.Fl x
option.
.Dl cdpack -x /tmp/foo /usr/pkgsrc/packages/All /u2/cdimages
.Sh SEE ALSO
.Xr cddist 1 ,
.Xr mkisofs 1
.Sh HISTORY
The
.Nm
utility first appeared in the
.Nx
packages collection in April, 2001.
.Sh AUTHORS
.An -nosplit
The
.Nm
utility was written by
.An Dan McMahill
.Aq dmcmahill@NetBSD.org .
The
idea of duplicating packages to eliminate inter-CD/DVD dependencies
came from
.An Todd Vierling
.Aq tv@NetBSD.org .

730
pkgtools/cdpack/files/cdpack.sh Executable file
View File

@@ -0,0 +1,730 @@
#!/bin/sh
# $NetBSD: cdpack.sh,v 1.14 2009/02/20 07:50:15 wiz Exp $
#
# Copyright (c) 2001, 2002, 2003, 2005, 2009 Dan McMahill, All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. All advertising materials mentioning features or use of this software
# must display the following acknowledgement:
# This product includes software developed by Dan McMahill
# 4. The name of the author may not be used to endorse or promote
# products derived from this software without specific prior written
# permission.
#
# THIS SOFTWARE IS PROVIDED BY DAN MCMAHILL
# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# $0 = program name
# $2 = packages directory (/usr/pkgsrc/packages/All)
# $3 = output directory
prog=cdpack
progver=@progver@
TMPDIR=${TMPDIR:-/tmp}
TMP=${TMPDIR}/${prog}.$$
AWK=${AWK:-@AWK@}
BZCAT=${BZCAT:-@BZCAT@}
EXPR="@EXPR@"
GZCAT=${GZCAT:-@GZCAT@}
SORT="@SORT@"
TAR="${TAR:-@TAR@}"
TSORT="@TSORT@"
depf=$TMP/depf
depf2=$TMP/depf2
deptree=$TMP/deptree
exclude=$TMP/exclude
order=$TMP/order
cdlist=$TMP/cdlist
indexf=$TMP/indexf
readme=$TMP/readme
warnings=$TMP/warnings
restricted=$TMP/restricted
rm -fr $TMP
mkdir $TMP
touch $exclude
usage(){
echo " "
echo "$prog - generates ISO9660 images for a multi-CD/DVD binary package collection"
echo "Usage: $prog [-dnRVv] [-ac | -ec] [-af | -ef] [-dvd] [-h|--help]"
echo " [-l logfile] [-m machine] [-o opsys] [-r release]"
echo " [-X dir] [-x dir] packages_directory cdimage_directory"
echo "Example: $prog /usr/pkgsrc/packages/All /images/netbsd-1.5/alpha"
echo "Please refer to the manual page for complete documentation."
echo " "
}
clean_and_exit(){
if [ "x$DEBUG" = "xno" ]; then
rm -fr $TMP
else
echo "Debugging output left in $TMP"
fi
exit 1
}
# return the full path name from a path which may
# be a full path name or a relative path name
fullpath(){
local x
x=$1
case $x in
/*) # do nothing, its an absolute path
;;
*) x=`pwd`/$x
;;
esac
echo $x
}
ADD_README=no
ARCH=`uname -m`
DEBUG=no
DUP=yes
OS=`uname -s`
RELEASE=`uname -r`
VERBOSE=no
VERSION=no
USE_XTRA=no
USE_OTHERS=no
mkisofslog=/dev/null
ALLOW_NO_BIN_ON_CDROM=no
ALLOW_NO_BIN_ON_FTP=yes
DVD=no
while
test -n "$1"
do
case "$1"
in
# allow NO_BIN_ON_CDROM packages
-ac) ALLOW_NO_BIN_ON_CDROM=yes
shift
;;
# allow NO_BIN_ON_FTP packages
-af) ALLOW_NO_BIN_ON_FTP=yes
shift
;;
# enable debugging
-d) DEBUG=yes
shift
;;
# make a DVD sized image
-dvd) DVD=yes
shift
;;
# exclude NO_BIN_ON_CDROM packages
-ec) ALLOW_NO_BIN_ON_CDROM=no
shift
;;
# exclude NO_BIN_ON_FTP packages
-ef) ALLOW_NO_BIN_ON_FTP=no
shift
;;
# help
-h|--help) usage
exit 0
;;
# log file for the output of mkisofs -v -v
-l) mkisofslog=$2
shift 2
;;
# target machine architecture for these packages
-m) ARCH=$2
shift 2
;;
# do not duplicate packages to avoid inter-CD dependencies
-n) DUP=no
shift
;;
# target os for these packages
-o) OS=$2
shift 2
;;
# target os version for these packages
-r) RELEASE=$2
shift 2
;;
# automatically generate a README.txt file for each CD-ROM
-R) ADD_README=yes
shift
;;
# be verbose
-v) VERBOSE=yes
shift
;;
# display version
-V) VERSION=yes
shift
;;
# extra directory to go on each CD.
-x) extra=$2
USE_XTRA=yes
shift 2
;;
# extra directory to go on only 1 CD (pkgsrc.tar.gz for example)
-X) others=$2
USE_OTHERS=yes
shift 2
;;
-*) echo "$prog: ERROR: $s is not a valid option"
usage
clean_and_exit
;;
*) ARGS="$ARGS $1"
shift
;;
esac
done
if [ $VERSION = "yes" ]; then
echo "$prog-$progver"
rm -fr $TMP
exit 0
fi
set -- $ARGS
if [ $# -ne 2 ]; then
echo "$prog: wrong number of arguments"
usage
clean_and_exit
fi
packages=$1
cddir=$2
if [ "$VERBOSE" = "yes" ]; then
echo "Verbose output is on"
if [ "$ALLOW_NO_BIN_ON_CDROM" = "yes" ]; then
echo "NO_BIN_ON_CDROM Packages will be included in the images"
else
echo "NO_BIN_ON_CDROM Packages will be excluded in the images"
fi
if [ "$ALLOW_NO_BIN_ON_FTP" = "yes" ]; then
echo "NO_BIN_ON_FTP Packages will be included in the images"
else
echo "NO_BIN_ON_FTP Packages will be excluded in the images"
fi
if [ "$DVD" = "yes" ]; then
echo "A DVD sized image will be created"
else
echo "A CD-ROM sized image will be created"
fi
cat << EOF
The following tools will be used:
AWK = ${AWK}
BZCAT = ${BZCAT}
EXPR = ${EXPR}
GZCAT = ${GZCAT}
SORT = ${SORT}
TAR = ${TAR}
TSORT = ${TSORT}
EOF
fi
#
# Make sure all required directories and files exist
#
if [ ! -d $packages ]; then
echo "$prog: packages directory \"$packages\" does not exist"
usage
clean_and_exit
else
packages=`fullpath $packages`
fi
if [ ! -d $cddir ]; then
echo "$prog: cd image directory \"$cddir\" does not exist"
usage
clean_and_exit
else
cddir=`fullpath $cddir`
fi
if [ "$USE_XTRA" = "yes" -a ! -d $extra ]; then
echo "$prog: extra directory \"$extra\" specified with -x does not exist"
usage
clean_and_exit
fi
if [ "$USE_XTRA" = "yes" ]; then
extra=`fullpath $extra`
XTRA_SIZE=`du -sk $extra | ${AWK} '{print $1}'`
if [ "$VERBOSE" = "yes" ]; then
echo "Extra directory full path name is \"$extra\". It contains $XTRA_SIZE kB."
fi
else
XTRA_SIZE=0
fi
if [ "$USE_OTHERS" = "yes" -a ! -d $others ]; then
echo "$prog: other files directory \"$others\" specified with -X does not exist"
usage
clean_and_exit
fi
if [ "$USE_OTHERS" = "yes" ]; then
others=`fullpath $others`
OTHER_SIZE=`du -sk $others | ${AWK} '{print $1}'`
if [ "$VERBOSE" = "yes" ]; then
echo "Other files directory full path name is \"$others\". It contains $OTHER_SIZE kB."
fi
else
OTHER_SIZE=0
fi
echo " "
echo "$prog starting: `date`"
echo " "
#
# go through all the binary packages and extract their run depends and
# also any package conflicts.
#
# $deptree = this file contains a tsort(1) run depends tree using
# binary package names.
#
npkgs=0
echo "Extracting all dependency information for the binary packages..."
for pkg in $packages/*
do
case $pkg in
# this allows foo.tgz to have a foo.txt containing some readme info
# about the file to exist without throwing off cdpack.
*.txt)
continue
;;
*.tbz)
pkg_sufx=.tbz
ZCAT=${BZCAT}
;;
*.tgz)
pkg_sufx=.tgz
ZCAT=${GZCAT}
;;
esac
if [ -f $pkg ]; then
# extract the packge name
pkgname=`basename $pkg $pkg_sufx`
# extract the packing list
cat $pkg | (cd $TMP; $ZCAT - | ${TAR} --fast-read -xf - +BUILD_INFO +CONTENTS)
# extract the depends
deps=`${AWK} '/^@pkgdep/ {printf("%s ",$2)}' $TMP/+CONTENTS`
# extract any conflicts
cfls=`${AWK} '/^@pkgcfl/ {printf("%s ",$2)}' $TMP/+CONTENTS`
# check to see if we're allowed to add this package to the CD set
NO_BIN_ON_CDROM=`${AWK} -F "=" '/NO_BIN_ON_CDROM/ {print $2}' $TMP/+BUILD_INFO`
if [ ! -z "$NO_BIN_ON_CDROM" ]; then
if [ "$ALLOW_NO_BIN_ON_CDROM" = "no" ]; then
echo "EXCLUDED $pkgname: NO_BIN_ON_CDROM=$NO_BIN_ON_CDROM" >> $restricted
if [ "$VERBOSE" = "yes" ]; then
echo "Excluding $pkg because NO_BIN_ON_CDROM=$NO_BIN_ON_CDROM"
echo "$pkgname" >> $exclude
fi
else
echo "INCLUDED $pkgname: NO_BIN_ON_CDROM=$NO_BIN_ON_CDROM" >> $restricted
fi
fi
NO_BIN_ON_FTP=`${AWK} -F "=" '/NO_BIN_ON_FTP/ {print $2}' $TMP/+BUILD_INFO`
if [ ! -z "$NO_BIN_ON_FTP" ]; then
if [ "$ALLOW_NO_BIN_ON_FTP" = "no" ]; then
echo "EXCLUDED $pkgname: NO_BIN_ON_FTP=$NO_BIN_ON_FTP" >> $restricted
if [ "$VERBOSE" = "yes" ]; then
echo "Excluding $pkg because NO_BIN_ON_FTP=$NO_BIN_ON_FTP"
echo "$pkgname" >> $exclude
fi
else
echo "INCLUDED $pkgname: NO_BIN_ON_FTP=$NO_BIN_ON_FTP" >> $restricted
fi
fi
# cleanup
rm $TMP/+CONTENTS $TMP/+BUILD_INFO
# store the results
echo "$pkgname | $deps | $cfls" >> $depf
# also process all of the listed depends with
# 'pkg_admin lsbest' to handle glob patterns
bestdeps=" "
listed=no
if [ ! -z "$deps" ]; then
for dep in $deps
do
best=`/usr/sbin/pkg_admin lsbest "${packages}/${dep}"`
if [ ! -z "$best" ]; then
best=`basename $best .tgz`
best=`basename $best .tbz`
bestdeps=$bestdeps" "$best
echo "$best $pkgname" >> $deptree
listed=yes
else
echo "WARNING: $pkgname depends on \"$dep\" which was not found"
echo "WARNING: $pkgname depends on \"$dep\" which was not found" >> $warnings
fi
done
fi
if [ "$listed" = "no" ]; then
# make sure we add ourselves to the tree if we have no depends or if the
# depends were not found
echo "$pkgname $pkgname" >> $deptree
fi
echo "$pkgname | $bestdeps | $cfls | $pkg_sufx" >> $depf2
npkgs=`${EXPR} $npkgs + 1`
else
# Don't bomb out on 1 package
echo "$prog: warning: $pkg not readable"
fi
done
echo " "
echo "Finished extracting dependencies and conflicts for $npkgs binary packages"
echo " "
#
# sort the packages in dependency order
#
mv $deptree ${deptree}.bak
${SORT} -u ${deptree}.bak > ${deptree}
${TSORT} $deptree > $order
#
# Run the awk program which figures out which packages go on which CD.
# This program also creates subdirectories and populates them with
# links to the actual binary pacakges.
#
# packages = ARGV[1];
# cddir = ARGV[2];
# deptree = ARGV[3];
# exclude = ARGV[4];
# order = ARGV[5];
# cdlist = ARGV[6];
#
if [ "$VERBOSE" = "yes" ]; then
echo "${AWK} -f @prefix@/libexec/cdgen.awk $packages $cddir $deptree $exclude $order $cdlist dup=$DUP verbose=$VERBOSE dvd=$DVD $XTRA_SIZE $OTHER_SIZE"
fi
${AWK} -f @prefix@/libexec/cdgen.awk $packages $cddir $deptree $exclude $order $cdlist dup=$DUP verbose=$VERBOSE dvd=$DVD $XTRA_SIZE $OTHER_SIZE
if [ $? -ne 0 ]; then
echo "$prog: ERROR: cdgen.awk has failed"
clean_and_exit
fi
#
# Generate a README
#
if [ "$DVD" = "yes" ]; then
what="DVD"
space=" "
else
what="CD-ROM"
space=""
fi
cat <<EOF > $readme
This is @DISKNAME@ of a @NDISKS@-disk ${what} collection
containing binary packages for version ${RELEASE} of
the ${OS} operating system running on a ${ARCH} platform.
These binary packages were created with the NetBSD packages
colection (pkgsrc). For information on the NetBSD package
collection, please visit the NetBSD pkgsrc homepage at
http://www.pkgsrc.org
For more information about the NetBSD project, please visit the
project's homepage at
http://www.NetBSD.org
EOF
if [ "$DUP" = "yes" ]; then
cat <<EOF >> $readme
The packages on this ${what} have been arranged to eliminate all
inter-${what} dependencies. In other words, each package on this
${what} should have all of its dependencies (if they are allowed
to be provided on a ${what}) present on the same ${what}.
Installation of an individual package on this ${what} should not
require any swapping of media. Some packages may be duplicated on
multiple ${what}s.
EOF
else
cat <<EOF >> $readme
The packages on this ${what} have been arranged by order of their
dependencies across the entire @NDISKS@ set of ${what}s. This allows
you to make a single pass through the ${what} set when installing
any group of pacakges provided on the set, or when installing or
loading the entire collection.
EOF
fi
#
# NO_BIN_ON_CDROM Comment
#
if [ "$ALLOW_NO_BIN_ON_CDROM" = "no" ]; then
cat <<EOF >> $readme
All packages with NO_BIN_ON_CDROM set have been excluded from
this ${what} collection.
EOF
else
cat <<EOF >> $readme
This ${what} collection includes packages with NO_BIN_ON_CDROM set.
Please do not violate license agreements by selling this ${what}
without verifying that you are allowed to. A list of these
packages may be found in the ".restricted" file on this ${what}.
EOF
fi
#
# NO_BIN_ON_FTP Comment
#
if [ "$ALLOW_NO_BIN_ON_FTP" = "no" ]; then
cat <<EOF >> $readme
All packages with NO_BIN_ON_FTP set have been excluded from
this ${what} collection.
EOF
else
cat <<EOF >> $readme
This ${what} collection includes packages with NO_BIN_ON_FTP set.
Please do not violate license agreements by placing this image or
its contents on a public FTP site without verifying that you are
allowed to. A list of these packages may be found in the
".restricted" file on this ${what}.
EOF
fi
#
# cdpack Comment
#
cat <<EOF >> $readme
This file, along with the ${what} layout was created using the
cdpack program which is available as part of the NetBSD
packages collection at
ftp://ftp.NetBSD.org/pub/NetBSD/packages/pkgsrc/pkgtools/cdpack/
EOF
#
# Generate an index file which lists the contents of each CD.
#
echo "Creating Disk Index File"
numdisks=0
for cdname in `cat ${cdlist}`
do
#
# cdgen shouldn't have included any restricted pkgs, but
# make 100% sure now!
#
if [ "$VERBOSE" = "yes" ]; then
echo "Removing any leftover restricted packages from $cdname"
fi
for pkg in `cat $exclude`
do
rm -f ${cddir}/${cdname}/packages/*/$pkg
done
if [ "$VERBOSE" = "yes" ]; then
echo "Creating index for $cdname"
fi
for pkg in ${cddir}/${cdname}/packages/All/*
do
echo "`basename $pkg` $cdname" >> $indexf
done
numdisks=`${EXPR} ${numdisks} + 1`
done
mv $indexf ${indexf}.tmp
${SORT} ${indexf}.tmp > $indexf
#
# Populate the cd's with the index file and readme
# actually copy these over since the originals will
# be deleted from their temporary directories
#
# Also make symlinks to the stuff in the extra directory
# specified with the -x flag.
#
if [ "$VERBOSE" = "yes" ]; then
echo "Copying .index and .restricted files to the image directories."
fi
ncds=0
for cdname in `cat $cdlist`
do
if [ -f $indexf ]; then
(cd ${cddir}/${cdname} && cp $indexf .index)
fi
if [ -f $restricted ]; then
(cd ${cddir}/${cdname} && cp $restricted .restricted)
fi
if [ "$ADD_README" = "yes" ]; then
if [ "$VERBOSE" = "yes" ]; then
echo "Copying README.txt file"
fi
(cd ${cddir}/${cdname} && sed -e "s/@DISKNAME@/$cdname/g" \
-e "s/@NDISKS@/${numdisks}/g" \
< $readme > README.txt)
fi
if [ $USE_XTRA = "yes" ]; then
for f in $extra/*
do
(cd ${cddir}/${cdname} && ln -sf $f)
done
fi
ncds=`${EXPR} $ncds + 1`
done
if [ "$USE_OTHERS" = "yes" ]; then
if [ "$VERBOSE" = "yes" ]; then
echo "Creating symlinks from $others (specified with -X) to ${cddir}/${cdname}"
fi
cdname=`tail -1 $cdlist`
for f in $others/*
do
(cd ${cddir}/${cdname} && ln -sf $f)
done
fi
#
# Create the ISO Images
#
volid=PkgsrcDisk
#mkisofs_flags="-f -l -r -J -allow-leading-dots -volset-size $ncds -V $volid "
mkisofs_flags="-v -v -f -l -r -J -allow-leading-dots "
echo "Creating the ISO images"
cdn=1
for cdname in `cat $cdlist`
do
echo "----------- $cdname ----------"
#mkisofs_flags2=" -volset-ID $cdname -volset-seqno $cdn "
mkisofs_flags2=" -V ${volid}$cdn "
echo "( cd ${cddir} && mkisofs $mkisofs_flags $mkisofs_flags2 -o ${cdname}.iso $cdname >> $mkisofslog 2>&1)"
( cd ${cddir} && mkisofs $mkisofs_flags $mkisofs_flags2 -o ${cdname}.iso $cdname >> $mkisofslog 2>&1)
if [ $? != 0 ]; then
echo "mkisofs failed"
clean_and_exit
fi
cdn=`${EXPR} $cdn + 1`
done
echo "ISO Images are available in $cddir"
if [ -f $warnings ]; then
echo "There were warnings generated:"
cat $warnings
fi
echo "-----------------------------------------------------------"
echo "* Please note: This ${what} set was created with ${space}*"
echo "* *"
if [ "$ALLOW_NO_BIN_ON_CDROM" = "no" ]; then
echo "* - NO_BIN_ON_CDROM packages excluded. *"
else
echo "* - NO_BIN_ON_CDROM packages INCLUDED. Please *"
echo "* verify that you will not violate any licenses *"
echo "* with this ${what} set. Refer to the /.restricted ${space}*"
echo "* file which has been placed on each ${what} in the ${space}*"
echo "* set for details. *"
fi
echo "* *"
if [ "$ALLOW_NO_BIN_ON_FTP" = "no" ]; then
echo "* - NO_BIN_ON_FTP packages excluded. *"
else
echo "* - NO_BIN_ON_FTP packages INCLUDED. You should *"
echo "* not make this ${what} set available via FTP as it ${space}*"
echo "* would violate the license on one or more *"
echo "* packages. Refer to the /.restricted file *"
echo "* which has been placed on each ${what} in the ${space}*"
echo "* set for details. *"
fi
echo "* *"
echo "-----------------------------------------------------------"
echo " "
echo "$prog finished: `date`"
echo " "
if [ "x$DEBUG" = "xno" ]; then
rm -fr $TMP
else
echo "Debugging output left in $TMP"
fi
exit 0