piuparts/0000755000000000000000000000000013061276553007623 5ustar piuparts/.gitignore0000644000000000000000000000002012567171321011600 0ustar *.pyc /deps.png piuparts/CONTRIBUTING0000644000000000000000000000213112567171321011447 0ustar Contributing code to this project --------------------------------- It's helpful to track fixes or new features via wishlist bugs against the 'piuparts' package, eg with the 'reportbug' tool ('devscripts' package). The BTS will ensure the developers' mailing list piuparts-devel@lists.alioth.debian.org is notified. Patches can be submitted by mail (git format-patch, see below) or as requests to pull from a publicly-visible git repository. In either case, please make a topic branch based on the 'develop' branch. You can send patches or requests to the development list, or to the tracking bug: @bugs.debian.org. One possible workflow: git clone git://anonscm.debian.org/piuparts/piuparts.git git checkout origin/develop -b git commit -a git format-patch -M origin/develop reportbug piuparts Contributing bugs to other projects ----------------------------------- Another very useful type of contributions are filing bug reports based on piuparts runs. This is described in https://piuparts.debian.org/bug_howto.html piuparts/ChangeLog0000644000000000000000000002552012567171321011376 0ustar 2009-12-18 Holger Levsen * This changelog is obsolete since several years. See debian/changelog instead. 2005-11-12 Lars Wirzenius * Further detailed changes to be seen in the bzr commit message log, and summarized in the NEWS file for each release. 2005-10-17 Lars Wirzenius * Version 0.12. * /var/log/faillog ignored by default. * Documented that upgrade testing to experimental doesn't work. 2005-10-14 Lars Wirzenius * Version 0.11. * Summary of changes: - Checks for missing package names on command line immediately, not after creating chroot. - apt-get in the chroot gets configured to allow unauthenticated repositories. - Tweaks to list of files ignored by default. 2005-09-15 Lars Wirzenius * Version 0.10. 2005-09-15 Lars Wirzenius * piuparts.py: When creating policy-rc.d, make it executable. * piuparts.docbook: Added missing "if". Thanks, Jonas Meurer. * run-piuparts.py: Wrote. 2005-09-09 Lars Wirzenius * Version 0.9. 2005-09-08 Lars Wirzenius * piuparts.py: Added --resolv-deps to debootstrap call, so that it works with testing/sid. 2005-09-05 Lars Wirzenius * piuparts.py, piuparts.docbook: Added -k (--keep-tmpdir) option. * piuparts.py: Added some more default ignores. * piuparts.py: Remember which packages own which files before removing packages. 2005-08-31 Lars Wirzenius * piuparts.py: Made panic() work. Added feature to report which packages own the new/modifed/removed files. Requested in Debian bug 324248 by Anibal Monsalve Salazar. 2005-08-30 Lars Wirzenius * piuparts.py: Use Python 2.3's logging module instead of NIH code. Also, log everything identically to stdout and log file, since the old behavior was confusing to many people, as observed on IRC and at Debconf5. This also obsoletes -v (--verbose). * piuparts.docbook: Document that -v (--verbose) is obsolete. 2005-08-30 Lars Wirzenius * piuparts.docbook: It's , not , and you need to pay attention to warning messages, you imbecile. 2005-08-29 Lars Wirzenius * Version 0.8. 2005-08-29 Lars Wirzenius * piuparts.py: Accept matches for any part of a filename, instead of requiring that they match everything. This is less surprising to users and they can still use ^ and $ to anchor a match to the ends, if they need it. * Makefile: Ignore the fdmount temp files that cause upgrades to etch/sid fail at the moments, and re-enable upgrade tests. * piuparts.py, piuparts.docbook: Added option -t and changed the default location for temporary files and directorys (including the chroot) to be $TMPDIR (or /tmp if not set). 2005-08-25 Lars Wirzenius * Makefile: Upgrade testing from sarge to etch fails for me, because of something to do with fdutils, so I have disabled it for now. * piuparts.py: Only use the first mirror found in sources.list, which is anyway what the manual page claims we are doing. This should reduce the problems people have been having with piuparts creating broken source.lists in the chroot when people have more than a canonical Debian repository listed. * piuparts.py: Remove temporary files upon error exit. * piuparts.docbook: Use code names instead of stable/testing/unstable in the example. 2005-07-15 Lars Wirzenius * Version 0.7. 2005-07-15 Lars Wirzenius * Makefile: Moved binary to sbin since only root can usefully run it. 2005-07-13 Lars Wirzenius * piuparts.py: Create a /usr/sbin/policy-rc.d inside the chroot when it is created. Suggested by Frank Lichtenheld. * piuparts.py: Fixed a message from "FAIL:" to "PASS:". Oops. * Makefile: Commented out an explicit mirror definition so it should now use sources.list defaults. 2005-07-10 Lars Wirzenius * piuparts.py: Suggested by Frank Lichtenheld: don't run dpkg and do other work if there are no files (cleans up log file, saves a couple of seconds). * piuparts.py, piuparts.docbook: Implemented and documented a limit on the size of outputs of command piuparts runs. 2005-07-08 Lars Wirzenius * piuparts.py: When a command has been run and it failed, show the output it gave. Removed extra dashes from argument to getopt. Changed references to "dit" (the old name for the program) to "piuparts". Changed some log file lines a bit easier to grep for, and made the start of a log easier to see. Bugs reported and wishes expressed by Frank Lichtenheld. 2005-07-08 Lars Wirzenius * piuparts.docbook: Added notes that piuparts needs to be run as root and that options must come before other command line arguments. 2005-07-05 Lars Wirzenius * Version 0.6. 2005-07-05 Lars Wirzenius * piuparts.py: Bugfix: when removing packages from the chroot, don't remove packages being depended on before the packages doing the depending. That won't work. Any fool would know it, except me. * piuparts.py: Implemented saving of meta data for the upgrade testing between Debian releases. 2005-07-04 Lars Wirzenius * piuparts.py: Made it possible to specify components to use for a mirror. * piuparts.docbook: Documented the user visible changes. 2005-06-27 Lars Wirzenius * piuparts.py: Remove and purge dependencies before the packages that were installed, so that problems with postrm scripts using non-essential packages upon purge become evident. 2005-06-27 Lars Wirzenius * piuparts.py, piuparts.docbook: Added -n / --no-ignores option. 2005-06-27 Lars Wirzenius * piuparts.py: Made mirrors mentioned in /etc/apt/sources.list be the default mirrors. Also, --mirror can be used multiple times now. * piuparts.docbook: Documented this. * Makefile: log to tmp3.log instead of twice to tmp2.log. 2005-06-27 Lars Wirzenius * Makefile: Use -f with gzip so it doesn't complain when installing new version on top of old. 2005-06-27 Lars Wirzenius * piuparts.py: Added /etc/ld.so.conf to list of ignored files and changed exim related ignore patterns (adding exim4, for example). * piuparts.py: When purging, use --purge and not --remove to dpkg. * piuparts.py: Report version and command line arguments at startup. * piuparts.py: When calling install_purge_test, report package list when command line args are package names, instead of empty list. 2005-06-23 Lars Wirzenius * Version 0.5. 2005-06-23 Lars Wirzenius * piuparts.py: Check symlink targets as well. 2005-06-23 Lars Wirzenius * piuparts.docbook: Added --ignore-regexp to manual page. 2005-06-23 Lars Wirzenius * TODO: Added. * Makefile: Added test case for upgrading between distros. * piuparts.docbook: Rewrote DESCRIPTION and EXAMPLES sections. * piuparts.py: Added /var/cache/man/index.db and /var/log/dpkg.log to ignored files. Added regexps for more versatile ignoring and -I option. * piuparts.py: Disabled comparison of mtimes of files for now. * piuparts.py: File listings are now sorted. * piuparts.py: Added upgrade tests between Debian releases. 2005-06-20 Lars Wirzenius * piuparts.py: Refactored things more than a bit in preparation for new functionality. 2005-06-19 Lars Wirzenius * Version 0.4. 2005-06-19 Lars Wirzenius * piuparts.docbook, piuparts.py: Added -p option. * piuparts.docbook: Updated a description of what the program does. 2005-06-19 Lars Wirzenius * piuparts.docbook: Documented upgrade testing. 2005-06-19 Lars Wirzenius * piuparts.py: Added a simple form of upgrade testing. 2005-06-19 Lars Wirzenius * piuparts.py: Implemented suggestion from Frank Lichtenheld to do package purginging in two steps: first remove, then purge. This helps find postrm scripts that use things they aren't allowed to use. * README: Added naming credit to Tollef Fog Heen. 2005-06-18 Lars Wirzenius * Version 0.3. 2005-06-18 Lars Wirzenius * piuparts.py: Added /var/cache/debconf and /var/cache/debconf/passwords.dat to the list of filenames that are automatically ignored. * piuparts.py: After unpacking a base tarball, run apt-get update and clean on it, to be sure everything is up to date. 2005-06-18 Lars Wirzenius * piuparts.py: Added option -a (--apt) that lets the user install the packages to be tested via apt-get, instead of by specifying package files. * piuparts.docbook: Documented -a (--apt). 2005-06-17 Lars Wirzenius * piuparts.docbook: Added a note that piuparts is meant for people making Debian packages before they upload them. 2005-06-17 Lars Wirzenius * piuparts.py, piuparts.docbook: Added -V (--version) option. 2005-06-17 Lars Wirzenius * piuparts.py, README: Removed notice about python-apt / apt_pkg since it wasn't actually used. * piuparts.py: Changed things so that we us os.walk and os.stat to find filesystem object meta data, instead of creating a tarball and then scanning that. This makes things quite a bit faster: with a pre-generate base tarball (from pbuilder, for example), a simple installation test can now happen in less than 7 seconds wall clock time. 2005-06-14 Lars Wirzenius * Version 0.2. * Second public release. The first one was just a mention on an IRC channel, so it doesn't really count, and I didn't feel like doing change log entries before now. This version only does simple installation and removal testing, but should work at least for simple cases. piuparts/Makefile0000644000000000000000000001444513061276553011273 0ustar prefix = /usr/local sbindir = $(prefix)/sbin sharedir = $(prefix)/share mandir = $(sharedir)/man man1dir = $(mandir)/man1 man8dir = $(mandir)/man8 libdir = $(prefix)/lib docdir = $(prefix)/share/doc/piuparts/ site27 = $(libdir)/python2.7/dist-packages htdocsdir = $(sharedir)/piuparts/htdocs etcdir = $(prefix)/etc distribution=${shell dpkg-parsechangelog | sed -n 's/^Distribution: *//p'} ifeq ($(distribution),UNRELEASED) version := ${shell echo "`dpkg-parsechangelog | sed -n 's/^Version: *//p'`~`date +%Y%m%d%H%M`~`git describe --dirty`"} else version := ${shell dpkg-parsechangelog | sed -n 's/^Version: *//p'} endif # generate several scripts, conffiles, ... from templates (*.in, *.py) # by substituting placeholders SCRIPTS_TEMPLATES = $(wildcard *.in master-bin/*.in slave-bin/*.in conf/*.in) SCRIPTS_PYTHON_BINARY = $(wildcard *.py master-bin/*.py slave-bin/*.py) SCRIPTS_GENERATED = $(SCRIPTS_TEMPLATES:.in=) $(SCRIPTS_PYTHON_BINARY:.py=) DOCS_GENERATED = piuparts.1 piuparts.1.html piuparts_slave_run.8 piuparts_slave_join.8 piuparts_slave_stop.8 README.html README_server.html define placeholder_substitution sed -r \ -e 's/__PIUPARTS_VERSION__/$(version)/g' \ -e 's%@sharedir@%$(sharedir)%g' \ $< > $@ endef %: %.in Makefile $(placeholder_substitution) %: %.py Makefile $(placeholder_substitution) all: build python_scripts = $(wildcard *.py piupartslib/*.py master-bin/*.py slave-bin/*.py) python-syntax-check: @set -e -x; $(foreach py,$(python_scripts),python -m py_compile $(py);) $(RM) $(python_scripts:=c) build: build-stamp build-stamp: $(SCRIPTS_GENERATED) $(DOCS_GENERATED) Makefile $(MAKE) -C instances $(MAKE) python-syntax-check touch $@ build-doc: $(DOCS_GENERATED) README.html: README.txt a2x --copy -a toc -a toclevels=3 -f xhtml -r /etc/asciidoc/ README.txt README_server.html: README_server.txt a2x --copy -a toc -a toclevels=3 -f xhtml -r /etc/asciidoc/ README_server.txt piuparts.1: piuparts.1.txt a2x -f manpage piuparts.1.txt piuparts_slave_run.8: piuparts_slave_run.8.txt a2x -f manpage piuparts_slave_run.8.txt piuparts_slave_join.8: piuparts_slave_join.8.txt a2x -f manpage piuparts_slave_join.8.txt piuparts_slave_stop.8: piuparts_slave_stop.8.txt a2x -f manpage piuparts_slave_stop.8.txt piuparts.1.html: piuparts.1.txt a2x --copy -f xhtml piuparts.1.txt install-doc: build-stamp install -d $(DESTDIR)$(docdir)/ install -m 0644 README.txt README.html README_server.txt README_server.html docbook-xsl.css $(DESTDIR)$(docdir)/ install -d $(DESTDIR)$(man1dir) install -m 0644 piuparts.1 $(DESTDIR)$(man1dir)/ install -d $(DESTDIR)$(man8dir) install -m 0644 piuparts_slave_run.8 piuparts_slave_join.8 piuparts_slave_stop.8 $(DESTDIR)$(man8dir)/ gzip -9f $(DESTDIR)$(man1dir)/piuparts.1 gzip -9f $(DESTDIR)$(man8dir)/piuparts_slave_run.8 gzip -9f $(DESTDIR)$(man8dir)/piuparts_slave_join.8 gzip -9f $(DESTDIR)$(man8dir)/piuparts_slave_stop.8 install -m 0644 piuparts.1.html $(DESTDIR)$(docdir)/ install-conf: build-stamp install -d $(DESTDIR)$(etcdir)/piuparts install -m 0644 conf/piuparts.conf.sample $(DESTDIR)$(etcdir)/piuparts/piuparts.conf install -m 0644 conf/distros.conf $(DESTDIR)$(etcdir)/piuparts/ install -d $(DESTDIR)$(etcdir)/apache2/conf-available install -m 0644 conf/piuparts-master.conf $(DESTDIR)$(etcdir)/apache2/conf-available/ install-conf-4-running-from-git: build-stamp install -d $(DESTDIR)$(etcdir)/piuparts install -m 0644 conf/crontab-master $(DESTDIR)$(etcdir)/piuparts/ install -m 0644 conf/crontab-slave $(DESTDIR)$(etcdir)/piuparts/ install -m 0644 conf/distros.conf $(DESTDIR)$(etcdir)/piuparts/ install -m 0644 instances/piuparts.conf.* $(DESTDIR)$(etcdir)/piuparts/ install -d $(DESTDIR)$(sharedir)/piuparts/slave install -m 0755 update-piuparts-slave-setup $(DESTDIR)$(sharedir)/piuparts/slave/ install -d $(DESTDIR)$(sharedir)/piuparts/master install -m 0755 update-piuparts-master-setup $(DESTDIR)$(sharedir)/piuparts/master/ install: build-stamp install -d $(DESTDIR)$(sbindir) install -m 0755 piuparts $(DESTDIR)$(sbindir)/ install -d $(DESTDIR)$(sharedir)/piuparts install -m 0755 piuparts-slave piuparts-master piuparts-master-backend piuparts-report piuparts-analyze $(DESTDIR)$(sharedir)/piuparts/ install -d $(DESTDIR)$(site27)/piupartslib install -m 0644 piupartslib/*.py $(DESTDIR)$(site27)/piupartslib/ install -d $(DESTDIR)$(sharedir)/piuparts/lib install -m 0644 lib/*.sh $(DESTDIR)$(sharedir)/piuparts/lib/ # do not install the templates (*.in, *.py) install -d $(DESTDIR)$(sharedir)/piuparts/master install -m 0755 $(filter-out %.in %.py,$(wildcard master-bin/*)) $(DESTDIR)$(sharedir)/piuparts/master/ install -d $(DESTDIR)$(sharedir)/piuparts/known_problems install -m 0644 known_problems/*.conf $(DESTDIR)$(sharedir)/piuparts/known_problems/ # do not install the templates (*.in, *.py) install -d $(DESTDIR)$(sharedir)/piuparts/slave install -m 0755 $(filter-out %.in %.py,$(wildcard slave-bin/*)) $(DESTDIR)$(sharedir)/piuparts/slave/ install -d $(DESTDIR)$(htdocsdir) install -m 0644 htdocs/*.* $(DESTDIR)$(htdocsdir)/ install -d $(DESTDIR)$(htdocsdir)/images install -m 0644 htdocs/images/*.* $(DESTDIR)$(htdocsdir)/images/ install -d $(DESTDIR)$(htdocsdir)/templates/mail install -m 0644 bug-templates/*.mail $(DESTDIR)$(htdocsdir)/templates/mail/ install -d $(DESTDIR)$(etcdir)/piuparts @set -e -x ; \ for d in $$(ls custom-scripts) ; do \ install -d $(DESTDIR)$(etcdir)/piuparts/$$d ; \ install -m 0755 custom-scripts/$$d/* $(DESTDIR)$(etcdir)/piuparts/$$d/ ; done #install -d $(DESTDIR)$(etcdir)/piuparts/known_problems #install -m 0644 known_problems/*.conf $(DESTDIR)$(etcdir)/piuparts/known_problems/ check: nosetests --verbose clean: rm -f build-stamp rm -f $(DOCS_GENERATED) rm -f piuparts.1.xml README.xml README_server.xml docbook-xsl.css piuparts.html rm -f *.pyc piupartslib/*.pyc master-bin/*.pyc slave-bin/*.pyc tests/*.pyc rm -f $(SCRIPTS_GENERATED) $(MAKE) -C instances clean # for maintainer convenience only check-whitespace: grep -r --exclude-dir .git --exclude '*.pyc' --exclude '*.png' --exclude '*.ico' -E '\s+$$' . || echo "no trailing whitespace found" grep -r --exclude-dir .git --exclude '*.pyc' --exclude '*.png' --exclude '*.ico' -P ' \t' . || echo "no space-tab combo found" # for maintainer convenience only tg-deps: tg summary --graphviz | dot -T png -o deps.png xli deps.png & piuparts/NEWS0000644000000000000000000001465512567171321010332 0ustar NEWS file for piuparts ====================== piuparts is a tool for testing that .deb packages can be installed, upgraded, and removed without trouble. See the README file and the manual page for more information. This file summarizes the major changes, particularly user visible changes, for each release. Detailed change information can be found in bzr commit messages. This file is _deprecated_ now, see debian/NEWS.Debian instead! Version 0.20, September 22, 2006 -------------------------------- When running external commands, use subprocess.Popen so that there is no need for tricky (and therefore buggy) quoting of shell command line arguments. Version 0.19, September 8, 2006 ------------------------------- When reporting a bad symlink, show the target. Version 0.18, September 7, 2006 ------------------------------- New features Piuparts now checks for symlinks whose target does not exist. Option parsing code has been rewritten, and --help now works better. The chroot is now minimized before used: all unnecessary packages are purged. /dev/MAKEDEV, /etc/nologin, /usr/doc/cpio, /var/spool/cron added to default ignores. A version number may now begin with a + character. There was a package that did that and piuparts crashed. Version 0.17 ------------ Bug fixes The configuration files of piuparts-master/slave are now documented in the README. The Python profiler is no longer used. It used to be, but that was a leftover from development (also known as failure to read diffs before committing). When testing upgrades between distributions, piuparts now makes sure that the packages being tested are upgraded, even if it means removing an old version of a dependency. New features Piuparts now checks for processes running inside the chroot after it has installed or purged a package. Because it uses policy-rc.d to prevent any services from starting, no processes should run inside the chroot after an installation or a purge has completed. This check then finds packages that don't use invoke-rc.d to start new processes. A number of new default ignores have been added: /etc/modprobe.d, compiled versions of debconf's Python modules, papercut, ssl certificates. /proc is now mounted (and unmounted) inside the chroot. Version 0.16 ------------ Bug fixes The temporary directory for the chroot was not being removed in all cases when running apt-get failed. This has now been fixed. New features Added piuparts-analyze.py, which looks at new logs of failed tests and compares them to logs of failed tests for earlier versions of the same packages, and if so, moves them automatically around. This saves a bit of manual works. Thanks to Goswin Brederlow for the idea. When piuparts creates a chroot from a tarball (option -b), it now upgrades it before using it. A number of new entries to the default ignores list. Log files are now created with permissions that are 0666 modified with the process umask. piuparts-report.py has been optimized somewhat. Version 0.15 ------------ Bug fixes The dependency parser in piupartslib now understands < and > (they're deprecated but one or two packges still use them). It also now allows underscores in package names because of the type-handling package. Small fixes to the manual page. New features and significant user visible changes piuparts-master now understands Provides: headers. A number of new entries to the default ignores list. New option --keep-sources-list from John Wright. Version 0.14 ------------ Bug fixes Specifications for ignoring some directories were buggy and have now been fixed: /var/spool/news, /var/lib/cvs. When testing a .deb file given on the command line, if any of its dependencies were missing, the package itself would be removed instead of piuparts reporting an error. This has been fixed. The check for whether a package is marked untestable for piuparts-master now works. New features and significant user visible changes New program piuparts-report.py produces some "statistics" about packages and their status with regard to testing with piuparts-slave. The chroot is always set up for piuparts, even if it is unpacked from a tarball. This reduces problems with out-of-date chroots and with using the pbuilder base.tgz tarball. Now ignored by default: /var/lib/firebird, /var/spool/news, /var/lib/rbldns, /home/ftp. Version 0.13 ------------ Bug fixes The configuration for apt-get (in the chroot) to allow un-authenticated sources now actually works. There used to be a typo. New features and other user visible changes The old run-piuparts.py script has been replaced by a distributed system, consisting of piuparts-master.py and piuparts-slave.py, plus a reporting script piuparts-report.py. Since these are not useful for most users, they're not installed on $PATH, but in /usr/share/piuparts instead. The slave part also runs upgrade tests between Debian releases, which run-piuparts.py didn't. Some additional files are ignored by default when comparing the state of the chroot before and after package installation. piuparts/README.txt0000644000000000000000000002143413036653026011321 0ustar piuparts README --------------- Author: Lars Wirzenius Email: After reading this README you probably also want to have a look at the piuparts manpage, to learn about the available options. But read this document first! == Introduction piuparts is a tool for testing that .deb packages can be installed, upgraded, and removed without problems. The name, a variant of something suggested by Tollef Fog Heen, is short for "package installation, upgrading, and removal testing suite". piuparts is licensed under the GNU General Public License, version 2, or (at your option) any later version. https://piuparts.debian.org has been testing the Debian archive since the Lenny release in 2009, though responsible maintainers run piuparts locally before uploading packages to the archive. == How to use piuparts in 5 minutes === Basic Usage Testing your packages with piuparts is as easy as typing at the console prompt: ---- # piuparts sm_0.6-1_i386.deb ---- Note that in order to work, piuparts has to be executed as user root, so you need to be logged as root or use 'sudo'. This will create a sid chroot with debootstrap, where it'll test your package. If you want to test your package in another release, for example, testing, you can do so with: ---- # piuparts ./sm_0.6-1_i386.deb -d testing ---- By default, this will read the first mirror from your '/etc/apt/sources.list' file. If you want to specify a different mirror you can do it with the option '-m': ---- # piuparts ./sm_0.6-1_i386.deb -m http://ftp.de.debian.org/debian ---- It's possible to use -d more than once. For example, to do a first installation in stable, then upgrade to testing, then upgrade to unstable and then upgrade to the local package use this: ---- # piuparts -d stable -d testing -d unstable ./sm_0.6-1_i386.deb ---- === Some tips piuparts also has a manpage, where all available options are explained. If you use piuparts on a regular basis, waiting for it to create a chroot every time takes too much time, even if you are using a local mirror or a caching tool such as approx. Piuparts has the option of using a tarball as the contents of the initial chroot, instead of building a new one with debootstrap. A easy way to use this option is use a tarball created with pbuilder. If you are not a pbuilder user, you can create this tarball with the command (again, as root): ---- # pbuilder --create ---- then you only have to remember to update this tarball with: ---- # pbuilder --update ---- To run piuparts using this tarball: ---- # piuparts -p ./sm_0.6-1_i386.deb ---- If you want to use your own pre-made tarball: ---- # piuparts --basetgz=/path/to/my/tarball.tgz ./sm_0.6-1_i386.deb ---- Piuparts also has the option of using a tarball as the contents of the initial chroot, instead of building a new one with pbuilder. You can save a tarball for later use with the '-s' ('--save') piuparts option. Some people like this, others prefer to only have to maintain one tarball. Read the piuparts manpage about the '-p', '-b' and '-s' options While pbuilder itself supports using cdebootstrap, this is not fully supported by piuparts: You will need to use debootstrap or use the '--warn-on-debsums-errors' option for piuparts and then you will still see spurious warnings in the log. === Piuparts tests By default, piuparts does two tests: . Installation and purging test. . Installation, upgrade and purging tests. The first test installs the package in a minimal chroot, removes it and purges it. The second test installs the current version in the archive of the given packages, then upgrades to the new version (deb files given to piuparts in the input), removes and purges. If you only want to perfom the first test, you can use the option: '--no-upgrade-test' === Testing packages in the config-files-remaining state The --install-remove-install option modifies the three piuparts tests in order to test package installation while config files from a previous installation are remaining, but the package itself was removed inbetween. This exercises different code paths in the maintainer scripts. . Installation and purging test: install, remove, install again and purge. . Installation, upgrade and purging test: install the old version, remove, install the new version and purge. . Distupgrade test: install the version from the first distribution, remove, distupgrade to the last distribution, install the new version. === Analyzing piuparts results When piuparts finishes all the tests satisfactorily, you will get these lines as final output: ---- 0m39.5s INFO: PASS: All tests. 0m39.5s INFO: piuparts run ends. ---- Anyway, it is a good idea to read the whole log in order to discover possible problems that did not stop the piuparts execution. If you do not get those lines, piuparts has failed during a test. The latest lines should give you a pointer to the problem with your package. == Custom scripts with piuparts You can specify several custom scripts to be run inside piuparts. You have to store them in a directory and give it as argument to piuparts: '--scriptsdir=/dir/with/the/scripts' This option can be given multiple times. The scripts from all directories will be merged together (and later ones may overwrite earlier scripts with the same filename). By default this is *not* set to anything. Have a look at '/etc/piuparts/scripts*' to learn which scripts and script directories are shipped by the package. The script prefix determines in which step it is executed. You can run several scripts in every step, they are run in alphabetical order. The scripts need to be executable and are run *inside* the piuparts chroot and can only be shell scripts. If you want to run Python or Perl scripts, you have to install Python or Perl. The chroot where piuparts is run is minimized and does not include Perl. The variable PIUPARTS_OBJECTS is set to the packages currently being tested (seperated by spaces, if applicable) or the .changes file(s) being used. So when running in master-slave mode, it will be set to the (one) package being tested at a time. Depending on the current test, the variable PIUPARTS_TEST is set to . 'install' (installation and purging test), . 'upgrade' (installation, upgrade and purging tests) or . 'distupgrade'. During the 'upgrade' and 'distupgrade' tests, the variable PIUPARTS_PHASE is set to one of the following values: . 'install' while initially installing the packages from the repository, . 'upgrade' when upgrading to the .debs, . 'distupgrade' while reinstalling the packages after 'apt-get dist-upgrade' to ensure they were not removed accidently During the 'install' test, the PIUPARTS_PHASE variable is set to 'install'. The current distribution is available in the variable PIUPARTS_DISTRIBUTION. The following prefixes for scripts are recognized: 'post_chroot_unpack' - after the chroot has been unpacked/debootrapped. Before the chroot gets updated/dist-upgraded initially. 'post_setup_' - after the *setup* of the chroot is finished. Before metadata of the chroot is recorded for later comparison. 'pre_test_' - at the beginning of each test. After metadata of the chroot was recorded for later comparison. 'is_testable_' - before *installing* your package. If this script returns a non-zero return value, the installation of the package will be skipped, but the test will be reported as successful. Use this to flag packages that cannot be be tested with piuparts by design (e.g. usrmerge), require not publicly available external ressources (e.g. some downloader packages) or are broken beyond repair (e.g. buggy packages in archived releases). 'pre_install_' - before *installing* your package. Depending on the test, this may be run multiple times. The PIUPARTS_TEST and PIUPARTS_PHASE variables can be used to distinguish the cases. 'post_install_' - after *installing* your package and its dependencies. Depending on the test, this may be run multiple times. The PIUPARTS_TEST and PIUPARTS_PHASE variables can be used to distinguish the cases. 'pre_remove_' - before *removing* your package. 'post_remove_' - after *removing* your package. 'post_purge_' - after *purging* your package. Right before comparing the chroot with the initially recorded metadata. 'pre_distupgrade_' - before *upgrading* the chroot to the *next distribution*. The next distribution is available in the variable PIUPARTS_DISTRIBUTION_NEXT. 'post_distupgrade_' - after *upgrading* the chroot to the *next distribution*. The previous distribution is available in the variable PIUPARTS_DISTRIBUTION_PREV. === Example custom scripts: '$ cat post_install_numbers' ---- #!/bin/bash number=`dpkg -l | wc -l` echo "There are $number packages installed." exit 0 ---- '$ cat post_setup_package' ---- #!/bin/sh echo "$PIUPARTS_OBJECTS will now get tested." exit 0 ---- == Distributed testing This is described in README_server.txt. // vim: set filetype=asciidoc: piuparts/README_pejacevic.txt0000644000000000000000000001416413061276553013340 0ustar Notes about the piuparts installation on pejacevic.debian.org and it's slave(s) =============================================================================== This document describes the setup for https://piuparts.debian.org - it's used for reference for the Debian System Administrators (DSA) as well as a guide for other setting up a similar system, with the piuparts source code installed from git. For regular installations we recommend to use the piuparts-master and piuparts-slaves packages as described in /usr/share/doc/piuparts-master/README_server.txt == Installation piuparts.debian.org is a setup running on three systems: * pejacevic.debian.org, running the piuparts-master instance and an apache webserver to display the results. * piu-slave-bm-a.debian.org, running four piuparts-slave nodes to run the actual tests. * piu-slave-ubc-01.debian.org, running four piuparts-slave nodes as well. === piuparts installation from source * basically, apt-get build-dep piuparts - in reality both systems get their package configuration from git.debian.org/git/mirror/debian.org.git * pejacevic runs a webserver as well (see below for apache configuration) * Copy 'https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/update-piuparts-master-setup?h=develop' and 'https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/update-piuparts-slave-setup?h=develop' to the hosts which should be master and slave. (It's possible and has been done for a long time to run them on the same host.( Run the scripts as the piupartsm and piupartss users and clone that git repository into '/srv/piuparts.debian.org/src' in the first place. Then checkout the develop branch. * Ideally provide '/srv/piuparts.debian.org/tmp' on (a sufficiently large) tmpfs. * `sudo ln -s /srv/piuparts.debian.org/etc/piuparts /etc/piuparts` * See below for further user setup instructions. === User setup On pejacevic the piuparts-master user piupartsm needs to be created, on piu-slave-bm-a and piu-slave-ubc-01 a piupartss user is needed for the slave. Both are members of the group piuparts and '/srv/piuparts.debian.org' needs to be chmod 2775 and chown piuparts(sm):piuparts. ==== '~/bashrc' for piupartsm and piupartss Do this for the piupartsm user on pejacevic and piupartss on the slave(s): ---- piupartsm@pejacevic$ cat >> ~/.bashrc <<-EOF # added manually for piuparts umask 0002 export PATH="~/bin:\$PATH" EOF ---- ==== set up ssh pubkey authentification Then create an SSH keypair for piupartss and put it into '/etc/ssh/userkeys/piupartsm' on pejacevic, so the piupartss user can login with ssh and run only piuparts-master. Restrict it like this: ---- $ cat /etc/ssh/userkeys/piupartsm command="/srv/piuparts.debian.org/share/piuparts/piuparts-master",from="2001:41c8:1000:21::21:7,5.153.231.7",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa ... ---- === Setup sudo for the slave(s) This is actually done by DSA: ==== '/etc/sudoers' for piu-slave-bm-a and piu-slave-ubc-01 ---- # The piuparts slave needs to handle chroots. piupartss ALL = NOPASSWD: /usr/sbin/piuparts *, \ /bin/umount /srv/piuparts.debian.org/tmp/tmp*, \ /usr/bin/test -f /srv/piuparts.debian.org/tmp/tmp*, \ /usr/bin/rm -rf --one-file-system /srv/piuparts.debian.org/tmp/tmp* ---- === Apache configuration Any other webserver will do but apache is used on pejacevic (and maintained by DSA): ---- ServerName piuparts.debian.org ServerAdmin debian-admin@debian.org ErrorLog /var/log/apache2/piuparts.debian.org-error.log CustomLog /var/log/apache2/piuparts.debian.org-access.log combined DocumentRoot /srv/piuparts.debian.org/htdocs AddType text/plain .log AddDefaultCharset utf-8 HostnameLookups Off UseCanonicalName Off ServerSignature On UserDir disabled # vim:set syn=apache: ---- == Running piuparts === Updating the piuparts installation Updating the master, pejacevic.debian.org: ---- holger@pejacevic~$ sudo su - piupartsm update-piuparts-master-setup develop origin ---- Updating the slave(s), for example on piu-slave-bm-a.debian.org: ---- holger@piu-slave-bm-a~$ sudo su - piupartss update-piuparts-slave-setup develop origin ---- === Running piuparts When running piuparts in master/slave mode, the master is never run by itself, instead it is always started by the slave(s). ==== Starting and stopping the slaves Run the following script under *your* user account to start four instances of piuparts-slave on pejacevic, piuparts-master will be started automatically by the slaves. ---- holger@piu-slave-bm-a:~$ sudo -u piupartss -i slave_run ---- There are several cronjobs installed via '~piupartsm/crontab' and '~piupartss/crontab') to monitor both master and slave as well as the hosts they are running on. It's possible to kill a slave any time by pressing Ctrl-C. Pressing Ctrl-C once will wait for the current test to finish, pressing twice will abort the currently running test (which will be redone). Clean termination may take some time and can be aborted by a third Ctrl-C, but that may leave temporary directories and processes around. See the 'piuparts_slave_run (8)' manpage for more information on 'slave_run'. ==== Joining an existing slave session Run the following script under *your* user account: ---- holger@pejacevic:~$ sudo -u piupartss -i slave_join ---- See the 'piuparts_slave_join (8)' manpage for more information on 'slave_join'. === Generating reports for the website 'piuparts-report' is run daily at midnight and at noon from '~piupartsm/crontab' on pejacevic. === Cronjobs to aid problem spotting Some cronjobs to aid problem spotting reside in '~piupartsm/bin/' and are run daily by '~piupartsm/crontab'. - 'detect_network_issues' should detect failed piuparts runs due to network issues on the host. - 'detect_stale_mounts' should detect stale mountpoints (usually of /proc) from failed piuparts runs. More checks should be added as we become aware of them. == Authors Last updated: February 2017 Holger Levsen // vim: set filetype=asciidoc: piuparts/README_server.txt0000644000000000000000000005757413061276553012731 0ustar piuparts README_server ---------------------- Authors: Lars Wirzenius, Holger Levsen and Andreas Beckmann Email: === piuparts runs itself and other stuff as root WARNING: Please note that running piuparts on unknown packages is somewhat risky, to say the least. There are security implications that you want to consider. It's best to do it on machines that you don't mind wiping clean at a moment's notice, and preferably so that they don't have direct network access. You have been warned. == piuparts in master/slave mode As part of the quality assurance efforts of Debian, piuparts is run on the Debian package archive. This requires a lot of processing power, and so the work can be distributed over several hosts. There is one central machine, the master, and any number of slave machines. Each piuparts-slave instance connects to the master, via ssh, and runs the piuparts-master program to report results of packages it has tested already, and to get more work. To set this up for yourself, the following steps should suffice: === Setting up the master . Pick a machine for running the piuparts master. It cannot be a chroot, but basically any real (or properly virtualized) Debian system is good enough. . Install the package 'piuparts-master' on it. . Create an account for the master, if you install the piuparts-master package it will automatically create a 'piupartsm' user for you. . Configure '/etc/piuparts/piuparts.conf' appropriately. . Create the master and backup directories as defined in that 'piuparts.conf' and make sure master owns them. . To generate the web reports, configure your webserver as needed. If you want to use the supplied 'conf-available/piuparts-master.conf' for apache2, you will need to do two things: a.) enable it and b.) link the htdocs directory defined in 'piuparts.conf' to '/var/lib/piuparts/htdocs' (thats the DocumentRoot as defined in 'conf-available/piuparts-master.conf'). === Setting up the slave(s) . Pick one or more machines for running one or several piuparts slaves. You can use the machine which is running the master also for running a slave. It's also perfectly ok to run several slaves on a multi-core machine which has lots of IO available. . Install the package 'piuparts-slave' on it. . Configure '/etc/piuparts/piuparts.conf' appropriately - if master and slave share the machine, they also share the config file. If you want to run more than one slave on a machine, set the slave-count parameter as desired. By default one slave will be run. . Create the slave and tmp directories as defined in that 'piuparts.conf' and make sure the slave can read and write there. . Create an account for the slave. This must be different from the master account. The piuparts-slave package will create a 'piupartss' user on installation. Whether you run one or many slaves, they run with the same user. . Create an ssh keypair for the slave. No passphrase. If you installed the piuparts-slave package this was done automatically and the public key can be found in '/var/lib/piuparts/piupartss/.ssh/id_rsa.pub' . Copy the slave's public key to the master's '.ssh/authorized_keys', for an installation from packages this will be '/var/lib/piuparts/piupartsm/.ssh/authorized_keys'. The key should be restricted to only allow running 'piuparts-master' by prefixing it with 'command="/usr/share/piuparts/piuparts-master",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ' . Configure sudo to allow the slave account to run several commands as root as root without password. See the example provided in '/usr/share/doc/piuparts-slave/examples/' to learn which. . Run '/usr/bin/piuparts_slave_run' and 'piuparts_slave_join' to actually let the slave(s) run and to join their sessions. . Run '/usr/bin/piuparts_slave_stop' to stop all piuparts-slaves on a host. . The logs go into the master account, into subdirectories. === Tuning the setup The piuparts-server package installs a piuparts server along the lines of https://piuparts.debian.org/. Custome '/etc/piuparts/piuparts.conf' according to your needs, most probably you will want to re-define the 'sections' to be tested (e.g. 'sid') and also maybe use a different Debian mirror. Note that the server can place a significant load on the repository. Consider setting up a local mirror, or a caching proxy for http and apt-get, to reduce the load. Running multiple slaves on a fast host can easily saturate a 100 MBit link. Logs are stored under '/var/lib/piuparts' by default. They are stored there because they are basically the result of piuparts running. There are maintenance cron jobs defined in /usr/share/doc/piuparts-(master|slave)/examples/. In particular, piuparts-report will create static html pages, defaulting to http://localhost/piuparts to be served by any webserver. === Setup from GIT https://piuparts.debian.org has been set up directly from GIT, this is described in '/usr/share/doc/piuparts-master/README_pejacevic.txt'. == Distributed piuparts testing protocol The slave machine and the piuparts-master program communicate using a simplistic line based protocol. SSH takes care of authentication, so there is nothing in the protocol for that. The protocol is transaction based: the slave gives a command, the master program responds. Commands and responses can be simple (a single line) or long (a status line plus additional data lines). Simple commands and responses are of the following format: 'keyword arg1 arg2 arg3 ... argN' The keyword is a command or status code ("ok"), and it and the arguments are separated by spaces. An argument may not contain a space. A long command or response is deduced from the context: certain commands always include additional data, and certain commands always get a long response, if successful (error responses are always simple). The first line of a long command or response is the same as for a simple one, the additional lines are prefixed with a space, and followed by a line containing only a period. A sample session (">>" indicates what the slave sends, "<<" what the master responds with): ---- << hello >> section sid << ok >> pass liwc 1.2.3-4 >> The piuparts >> log file comes >> here >> . << ok >> reserve << ok vorbisgain 2.3-4 ---- Here the slave first reports a successful test of package liwc, version 1.2.3-4, and sends the piuparts log file for it. Then it reserves a new package to test and the master gives it vorbisgain, version 2.3-4. The communication always starts with the master saying "hello". The slave shall not speak until the master has spoken. Commands and responses in this protocol: ---- Command: section Success: ok Failure: error Failure: busy ---- Slave asks master to select the given section. This must be the very first command sent by the slave, but may be repeated later on to switch between sections. It will return "error" if the section is unknown and "busy" if it is currently processed by another master instance. If the section command fails, no other commands than "section" will be allowed until one succeeds. ---- Command: recycle Success: ok Failure: error ---- Slave asks master to enable logfile recycling mode. In this mode logfiles that have been marked for rechecking will be deleted and reissued in subsequent "reserve" commands. The "recycle" command must be issued before the first "reserve" (or "status") command. It will return "error" if no more logfiles are marked for rechecking or the command is issued too late. ---- Command: idle Success: ok ---- Slave asks master whether it remembers having no packages available at a previous "reserve" command. Returns 0 (not known to be idle or timeout expired) or the number of seconds until the master wants to recompute the package state. This command should be given after "recycle" and logfile submission, but before "reserve" or "status" commands. If the slave closes the connection without issuing a "reserve" or "status" command, the expensive Packages file parsing and status computation will be skipped. ---- Command: reserve Success: ok Failure: error ---- Slave asks master to reserve a package (a particular version of it) for the slave to test. The slave may reserve any number of packages to test. If the transaction fails, there are no more packages to test, and the slave should disconnect, wait some time and try again. ---- Command: unreserve Success: ok ---- Slave informs master it cannot test the desired version of a package and the package should be rescheduled by the master. ---- Command: pass log file contents . Success: ok ---- Slave reports that it has tested a particular version of a package and that the package passed all tests. Master records this and stores the log file somewhere suitable. ---- Command: fail log file contents . Success: ok ---- Same as "pass", but package failed one or more tests. ---- Command: untestable log file contents . Success: ok ---- Slave informs master it cannot test the desired version of a package (perhaps it went away from the mirror?). ---- Command: status Success: ok = =... ---- Slave asks master to report the number of packages in all different states. The "status" command should only be issued after all logs have been transmitted ("pass", "fail", and "untestable" commands). In all cases, if the master cannot respond with "ok" (e.g., because of a disk error storing a log file), it aborts and the connection fails. The slave may only assume the command has succeeded if the master responds with "ok". The master may likewise abort, without an error message, if the slave sends garbage, or sends too much data. == piuparts.conf configuration file piuparts-master, piuparts-slave and piuparts-report share the configuration file '/etc/piuparts/piuparts.conf'. The syntax is defined by the Python ConfigParser class, and is, briefly, like this: ---- [master] foo = bar ---- === global configuration These settings have to be placed in the [global] section and are used for all further sections. * "sections" defaults to sid and defines which sections should be processed in master-slave mode. Each section defined here has to have a section with the section specific settings explained below. The first section defined should always be sid, because the data from first section a package is in is used for the source package html report. * "basetgz-sections" is an additional list of sections that are only used to maintain the basetgz tarballs and will therefore be ignored by all scripts except piuparts-slave. This list is empty by default. * "master-host" is the host where the master exists. The slave will give this host to ssh. This option is mandatory. * "master-user" is the username of the master. The slave will log in using this username. This option is mandatory. * "master-directory" is the directory where the master keeps its files. Can be relative to the master's home directory. * "slave-directory" is the directory where the slave keeps its files. Can be relative to the slave's home directory. * "slave-count" is the number of concurrent slaves to start. Default: "1". * "output-directory" is the directory where piuparts-report places the logfiles, generated html files, charts, ... that can be served by a webserver. * "backup-directory" is the directory where the prepare_backup script will place copies of the history data needed to generate the plots. This directory should be included in system backups while the logfiles and html pages in 'master-directory' and 'output-directory' (several GB of data) are regeneratable with some effort and can be excluded from backups. By default this is undefined meaning that no backups of the history data will be made. * "web-host" is the domain name for the reporting web server. Default: "piuparts.debian.org". * "doc-root" is the location where the webserver will serve the piuparts report from. Default: "/". * "slave-load-max" specifies the system load limit when piuparts-slave will enter sleep mode. Operation will be resumed after load drops below 'slave-load-max - 1.0'. Floating point value. Defaults to 0 (= disabled). * "proxy" sets the http_proxy that will be used for fetching Packages files etc. (by master/slave/report) and .debs etc. (by piuparts). This will override a http_proxy setting in the environment. By default (no value being set) the http_proxy variable from the environment will be used (and no proxy will be used if this is not set). It is highly recommended to use a proxy running on localhost (e.g. installing squid and using a setting of "http://localhost:3128") due to the high bandwidth consumption of piuparts and repeated downloading of the same files. === section specific configuration The section specific settings will be reloaded each time a section is being run. All these keys can be specified in the [global] section, too, and will serve as defaults for all other sections (overriding the builtin defaults). * "master-command" is the command to run on master-host to start the master. Better then setting it here is actually setting it in '~piupartsm/.ssh/authorized_keys' to limit ssh access to that single command. The key should be restricted to only allow running 'piuparts-master' by prefixing it with 'command="/usr/share/piuparts/piuparts-master",no-pty,no-port-forwarding'. * "idle-sleep" is the length of time the slave should wait before querying the master again if the master didn't have any new packages to test. In seconds, so a value of 300 would mean five minutes, and that seems to be a good value when there are fairly few slaves per master. The default is 300 seconds. * "max-tgz-age" is used to specify the maximum age (in seconds) after which basesystem tarballs will be recreated. If recreation fails, the old tarball will be used again. The default is 2592000 seconds, which is 30 days. A value of 0 disables recreation. * "min-tgz-retry-delay" is used to specify the minimum time (in seconds) between attempts to recreate a tarball which was created more than "max-tgz-age" seconds ago. The default is 21600 seconds, which is 6h. * "log-file" is the name of a file to where the master should write its log messages. In the default configuration file it is "$SECTION/master.log". To disable logging, set it to "/dev/null". The global "log-file" setting (defaulting to master-error.log) is used for logging stderr output from piuparts-master. This logfile will be placed in the 'master-directory' and has the PID appended. * "piuparts-command" is the command the slave uses to start piuparts. It should include 'sudo' if necessary so that piuparts runs with sufficient priviledges to do its testing (and that means root priviledges). This command should be given in the [global] section and include all flags that are common for all sections. * "piuparts-flags" are appended to "piuparts-command" and should contain the section-specific flags. * "tmpdir" is the scratch area where piuparts will create the chroots. Note: the filesystem where this is located must not be mounted with the nodev or nosuid options. This is a mandatory setting with no default. The scripts that are monitoring this directory for leftover mountpoints and chroots only evaluate the [global] setting. * "description" is a synopsis of the test used in the report. A default description will be generated if this is not set or will be prepended (appended) if the description starts (ends) with '+'. * "mirror" tells the slave which mirror it is to use. The slave gives this to piuparts when it runs it. The URLs for Packages and Sources files will be generated from this setting, too. Default (for fetching Packages/Sources): "http://deb.debian.org/debian". * "distro" is the distribution the slave should tell piuparts to use for basic install/purge testing. It is also possible to use a "partial" distribution as defined in distros.conf. No default. If 'upgrade-test-distros' is set, this selects the distribution that will be used for getting the packages to be tested. Defaults to the last entry in 'upgrade-test-distros', but other useful settings are the first entry (to test upgrades of "disappearing" packages) or the restricted set in a partial distribution (e.g. stable to backports to testing). The special keyword "None" is used to denote that no packages are to be tested, but only the basetgz tarball will be created and refreshed regularily (for the distribution given in 'upgrade-test-distros'). This reference basetgz can be shared between several sections without being affected by their flags. * "area" is the archive area used to get the list of packages to be tested. The Packages file for this area will be loaded. The default is "main" and the possible values depend on the vendor, for Debian these are main, contrib, non-free. * "components" sets the archive areas that will be available when testing the packages selected via the "area" setting. These will be enabled in the generated sources.list. Defaults to "", which means all components will be available. A useful setting is "main" together with area = main to avoid using packages outside main. Testing packages from a 'partial' area like contrib or non-free usually requires additional or all components to be available. * "arch" is the architecture to use. Default: dpkg --print-architecture. * "chroot-tgz" is the name of the file the slave should use for the tarball containing the base chroot. The default name is generated automatically from the "distro" or "upgrade-test-distros" setting. If the tarball doesn't exist, the slave creates it. * "basetgz-directory" is the directory where "chroot-tgz" (or the automatically selected default name) is located. The default is '.'. * "upgrade-test-distros" is the space delimited list of distributions the slave should use for testing upgrades between distributions (i.e., Debian versions). Using "partial" distributions as defined in distros.conf is possible. Currently, "jessie stretch sid" is a good choice. Setting this switches from doing install/purge tests to dist-upgrade tests. Not set by default. * "max-reserved" is the maximum number of packages the slave will reserve at once. It should be large enough that the host that runs master is not unduly stressed by frequent ssh logins and running master (both of which take quite a bit of CPU cycles), yet at the same time it should not be so large that one slave grabs so many packages all other slaves just sit idle. The number obviously depends on the speed of the slave. A good value seems to be enough to let the slave test packages for about an hour before reporting results and reserving more. For a contemporary AMD64 machine with a reasonably fast disk subsystem the value 50 seems to work fine. To disable a section set this to 0. * "keep-sources-list" controls whether the slave runs piuparts with the '--keep-sources-list' option. This option does not apply to upgrade tests. The value should be "yes" or "no", with the default being "no". Use this option for dists that you need a custom sources.list for, such as "stable-proposed-updates". * "precedence" controls the order the sections are being processed by the slave. Sections with a larger precedence value will be run only if all sections with a smaller precedence value are idle, i.e. master does not have any packages that this slave could test. Sections with the same precedence value will be processed round-robin until they are all idle (or a more important section has packages to be tested). The default is 1. * "depends-sections" lists additional sections that will be searched for dependencies that are not available in the current section if that describes a partial distro. * "known-problem-directory" is the path to the directory containing definitions of known problems. Default: "${prefix}/share/piuparts/known_problems" * "debug" tells the slave whether to log debug level messages. The value should be "yes" or "no", with the default being "no". piuparts itself currently always produces debug output and there is no way to disable that. * "PYTHONPATH" (global) sets the search path to the piupartslib python modules if they are not installed in their default location in /usr. * "reschedule-untestable-days" (global) sets the rescheduling delay for untestable packages (e.g. due to unsatisfied dependencies). This is handled by the 'report_untestable_packages' script and the default is "7" days. * "reschedule-old-days" (global, section) and the following five settings define the rescheduling scheme that it performed by the 'reschedule_oldest_logs' script. Passed/failed logs that are older than reschedule-(old|fail)-days will be marked for rechecking (limited to reschedule-(old|fail)-count). Only packages that are actually testable will be reissued by piuparts-master (and the "old" log will be deleted at that time). Logs that are marked for recycling but have not been rechecked due to missing/failing dependecies will be deleted anyway if they are older than expire-(old|fail)-days. * "reschedule-old-count" (global, section) is the maximum number of passed logs that will be marked for recycling. Set to 0 to disable rescheduling passed logs. * "expire-old-days" (global, section) can be set to a value larger than 'reschedule-old-days' to delete logs older than the setting that are marked for recycling but haven't been rechecked due to failing or missing dependecies. Disabled by default ("0"). * "reschedule-fail-days" (global, section) sets the minimum age of failing logs (fail/*.log or affected/*.log) before they will be rechecked. * "reschedule-fail-count" (global, section) is the maximum number of failed logs that will be marked for recycling. Set to 0 to disable rescheduling failed logs. * "expire-fail-days" (global, section) can be set to a value larger than 'reschedule-fail-days' to delete logs older than the setting that are marked for recycling but haven't been rechecked due to failing or missing dependecies. Disabled by default ("0"). * "auto-reschedule" (section) can be set to "no" to disable rescheduling of passed and failed packages. To disable only rescheduling one of passed or failed logs, set the corresponding -count variable to zero. * "json-sections" is a space-separated list of the section/distribution names which receive test results for this section. The results, by package, are stored with this name/these names in the section and global test summary.json files. If "json-sections" is undefined, or defined as "default", piuparts will assign the section to one of "unstable", "testing", "stable", "oldstable", "experimental", or "unknown". If "json-sections" is "none", the summary will not be created. The "json-sections" name "overall" is reserved. Some of the configuration items are not required, but it is best to set them all to be sure what the configuration actually is. === piuparts.debian.org specific configuration In addition to some of the above settings the following configuration settings are used by the scripts in '~piuparts?/bin/' used to run piuparts.debian.org. They are all optional, default values are set in the scripts. * "urlbase" (global) is the base url of the webserver serving this piuparts instance. Used to provide links to logfiles in email reports. It defaults to "https://piuparts.debian.org". == Running piuparts-report as it is done for piuparts.debian.org If you want to run piuparts-report (which is only+very useful if you run piuparts in master-slave mode), you need to 'apt-get install python-rpy r-recommended r-base-dev'. For more information see link:https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/README_pejacevic.txt[https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/README_pejacevic.txt]. To generate the report on the master host run: ---- piupartsm@pejacevic:~$ /usr/share/piuparts/master/generate_daily_report ---- // vim: set filetype=asciidoc: piuparts/TODO0000644000000000000000000002016613036653026010314 0ustar Things to do for piuparts ========================= Please also check the BTS, especially the bugs with a severity higher than wishlist! And see the bottom of this file for wishlist bugs someone once expressed… for 0.7x, during the stretch development cycle: - README_server.txt: rewrite style a bit more. its super easy to setup now! - ==== piuparts.debian.org specific configuration ^^^^^^^ scripts should be the headline, not piu.d.o - move more bits from README_pejacevic.txt to README_server.txt - split out README_protocol and README_piuparts.conf? -> in piuparts.conf manpage maybe? - support multiple architectures: #762308 - piuparts-report should have a list of available arch and list packages only available on untested archs in a new state "(depends-)not-available-on-tested-archs" - master should (per default) only schedule packages which are not available on the master arch to slaves of different archs -> "schedule-evenly-to-slaves = no" - once that works, update README_pejacevic: search for "soon shall go into operation..." - piuparts-master: keep track of to whom a reservation was given - more stats and graphs: - new section stats page: - packages processed per day and section, master writes submissions.txt since 0.45 for all sections. - generate simple diagrams: number of source + binary packages in all single distros: lenny, squeeze, wheezy, jessie, stretch, sid. - graph about piuparts stati for all sections combined? (possible ignore successful) - master should create the master and backup directories, if they dont exit. If master does that remove that sentence from README_server.txt again, same with slave and tmp. - piuparts.conf.pejacevic: maybe use mirror via nfs (faster) - also test packages from security.d.o => is jessie-pu sufficient? this should quickly include these packages - maybe compress all logfiles - look for a solution to use the global debian mirror for debian-backports, too, to avoid hardcoding a specific mirror in distros.conf - if it weren't for 'slave-bin/slave_cleanup', the slave would only need rights to run "sudo piuparts" but nothing else. If we can clean this up, the sudoers.d should recommend sudo (lsof|kill|umount) for admins. - if there were real schroot support, piuparts could be used without sudo. (#708663) - use network namespaces to disable network during the tests: - < weasel> says: unshare --uts --ipc --net -- sh -c 'ip addr add 127.0.0.1/8 dev lo && ip link set dev lo up && stuff' and points to https://anonscm.debian.org/gitweb/?p=mirror/dsa-puppet.git;a=blob;f=modules/porterbox/files/dd-schroot-cmd#l104 - problem might be access to the mirror, either (bind mounted) nfs access will still work in the chroots or do as its done on the porterboxes: apt-get install -d , unshare apt-get install foo - add a sample config with all possible keys set to some useful value (like /usr/share/doc/apt/examples/configure-index.gz) - generate piuparts.1.txt automatically from piuparts.py - see this blog post for a nice howto: http://andialbrecht.wordpress.com/2009/03/17/creating-a-man-page-with-distutils-and-optparse/ - though this seems pretty complicated... maybe rather grep for parser.add_option and help= in piuparts.py ?! - requires merging all the additional infomation in piuparts.1.txt into piuarts.py - parsing piuparts --help output may be easier than parsing piuparts.py - elso: examples are duplicated in piuparts.1.txt and README.txt - check the logfiles (especially pass/) for - "Exception in thread" - java stacktraces - "Can't locate .* in @INC" - we should probably have an install test with --enable-recommends and without --warn-on-others to avoid adding artificial barriers where package subsets are configured (wheezy2jessie-rcmd is *not* the solution for this) - record file ownership by user/group name, not id new dynamic system users added to the base system may have a lower id in the reference system than after actually testing some packages e.g. group of /usr/lib/dbus-1.0/dbus-daemon-launch-helper (wheezy2jessie-pu) - report actually ignored files/patterns ROT13 encoded to be able to spot and reschedule such tests - p-r: in the section summary page, report the piuparts flags being used - p-s: report age of the basetgz being used for 0.8x and later: - install from git/Makefile: remove the need for /etc/piuparts - accept a PIUPARTS_CONF environment variable everywhere to point to a different piuparts.conf - write reportbug-like wrapper for mass bug filing (start simple, make it more sophisticated later). - rework known_problems: - use a number prefix for sorting - add title information - piuparts-report: "discover" the available known_problems, dont hardcode the list - drop _issue/_error duplication, have flags inside to indicate thether to generate _issues.tpl (pass/) and/or _error.tpl (fail/ bugged/ affected/) - rework known problems to a python-friendlier format - the templates used by update-reports.py and detect_well_known_errors should be taken from /etc/piuparts/templates/ and not be included in the python source - a redirect of http://piuparts.d.o/foo to http://p.d.o/source/f/foo.html would be nice for 0.8x and later: - find_default_debian_mirrors: - check whether find_default_debian_mirrors produces something useful if sources.list does not exist (and sources.list.d/*.list is there instead) - maybe just copy sources.list(.d/*) instead? - make it possible to call aptitude (or similar) instead of apt-get and allow to override the commandline arguments. - mount perhaps others (usbfs, sysfs, etc.) in the chroot might be a good idea because some packages might need this. - rewrite piuparts-analyze to run over all sections and cache BTS responses - "decorate" (strike-through) bug links generated by piuparts-analyze to indicate resolved state (take package version into account!) - report: - write stats about the reasons for failures, as its done with shell scripts now (piuparts-analyze.py is an existing "fragment".) - RSS feeds of logs - do more fancy R graphs, eg. also per state - link (and target) to piuparts.d.o configuration is static to pejacevic. should refer to the actual hosts configuration if running somewhere else - not sure if it's a sensible thing to to, but provide a way to turn off debugging output for piuparts.py - see http://docs.python.org/library/logging.html - commandline-switches for all programms - move shell cronjobs functionality into master, slave & report - automated testing of piuparts using an archive of known broken packages: - create archive of broken packages to provide test cases for piuparts testing. - create emacspeak-broken-dpkg-preconfigure package for broken repo. (then later put more broken packages in there and use that for testing piuparts) (Closed) wishlist bugs are listed here so it's easier to find information about them should one wants to implement support for those. Usually wishlist bugs are closed not because the feature is unwanted, but because noone is working on them… see http://blog.liw.fi/posts/wishlist-bugs for a longer rationale. piuparts: - #664727 - support cowbuilder in addition to pbuilder - #588042 - add an option to override the log file instead of append to it - #354842 - new option: remove /usr/local - #558819 - new option: check that daemon stop scripts don't fail even if the daemon doesn't run - #660896 - new option: use etckeeper and show how modified files have changed - #403648 - new test: Try unpack, then remove? - #561444 - new test: install, remove configuration files, re-install, check that configuration files don't re-appear - #561445 - new test: install, mangle configuration file, re-install, check that manglement is still there - #665016 - new test: more checks on alternatives - #810323 - new test: deconfiguring and configuring a package again - #658731 - add pattern for owned files that are always an error - #682741 - add a second set of ignoreed files and patterns for "partial purge" tests piuparts-report: - #685813 - add a single report of all failures (from all sections) ordered by age piuparts/bug-templates/0000755000000000000000000000000013061276553012374 5ustar piuparts/bug-templates/broken_symlink.mail0000644000000000000000000000044713061276553016273 0ustar To: submit@bugs.debian.org Subject: broken symlink: Package: Version: Severity: normal User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package ships (or creates) a broken symlink. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/copyright_file_missing_after_upgrade.mail0000644000000000000000000000232612762537303022673 0ustar To: submit@bugs.debian.org Subject: copyright file missing after upgrade (policy 12.5) Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, a test with piuparts revealed that your package misses the copyright file after an upgrade, which is a violation of Policy 12.5: https://www.debian.org/doc/debian-policy/ch-docs.html#s-copyrightfile After the upgrade /usr/share/doc/$PACKAGE/ is just an empty directory. This was observed on the following upgrade paths: From the attached log (scroll to the bottom...): Additional info may be available here: https://wiki.debian.org/MissingCopyrightFile Note that dpkg intentionally does not replace directories with symlinks and vice versa, you need the maintainer scripts to do this. See in particular the end of point 4 in https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-unpackphase It is recommended to use the dpkg-maintscript-helper commands 'dir_to_symlink' and 'symlink_to_dir' (available since dpkg 1.17.14) to perform the conversion, ideally using d/$PACKAGE.maintscript. Do not forget to add 'Pre-Depends: ${misc:Pre-Depends}' in d/control. See dpkg-maintscript-helper(1) and dh_installdeb(1) for details. cheers, piuparts/bug-templates/creates_system_user_in_home.mail0000644000000000000000000000176512567171321021034 0ustar To: submit@bugs.debian.org Subject: creates system user in /home Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package creates a system user with a home directory in /home/$SYSUSER, which is a policy violation. The FHS (which is part of the policy) states: "/home : User home directories (optional)" "/home is a fairly standard concept, but it is clearly a site-specific filesystem. The setup will differ from host to host. Therefore, no program should rely on this location." https://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html#HOMEUSERHOMEDIRECTORIES System users are usually placed in /var/lib. Possible problems that can arise in case /home is a remote file system: * the local root user may not be allowed to write to /home * $SYSUSER may have different UIDs on different hosts For that particular piuparts test /home has been made unusable. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/cronjob_exits_with_error_after_package_removal.mail0000644000000000000000000000054212567171321024733 0ustar To: submit@bugs.debian.org Subject: cronjob exits with error after package removal Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your packages cronjob exits with error after the package has been removed. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/cronjob_produces_output_after_package_removal.mail0000644000000000000000000000054012567171321024575 0ustar To: submit@bugs.debian.org Subject: cronjob produces output after package removal Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your packages cronjob produces output after the package has been removed. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/doesnt_use_invoke-rc.d.mail0000644000000000000000000000112212567171321017600 0ustar To: submit@bugs.debian.org Subject: doesn't use invoke-rc.d Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package starts processes where it shouldn't. This is very probably due to not using invoke-rc.d as mandated by policy 9.3.3.2. This is seriously disturbing! ;-) See https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.3.3 and /usr/share/doc/sysv-rc/README.invoke-rc.d.gz as well as /usr/share/doc/sysv-rc/README.policy-rc.d.gz From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_install.mail0000644000000000000000000000060112567171321016560 0ustar To: submit@bugs.debian.org Subject: fails to install: Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to install. As per definition of the release team this makes the package too buggy for a release, thus the severity. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_install_due_to_incorrect_dependencies_in_init.d_LSB_header.mail0000644000000000000000000000104512567171321030643 0ustar To: submit@bugs.debian.org Subject: fails to install due to incorrect dependencies in init.d LSB header Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts User: initscripts-ng-devel@lists.alioth.debian.org Usertags: incorrect-dependency Hi, during a test with piuparts I noticed your package failed to install due to incorrect dependencies in the init.d LSB header. Some Debian notes are available from at https://wiki.debian.org/LSBInitScripts From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_install_due_to_insserv_rejecting_the_script_header.mail0000644000000000000000000000066412567171321027467 0ustar To: submit@bugs.debian.org Subject: fails to install due to insserv rejecting the script header Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to install due to insserv rejecting the script header. Some notes are available from at https://wiki.debian.org/LSBInitScripts From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_install_purge_install.mail0000644000000000000000000000126312567171321021515 0ustar To: submit@bugs.debian.org Subject: fails to install, purge, and install again Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to install, remove+purge, and install again. Like a plain failure on initial install this makes the package too buggy for a release, thus the severity. This is often a problem with the home directory of a system user: the user is not deleted during purge (which is fine), but the home directory is removed. Since the user already exists on the next installation, adduser won't recreate the home. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_install_remove_distupgrade_install.mail0000644000000000000000000000131612567171321024262 0ustar To: submit@bugs.debian.org Subject: fails to install, remove, distupgrade, and install again Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to install (in 'CODENAME1'), remove (but not purge), distupgrade to 'CODENAME2', and install again. Before the second installation the package is in config-files-remaining state. The configuration is remaining from the last version that was successfully configured - which is from the previous release. Like a plain failure on initial install this makes the package too buggy for a release, thus the severity. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_install_remove_install.mail0000644000000000000000000000125712567171321021673 0ustar To: submit@bugs.debian.org Subject: fails to install, remove, and install again Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to install, remove (but not purge), and install again. Before the second installation the package is in config-files-remaining state. The configuration is remaining from the last version that was successfully configured - which is the same version that is going to be installed again. Like a plain failure on initial install this makes the package too buggy for a release, thus the severity. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_install_trying_to_overwrite_other_packages_files.mail0000644000000000000000000000077512567171321027221 0ustar To: submit@bugs.debian.org Subject: fails to install, trying to overwrite other packages files Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to install because it tries to overwrite other packages files without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_purge.mail0000644000000000000000000000134312567171321016240 0ustar To: submit@bugs.debian.org Subject: fails to purge: Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to purge. According to policy 7.2 you cannot rely on the depends being available during purge, only the essential packages are available for sure. Filing this as important because a.) it's a clear policy violation (to not clean up at purge) b.) having a piuparts clean archive is a release goal since lenny and c.) this package being piuparts buggy blocks packages depending on it from being tested by piuparts (and thus possibly the detection of more severe problems). From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_purge_-_command_deluser_adduser_in_postrm_not_found.mail0000644000000000000000000000267712567171321027564 0ustar To: submit@bugs.debian.org Subject: fails to purge - command (deluser|adduser) in postrm not found Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to purge due to a command not found. According to policy 7.2 you cannot rely on the depends being available during purge, only the essential packages are available for sure. The fix should be easy: your package is using adduser or deluser from the adduser package, which is only priority important. Using useradd or userdel from the passwd package (priority required) should fix this problem. There is ongoing discussion how to handle system users on package removal, see https://bugs.debian.org/621833 Consensus seems to be not to remove system users (to avoid reusing UIDs which could grant access to the wrong files) but to "lock" them (where "locking"/"unlocking" is not yet precisely defined). Until that has been decided it should be sufficient to have the postrm script ignore any errors from deluser: deluser ... || true Filing this as important because a.) it's a clear policy violation (to not clean up at purge) b.) having a piuparts clean archive is a release goal since lenny and c.) this package being piuparts buggy blocks packages depending on it from being tested by piuparts (and thus possibly the detection of more severe problems). From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_purge_-_command_in_postrm_not_found.mail0000644000000000000000000000143312567171321024317 0ustar To: submit@bugs.debian.org Subject: fails to purge - command in postrm not found Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to purge due to a command not found. According to policy 7.2 you cannot rely on the depends being available during purge, only the essential packages are available for sure. Filing this as important because a.) it's a clear policy violation (to not clean up at purge) b.) having a piuparts clean archive is a release goal since lenny and c.) this package being piuparts buggy blocks packages depending on it from being tested by piuparts (and thus possibly the detection of more severe problems). From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_purge_-_command_ucf_in_postrm_not_found.mail0000644000000000000000000000164712567171321025163 0ustar To: submit@bugs.debian.org Subject: fails to purge - command ucf in postrm not found Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed to purge due to a command not found. According to policy 7.2 you cannot rely on the depends being available during purge, only the essential packages are available for sure. Please see the manpages ucf(1), ucfr(1) and the example maintainer scripts under /usr/share/doc/ucf/examples/ for correct usage of ucf. Filing this as important because a.) it's a clear policy violation (to not clean up at purge) b.) having a piuparts clean archive is a release goal since lenny and c.) this package being piuparts buggy blocks packages depending on it from being tested by piuparts (and thus possibly the detection of more severe problems). From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_purge_due_to_incorrect_dependencies_in_init.d_LSB_header.mail0000644000000000000000000000104312567171321030315 0ustar To: submit@bugs.debian.org Subject: fails to purge due to incorrect dependencies in init.d LSB header Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts User: initscripts-ng-devel@lists.alioth.debian.org Usertags: incorrect-dependency Hi, during a test with piuparts I noticed your package failed to purge due to incorrect dependencies in the init.d LSB header. Some Debian notes are available from at https://wiki.debian.org/LSBInitScripts From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_remove.mail0000644000000000000000000000042412567171321016412 0ustar To: submit@bugs.debian.org Subject: fails to remove: Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to remove. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_-_trying_to_overwrite.mail0000644000000000000000000000153012567171321023163 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'DISTRO1' - trying to overwrite ... Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'DISTRO1'. It installed fine in 'DISTRO1', then the upgrade to 'DISTRO2' fails because it tries to overwrite files that are owned by other packages without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces This problem is exposed during the upgrade of the 'EXPOSER' package, not by upgrading only the buggy package itself. In order to have piuparts automatically track this problem, I'll mark this bug as Affects/Found in the exposing package(s), too. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_jessie.mail0000644000000000000000000000056612567171321021140 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'jessie': Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'jessie'. It installed fine in 'jessie', then the upgrade to 'stretch' fails. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_jessie_-_trying_to_overwrite.mail0000644000000000000000000000111412567171321025546 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'jessie' - trying to overwrite ... Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'jessie'. It installed fine in 'jessie', then the upgrade to 'stretch' fails because it tries to overwrite other packages files without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_lenny.mail0000644000000000000000000000056312567171321021000 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'lenny': Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'lenny'. It installed fine in 'lenny', then the upgrade to 'squeeze' fails. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_lenny_to_squeeze_to_wheezy.mail0000644000000000000000000000070512567171321025336 0ustar To: submit@bugs.debian.org Subject: fails to upgrade lenny -> squeeze -> wheezy Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'lenny' to 'squeeze' to 'wheezy'. It installed fine in 'lenny', and upgraded to 'squeeze' successfully, but then the upgrade to 'wheezy' failed. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_lenny_to_squeeze_to_wheezy_-_trying_to_overwrite.mail0000644000000000000000000000123412567171321031754 0ustar To: submit@bugs.debian.org Subject: fails to upgrade lenny -> squeeze -> wheezy - trying to overwrite ... Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'lenny' to 'squeeze' to 'wheezy'. It installed fine in 'lenny', and upgraded to 'squeeze' successfully, but then the upgrade to 'wheezy' failed because it tries to overwrite other packages files without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_lenny_to_squeeze_to_wheezy_to_jessie.mail0000644000000000000000000000075012567171321027402 0ustar To: submit@bugs.debian.org Subject: fails to upgrade lenny -> squeeze -> wheezy -> jessie Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'lenny' to 'squeeze' to 'wheezy' to 'jessie'. It installed fine in 'lenny', and upgraded to 'squeeze' and 'wheezy' successfully, but then the upgrade to 'jessie' failed. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_sid_-_trying_to_overwrite.mail0000644000000000000000000000113212567171321025043 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'sid' - trying to overwrite ... Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'sid' to 'experimental'. It installed fine in 'sid', then the upgrade to 'experimental' fails because it tries to overwrite other packages files without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_squeeze.mail0000644000000000000000000000057012567171321021332 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'squeeze': Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'squeeze'. It installed fine in 'squeeze', then the upgrade to 'wheezy' fails. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_squeeze_-_trying_to_overwrite.mail0000644000000000000000000000111612567171321025747 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'squeeze' - trying to overwrite ... Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'squeeze'. It installed fine in 'squeeze', then the upgrade to 'wheezy' fails because it tries to overwrite other packages files without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_squeeze_to_wheezy_to_jessie.mail0000644000000000000000000000071012762537303025472 0ustar To: submit@bugs.debian.org Subject: fails to upgrade squeeze -> wheezy -> jessie Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'squeeze' to 'wheezy' to 'jessie'. It installed fine in 'squeeze', and upgraded to 'wheezy' successfully, but then the upgrade to 'jessie' failed. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_testing.mail0000644000000000000000000000056513061276553021335 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'testing': Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'testing'. It installed fine in 'testing', then the upgrade to 'sid' fails. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_testing_-_trying_to_overwrite.mail0000644000000000000000000000111312567171321025740 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'testing' - trying to overwrite ... Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'testing'. It installed fine in 'testing', then the upgrade to 'sid' fails because it tries to overwrite other packages files without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_wheezy.mail0000644000000000000000000000056512567171321021170 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'wheezy': Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'wheezy'. It installed fine in 'wheezy', then the upgrade to 'jessie' fails. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_wheezy_-_trying_to_overwrite.mail0000644000000000000000000000111312567171321025576 0ustar To: submit@bugs.debian.org Subject: fails to upgrade from 'wheezy' - trying to overwrite ... Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'wheezy'. It installed fine in 'wheezy', then the upgrade to 'jessie' fails because it tries to overwrite other packages files without declaring a Breaks+Replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/fails_to_upgrade_from_wheezy_to_jessie_to_stretch.mail0000644000000000000000000000071012762537303025465 0ustar To: submit@bugs.debian.org Subject: fails to upgrade wheezy -> jessie -> stretch Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to upgrade from 'wheezy' to 'jessie' to 'stretch'. It installed fine in 'wheezy', and upgraded to 'jessie' successfully, but then the upgrade to 'stretch' failed. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/installs_over_existing_symlink.mail0000644000000000000000000000265412567171321021610 0ustar To: submit@bugs.debian.org Subject: directory vs. symlink conflict: Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package installs files over an existing symlink shipped or created by another package. Your package ships: but package CONFLICTOR ships: Installing something over existing symlinks is considered bad practice. See e.g. https://lists.debian.org/87ehlevcrf.fsf@windlord.stanford.edu It may break in subtle ways and dpkg cannot detect this as a problem. * Your package might silently overwrite files installed at the symlink destination by other packages. * If the package shipping the symlink decides to make the link point somewhere else (or turn it into a real directory), the files owned by your package "will be lost" somewhere in the filesystem. * Depending on installation order the problematic path will be created either as a symlink or a directory: the package installed first will "win" and all others have "lost". Note that dpkg intentionally does not replace directories with symlinks and vice versa, see in particular the end of point 4 in https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-unpackphase (Note: Adding Pre-Depends is *not* a solution.) Please move the files shipped in your package to the "real" location. From the attached log (usually somewhere in the middle...): cheers, piuparts/bug-templates/leaves_alternatives_after_purge.mail0000644000000000000000000000311612567171321021661 0ustar To: submit@bugs.debian.org Subject: leaves alternatives after purge: Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left unowned files on the system after purge, which is a violation of policy 6.8: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails The leftover files are actually alternatives that were installed by the package but have not been properly removed. While there is ongoing discussion how to remove alternatives correctly (see https://bugs.debian.org/71621 for details) the following strategy should work for regular cases: * 'postinst configure' always installs the alternative * 'prerm remove' removes the alternative * 'postrm remove' and 'postrm disappear' remove the alternative In all other cases a maintainer script is invoked (e.g. upgrade, deconfigure) the alternatives are not modified to preserve user configuration. Removing the alternative in 'prerm remove' avoids having a dangling link once the actual file gets removed, but 'prerm remove' is not called in all cases (e.g. unpacked but not configured packages or disappearing packages) so the postrm must remove the alternative again (update-alternatives gracefully handles removal of non-existing alternatives). Note that the arguments for adding and removing alternatives differ, for removal it's 'update-alternatives --remove '. Filing this as important as having a piuparts clean archive is a release goal since lenny. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/leaves_diversions_after_upgrade.mail0000644000000000000000000000115612567171321021654 0ustar To: submit@bugs.debian.org Subject: leaves diversion after upgrade from squeeze Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package fails to remove some diversions after upgrading from squeeze and removing the package afterwards. Filing this as important as having a piuparts clean archive is a release goal since lenny. From the attached log (scroll to the bottom...): The test did the following: setup minimal squeeze chroot install $package/squeeze distupgrade wheezy remove $package purge $package cheers, piuparts/bug-templates/maintainer_script_unknown_argument.mail0000644000000000000000000000077212762537303022442 0ustar To: submit@bugs.debian.org Subject: foobar called with unknown argument `foobar' Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed the maintainer scripts of your package don't support all the ways they are going to be called. See policy 6.5 at https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-mscriptsinstact From the attached log (scroll to the bottom...): Testing reinstallation in sid: Cheers, piuparts/bug-templates/modifies_conffiles.mail0000644000000000000000000000320012567171321017057 0ustar To: submit@bugs.debian.org Subject: modifies conffiles (policy 10.7.3): Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package modifies conffiles. This is forbidden by the policy, see https://www.debian.org/doc/debian-policy/ch-files.html#s-config-files 10.7.3: "[...] The easy way to achieve this behavior is to make the configuration file a conffile. [...] This implies that the default version will be part of the package distribution, and must not be modified by the maintainer scripts during installation (or at any other time)." Note that once a package ships a modified version of that conffile, dpkg will prompt the user for an action how to handle the upgrade of this modified conffile (that was not modified by the user). Further in 10.7.3: "[...] must not ask unnecessary questions (particularly during upgrades) [...]" If a configuration file is customized by a maintainer script after having asked some debconf questions, it may not be marked as a conffile. Instead a template could be installed in /usr/share and used by the postinst script to fill in the custom values and create (or update) the configuration file (preserving any user modifications!). This file must be removed during postrm purge. ucf(1) may help with these tasks. See also https://wiki.debian.org/DpkgConffileHandling In https://lists.debian.org/debian-devel/2012/09/msg00412.html and followups it has been agreed that these bugs are to be filed with severity serious. debsums reports modification of the following files, from the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/modifying_files_from_another_package.mail0000644000000000000000000000063612567171321022627 0ustar To: submit@bugs.debian.org Subject: modifying files from another package: Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package modifies files from another package in /usr. This is so wrong, I'm not even bothered to look up the part of policy this violates ;-P From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/owned_after_purge_policy_6.8_+_10.7.3.mail0000644000000000000000000000102112567171321021720 0ustar To: submit@bugs.debian.org Subject: owned files after purge (policy 6.8 + 10.7.3): Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left owned files on the system after purge, which is a violation of policy 6.8 and 10.7.3: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails https://www.debian.org/doc/debian-policy/ch-files.html#s10.7.3 From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/owned_and_unowned_files_after_purge_policy_6.8_+_10.7.3.mail0000644000000000000000000000105112567171321025466 0ustar To: submit@bugs.debian.org Subject: owned and unowned files after purge (policy 6.8 + 10.7.3): Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left owned and unowned files on the system after purge, which is a violation of policy 6.8 and 10.7.3: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails https://www.debian.org/doc/debian-policy/ch-files.html#s10.7.3 From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/package_removed_processes_still_running.mail0000644000000000000000000000077512567171321023417 0ustar To: submit@bugs.debian.org Subject: package removed, processes still running Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left processes running after the package has been removed and/or purged. In https://lists.debian.org/debian-devel/2009/08/msg00182.html and followups it has been agreed that these bugs are to be filed with severity serious. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/partial-upgrade-file-overwrite.mail0000644000000000000000000000305212567171321021257 0ustar To: submit@bugs.debian.org Subject: FOO: missing Breaks+Replaces: BAR (<< VBAR) Package: FOO Version: VFOO Severity: serious User: treinen@debian.org Usertags: edos-file-overwrite Architecture: amd64 Distribution: squeeze->wheezy (partial) upgrade Hi, automatic installation tests of packages that share a file and at the same time do not conflict by their package dependency relationships has detected the following problem: This is a serious bug as it makes installation/upgrade fail, and violates sections 7.6.1 and 10.1 of the policy. As this problem can be demonstrated during partial upgrades from squeeze to wheezy (but not within squeeze or wheezy itself), this indicates a missing or insufficiently versioned Replaces+Breaks relationship. But since this particular upgrade ordering is not forbidden by any dependency relationship, it is possible that apt (or $PACKAGE_MANAGER) will use this erroneus path on squeeze->wheezy upgrades. Here is a list of files that are known to be shared by both packages (according to the Contents files for squeeze and wheezy on amd64, which may be slightly out of sync): The following relationships are currently defined: Package: Conflicts: n/a Breaks: n/a Replaces: n/a The following relationships should be added for a clean takeover of these files (https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces): Package: Breaks: Replaces: Cheers, PS: for more information about the detection of file overwrite errors of this kind see https://qa.debian.org/dose/file-overwrites.html piuparts/bug-templates/postinst_uses_usr_share_doc.mail0000644000000000000000000000136512567171321021064 0ustar To: submit@bugs.debian.org Subject: postinst uses /usr/share/doc content (Policy 12.3): Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, a test with piuparts revealed that your package uses files from /usr/share/doc in its maintainer scripts which is a violation of Policy 12.3: "Packages must not require the existence of any files in /usr/share/doc/ in order to function." https://www.debian.org/doc/debian-policy/ch-docs.html#s12.3 These files must be moved to /usr/share/$PACKAGE and may be symlinked from /usr/share/doc/$PACKAGE. This piuparts test prevents the installation of (most) files into /usr/share/doc with 'dpkg --path-exclude=...'. From the attached log (scroll to the bottom...): Cheers, piuparts/bug-templates/prompting_due_to_modified_conffiles_which_were_not_modified_by_the_user.mail0000644000000000000000000000225512567171321032023 0ustar To: submit@bugs.debian.org Subject: prompting due to modified conffiles which were not modified by the user: Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package failed the piuparts upgrade test because dpkg detected a conffile as being modified and then prompted the user for an action. As there is no user input, this fails. But this is not the real problem, the real problem is that this prompt shows up in the first place, as there was nobody modifying this conffile at all, the package has just been installed and upgraded... This is a violation of policy 10.7.3, see https://www.debian.org/doc/debian-policy/ch-files.html#s10.7.3, which says "[These scripts handling conffiles] must not ask unnecessary questions (particularly during upgrades), and must otherwise be good citizens." https://wiki.debian.org/DpkgConffileHandling should help with figuring out how to do this properly. In https://lists.debian.org/debian-devel/2009/08/msg00675.html and followups it has been agreed that these bugs are to be filed with severity serious. From the attached log (scroll to the bottom...): cheers, ././@LongLink0000644000000000000000000000015100000000000011600 Lustar rootrootpiuparts/bug-templates/prompts_user_without_following_Debian_Configuration_Management_Specification.mailpiuparts/bug-templates/prompts_user_without_following_Debian_Configuration_Management_Specification.0000644000000000000000000000311112567171321032120 0ustar To: submit@bugs.debian.org Subject: prompts user without following Debian Configuration Management Specification Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package prompts the user badly. Prompting in maintainer scripts must be done by communicating through a program such as debconf which conforms to the Debian Configuration Management Specification, version 2 or higher. Refer to Debian Policy Manual section 3.9.1 (Prompting in maintainer scripts) for details. Quoting from https://www.debian.org/doc/debian-policy/ch-binary.html#s-maintscriptprompt ------------------------------------------------------------------------------ [ 3.9.1 Prompting in maintainer scripts ] Package maintainer scripts may prompt the user if necessary. Prompting must be done by communicating through a program, such as debconf, which conforms to the Debian Configuration Management Specification, version 2 or higher. Packages which are essential, or which are dependencies of essential packages, may fall back on another prompting method if no such interface is available when they are executed. The Debian Configuration Management Specification is included in the debconf_specification files in the debian-policy package. ------------------------------------------------------------------------------ From the attached log (scroll to the bottom...): And as it's not possible to set two usertags at the same time, this also should to be done: User: lintian-maint@debian.org Usertags: read-in-maintainer-script cheers, piuparts/bug-templates/removal_makes_files_disappear.mail0000644000000000000000000000212612567171321021275 0ustar To: submit@bugs.debian.org Subject: removal of $OFFENDER makes files disappear from $VICTIM Package: $OFFENDER Affects: $VICTIM Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts replaces-without-breaks Hi, during a test with piuparts and DOSE tools I noticed your package causes removal of files that also belong to another package. This is caused by using Replaces without corresponding Breaks. The installation sequence to reproduce this problem is apt-get install $VICTIM # (1) apt-get install $OFFENDER apt-get remove $OFFENDER # (2) The list of installed files at points (1) and (2) should be identical, but the following files have disappeared: This is a serious bug violating policy 7.6, see https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces and also see the footnote that describes this incorrect behavior https://www.debian.org/doc/debian-policy/footnotes.html#f53 The $OFFENDER package has the following relationships with $VICTIM: Conflicts: Breaks: Replaces: From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/removes_files_installed_by_other_package.mail0000644000000000000000000000212612567171321023505 0ustar To: submit@bugs.debian.org Subject: removes files that were installed by another package: Package: Affects: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package removes files that were installed by another package. The removed files were already present before the package was installed, they may have been shipped or created by a dependency. This could be a violation of policy 10.7.4 ("Sharing configuration files"), see https://www.debian.org/doc/debian-policy/ch-files.html#s10.7.4 or policy chapter 6 ("Package maintainer scripts..."), see https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html or policy 7.6 ("Overwriting files..."), see https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces If a directory is used by several packages, all should ship it as part of the package (possibly empty, using $package.dirs to create it), and no package should mkdir/rmdir it in the maintainer scripts as dpkg will take care of this. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/unhandled_symlink_to_directory_conversion.mail0000644000000000000000000000331112762537303024001 0ustar To: submit@bugs.debian.org Subject: unhandled symlink to directory conversion: /usr/share/doc/PACKAGE Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, an upgrade test with piuparts revealed that your package installs files over existing symlinks and possibly overwrites files owned by other packages. This usually means an old version of the package shipped a symlink but that was later replaced by a real (and non-empty) directory. This kind of overwriting another package's files cannot be detected by dpkg. This was observed on the following upgrade paths: For /usr/share/doc/PACKAGE this may not be problematic as long as both packages are installed, ship byte-for-byte identical files and are upgraded in lockstep. But once one of the involved packages gets removed, the other one will lose its documentation files, too, including the copyright file, which is a violation of Policy 12.5: https://www.debian.org/doc/debian-policy/ch-docs.html#s-copyrightfile For other overwritten locations anything interesting may happen. Note that dpkg intentionally does not replace directories with symlinks and vice versa, you need the maintainer scripts to do this. See in particular the end of point 4 in https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-unpackphase It is recommended to use the dpkg-maintscript-helper commands 'dir_to_symlink' and 'symlink_to_dir' (available since dpkg 1.17.14) to perform the conversion, ideally using d/$PACKAGE.maintscript. Do not forget to add 'Pre-Depends: ${misc:Pre-Depends}' in d/control. See dpkg-maintscript-helper(1) and dh_installdeb(1) for details. From the attached log (usually somewhere in the middle...): cheers, piuparts/bug-templates/uninstallable_in_sid.mail0000644000000000000000000000044712567171321017424 0ustar To: submit@bugs.debian.org Subject: uninstallable in sid Package: Version: Severity: grave Justification: renders package unusable Tags: sid User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package is no longer installable in sid: Cheers, piuparts/bug-templates/unowned_directories_in_usr_local_after_purge_policy_6.8_and_9.1.2.mail0000644000000000000000000000117112567171321027747 0ustar To: submit@bugs.debian.org Subject: unowned directories in /usr/local after purge (policy 6.8 and 9.1.2) Package: Version: Severity: normal User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left unowned files on the system after purge, which is a violation of policy 6.8: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails From the attached log (scroll to the bottom...): Leaving directories in /usr/local after purge is also a violation of a should-directive in https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.1.2 cheers, piuparts/bug-templates/unowned_directory_after_purge.mail0000644000000000000000000000150712567171321021366 0ustar To: submit@bugs.debian.org Subject: unowned directory after purge: Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left unowned directories on the system after purge, which is a violation of policy 6.8: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails Filing this as important as having a piuparts clean archive is a release goal since lenny. The maintainer scripts create (and later remove) a file in that directory. Manual directory removal may be not appropriate as this directory is shared between several packages. If the package would ship this as an empty directory, dpkg would take care of the creation and removal (if it's empty). From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/unowned_files_after_purge_policy_6.8_and_10.8.mail0000644000000000000000000000105712567171321023726 0ustar To: submit@bugs.debian.org Subject: unowned files after purge (policy 6.8, 10.8): Package: Version: Severity: important User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left unowned files on the system after purge, which is a violation of policy 6.8 (or 10.8): https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails Filing this as important as having a piuparts clean archive is a release goal since lenny. From the attached log (scroll to the bottom...): cheers, piuparts/bug-templates/unowned_files_after_purge_policy_6.8_violating_FHS_policy_9.1_too.mail0000644000000000000000000000117612567171321030043 0ustar To: submit@bugs.debian.org Subject: unowned files after purge (policy 6.8) violating FHS (policy 9.1) too: Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left unowned files on the system after purge, which is a violation of policy 6.8: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails From the attached log (scroll to the bottom...): As putting files into /usr/local is also a violation of https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.1.2 I'm setting the severity to serious. cheers, piuparts/bug-templates/unowned_files_in_usr_local_after_purge_policy_6.8.mail0000644000000000000000000000115512567171321025166 0ustar To: submit@bugs.debian.org Subject: unowned files in /usr/local after purge (policy 6.8): Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left unowned files on the system after purge, which is a violation of policy 6.8: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails From the attached log (scroll to the bottom...): As putting files into /usr/local is also a violation of https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.1.2 I'm setting the severity to serious. cheers, piuparts/bug-templates/unowned_ls-R_file_in_usr_local_after_purge.mail0000644000000000000000000000254312762537303023733 0ustar To: submit@bugs.debian.org Subject: unowned file /usr/local/share/texmf/ls-R after purge (policy 6.8, 9.1.2) Package: Version: Severity: serious User: debian-qa@lists.debian.org Usertags: piuparts Hi, during a test with piuparts I noticed your package left unowned files on the system after purge, which is a violation of policy 6.8: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails As putting files into /usr/local is also a violation of https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.1.2 I'm setting the severity to serious. From the attached log (scroll to the bottom...): 0m53.8s ERROR: FAIL: Package purging left files on system: /usr/local/share/texmf/ not owned /usr/local/share/texmf/ls-R not owned This problem is usually caused by running mktexlsr (or texhash) without path arguments from a maintainer script. The recommended solution is to switch to use dh_installtex and have this generate most (or perhaps even all) of the maintainer scripts content. Otherwise run mktexlsr with the tree as argument where the package installs its files, which is usually mktexlsr /usr/share/texmf Please have a look at the Debian-TeX-Policy (in the tex-common package) for the current practice of handling TeX packages. For further TeX packaging related questions contact debian-tex-maint@lists.debian.org cheers, piuparts/bug-templates/usertag_piuparts.mail0000644000000000000000000000017012567171321016634 0ustar To: control@bugs.debian.org Subject: usertag piuparts user debian-qa@lists.debian.org usertag 123456 + piuparts thanks piuparts/conf/0000755000000000000000000000000013061276553010550 5ustar piuparts/conf/crontab-master.in0000644000000000000000000000176113061276553014026 0ustar # m h dom mon dow (0|7=sun,1=mon) command # # cleanup $HTDOCS/daily.lock # @reboot @sharedir@/piuparts/master/master_cleanup # # generate reports twice a day # (dinstall runs 1|7|13|19:52, so this is long after mirror pushes...) # 0 0,12 * * * @sharedir@/piuparts/master/generate_daily_report # # reschedule old logs twice a day # 0 3,15 * * * @sharedir@/piuparts/master/reschedule_oldest_logs # # monitor for problems once a day # - these may result in packages being retested # - if that's not the case, run them from within generate_daily_report # 30 3-21/6 * * * @sharedir@/piuparts/master/detect_network_issues 45 3-21/6 * * * @sharedir@/piuparts/master/detect_piuparts_issues 0 22 * * * @sharedir@/piuparts/master/detect_archive_issues 30 22 * * * @sharedir@/piuparts/master/report_untestable_packages 0 23 * * * @sharedir@/piuparts/master/report_stale_reserved_packages # # misc # 0 2 * * * @sharedir@/piuparts/master/prepare_backup 55 23 * * * @sharedir@/piuparts/master/gather_bts_stats piuparts/conf/crontab-slave.in0000644000000000000000000000122012762537303013633 0ustar # m h dom mon dow (0|7=sun,1=mon) command # # start slave_run automatically after reboots # @reboot @sharedir@/piuparts/slave/slave_cleanup ; @sharedir@/piuparts/slave/slave_run 42 * * * * @sharedir@/piuparts/slave/slave_cleanup ; ! @sharedir@/piuparts/slave/slave_run >/dev/null 2>&1 || echo "piuparts-slave started by cron" # # monitor slave session every hour # 16 * * * * @sharedir@/piuparts/slave/detect_slave_problems # # monitor for problems every six hours # 23 */6 * * * @sharedir@/piuparts/slave/detect_leftover_processes # # monitor for cruft in the temporary directory once a day # 00 0 * * * @sharedir@/piuparts/slave/detect_tmp_cruft piuparts/conf/distros.conf0000644000000000000000000000652713061276553013120 0ustar # # Omitted fields will be set to defaults and completely "missing" # [distribution] entries will automatically be generated as follows: # # [] # uri = # distribution = # components = # depends = # candidates = # target-release = # # These are the standard fields for sources.list entries: # uri, distribution, components # # A non-empty "target-release" will add a -t argument to apt-get: # apt-get -t ... # # The "depends" entry can be used to refer to the "parent distribution" # (e.g. stable) of a "partial distribution" (e.g. stable-backports). # These are resolved recursively and will be made available in the # sources.list file. # # The "candidates" entry can be used to build a (virtually) merged # Packages file from one or more partial distribution (and maybe a full # parent distribution). This is used for selecting (package,version) # tuples to be tested. No recursive lookup. [etch] uri = http://archive.debian.org/debian [lenny] uri = http://archive.debian.org/debian [squeeze] uri = http://archive.debian.org/debian [squeeze-lts] uri = http://archive.debian.org/debian depends = squeeze target-release = squeeze-lts [squeeze-backports] uri = http://archive.debian.org/debian-backports depends = squeeze target-release = squeeze-backports [squeeze-backports-sloppy] uri = http://archive.debian.org/debian-backports depends = squeeze squeeze-backports target-release = squeeze-backports-sloppy [wheezy/updates] uri = http://security.debian.org depends = wheezy [wheezy-updates] depends = wheezy [wheezy-proposed-updates] depends = wheezy [wheezy-proposed] uri = None depends = wheezy wheezy/updates wheezy-updates wheezy-proposed-updates candidates = wheezy wheezy/updates wheezy-proposed-updates [wheezy-backports] depends = wheezy wheezy/updates target-release = wheezy-backports [wheezy-backports-sloppy] depends = wheezy wheezy-backports target-release = wheezy-backports-sloppy [jessie/updates] uri = http://security.debian.org depends = jessie [jessie-updates] depends = jessie [jessie-proposed-updates] depends = jessie [jessie-proposed] uri = None depends = jessie jessie/updates jessie-updates jessie-proposed-updates candidates = jessie jessie/updates jessie-proposed-updates [jessie-backports] depends = jessie jessie/updates target-release = jessie-backports [stretch/updates] uri = http://security.debian.org depends = stretch [stretch-updates] depends = stretch [stretch-proposed-updates] depends = stretch [stretch-debug] uri = http://deb.debian.org/debian-debug depends = stretch [stretch-proposed] uri = None depends = stretch stretch/updates stretch-updates stretch-proposed-updates candidates = stretch stretch/updates stretch-proposed-updates [stretch-backports] depends = stretch stretch/updates target-release = stretch-backports [sid-debug] uri = http://deb.debian.org/debian-debug depends = sid # alias [stable] distribution = jessie # alias [testing] distribution = stretch [experimental] depends = sid target-release = experimental # It's also possible to have "virtual" entries by setting uri to # the string "None". This allows e.g. to combine several partial # distributions. Such virtual distros can be used for standalone # piuparts runs, but not in master-slave setup. [squeeze-current] uri = None depends = squeeze/updates squeeze-updates piuparts/conf/piuparts-master.conf0000644000000000000000000000037112762537303014560 0ustar Alias /piuparts /var/lib/piuparts/htdocs Require all granted Options indexes IndexOptions FancyIndexing NameWidth=* AddType text/plain .log AddDefaultCharset utf-8 # vim:set syn=apache: piuparts/conf/piuparts-slave.sudoers0000644000000000000000000000051712567171321015135 0ustar # # copy this file to /etc/sudoers.d/piuparts-slave # # The piuparts slave needs to handle chroots. piupartss ALL = NOPASSWD: /usr/sbin/piuparts *, \ /bin/umount /srv/piuparts.debian.org/tmp/tmp*, \ /usr/bin/test -f /srv/piuparts.debian.org/tmp/tmp*, \ /usr/bin/rm -rf --one-file-system /srv/piuparts.debian.org/tmp/tmp* piuparts/conf/piuparts.conf.sample0000644000000000000000000000214413061276553014547 0ustar # # This is the configuration file for piuparts running in master-slave mode. # Usually it's placed in /etc/piuparts/piuparts.conf # # You MUST make sure that master-host, master-user, master-directory, and # mirror are set correctly. # [global] sections = sid mirror = http://deb.debian.org/debian master-host = localhost master-user = piupartsm piuparts-command = sudo piuparts --scriptsdir /etc/piuparts/scripts master-directory = /var/lib/piuparts/master slave-directory = /var/lib/piuparts/slave basetgz-directory = /var/cache/piuparts/basetgz output-directory = /var/lib/piuparts/htdocs tmpdir = /var/cache/piuparts/tmp doc-root = /piuparts/ idle-sleep = 300 max-tgz-age = 604800 max-reserved = 50 expire-old-days = 120 reschedule-old-days = 90 reschedule-old-count = 150 expire-fail-days = 45 reschedule-fail-days = 30 reschedule-fail-count = 25 [sid] precedence = 1 description = "Debian sid / main" piuparts-flags = --no-symlinks distro = sid # area = main # arch = amd64 upgrade-test-distros = debug = no ## another example: ## [s-p-u-i386] ## distro = stable-proposed-updates ## # area = main ## arch = i386 piuparts/custom-scripts/0000755000000000000000000000000013061276553012622 5ustar piuparts/custom-scripts/scripts/0000755000000000000000000000000013061276553014311 5ustar piuparts/custom-scripts/scripts/is_testable_uninstallable0000755000000000000000000000500213061276553021447 0ustar #!/bin/sh set -e # Return a non-zero error code to have piuparts skip testing an # uninstallable (or otherwise untestable) package. # Packages in main don't qualify for "external services out of our # control". They only qualify for "utterly broken" once on archive.d.o. # There are very few exceptions to this rule: # - usrmerge, which is one-way. # - localepurge, which seems hard to undo without a complete --reinstall. skip_test() { echo "Skipping piuparts test: $*" exit 1 } skip_always() { skip_test "$@" } skip_distupgrade() { if [ "$PIUPARTS_TEST" = "distupgrade" ]; then skip_test "$@" fi } # * For uninstallable *broken* packages we usually run the normal # installation+purge tests (which will fail and get the package marked # as bugged) and only skip the distupgrade tests since there is no point # in upgrading something that could not be installed as a start. # Automatically marking the upgrade logs as bugged would also be more # difficult, since the bug is in some *old* version. # * For packages requiring *external resources* not available in piuparts # we usually skip the failing initial install, too, since that is # *not a bug* in the package. case ${PIUPARTS_OBJECTS%%=*}@${PIUPARTS_DISTRIBUTION} in bugzilla3@lenny) # removed after squeeze skip_distupgrade "was excluded from squeeze-lts" ;; distributed-net@lenny|\ distributed-net@squeeze) skip_distupgrade "#655922: noninteractive install: postinst loops infinitely" ;; dnscache-run@lenny) skip_distupgrade "#664848: breaks dns resolution in chroot" ;; dynare-matlab@squeeze) skip_distupgrade "requires a matlab installation" ;; dynare-matlab@*|\ matlab-gdf@*|\ matlab-support@*) skip_always "requires a matlab installation" ;; localepurge@*) skip_distupgrade "requires a full --reinstall to restore disappeared files" ;; nautilus-dropbox@wheezy) skip_distupgrade "#770402: downloads awfully slow, hangs during installation" ;; python-milter@lenny|\ spf-milter-python@lenny) skip_always "virus signatures in python-milter may trigger the firewall" ;; runit-run@lenny) skip_distupgrade "#542157: fails to install" ;; sdic-eijiro@lenny) skip_distupgrade "requires a dictionary CD" ;; sdic-eijiro@*) skip_always "requires a dictionary CD" ;; steam@jessie) skip_distupgrade "#772598: cannot preseed licence acceptance" ;; ttf-root-installer@lenny) skip_always "download location disappeared" ;; usrmerge@*) skip_always "performs piuparts-unfriendy filesystem layout changes" ;; esac piuparts/custom-scripts/scripts/post_distupgrade_base_cleanup0000755000000000000000000000313512762537303022322 0ustar #!/bin/sh set -e is_installed() { local pkg="$1" dpkg-query -s "$pkg" >/dev/null 2>&1 || return 1 local status="$(dpkg-query -W -f '${Status}' $pkg)" test "$status" != "unknown ok not-installed" || return 1 test "$status" != "deinstall ok config-files" || return 1 return 0 } # # Remove obsolete base packages from the reference chroot. # # These packages are part of a minimal chroot in release R, but no # longer exist in release R+1. # Package dependencies will cause removal of the obsolete packages # during a subset of upgrade paths. Since these packages cannot be # reinstalled in release R+1 ensure they are always gone from the # final reference chroot. # # Only while creating the reference chroot. test "$PIUPARTS_PHASE" = "" || exit 0 PURGE= mark_for_purge() { pkg="$1" # skip if the package is not installed is_installed "$pkg" || return 0 case ${PIUPARTS_OBJECTS%%=*} in $pkg) # keep it while testing it ;; *) PURGE="$PURGE $pkg" esac } if [ "$PIUPARTS_DISTRIBUTION" = "wheezy" ] || \ [ "$PIUPARTS_DISTRIBUTION" = "wheezy-proposed" ] ; then mark_for_purge libdb4.8 # gcc-4.4-base is part of the minimal squeeze chroot # but it no longer exists in wheezy mark_for_purge gcc-4.4-base fi if [ "$PIUPARTS_DISTRIBUTION" = "jessie" ] || \ [ "$PIUPARTS_DISTRIBUTION" = "jessie-proposed" ] ; then mark_for_purge libdb5.1 fi if [ "$PIUPARTS_DISTRIBUTION" = "stretch" ] || \ [ "$PIUPARTS_DISTRIBUTION" = "stretch-proposed" ] ; then mark_for_purge libprocps3 fi if [ -n "$PURGE" ]; then echo "Removing packages from base system:$PURGE" apt-get -y remove --purge $PURGE fi piuparts/custom-scripts/scripts/post_distupgrade_exceptions0000755000000000000000000000361313061276553022063 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ]; then case ${PIUPARTS_OBJECTS%%=*} in linpopup) # package removed after lenny log_debug for file in /var/lib/linpopup/messages.dat do test ! -f "$file" || chmod -c o-w "$file" done ;; esac fi if [ "$PIUPARTS_DISTRIBUTION" = "wheezy" ]; then # https://bugs.debian.org/687611 if [ -f /usr/share/keyrings/debian-archive-removed-keys.gpg~ ]; then echo "FIXING /usr/share/keyrings/debian-archive-removed-keys.gpg~" mv -v /usr/share/keyrings/debian-archive-removed-keys.gpg~ /usr/share/keyrings/debian-archive-removed-keys.gpg fi case ${PIUPARTS_OBJECTS%%=*} in kismet|\ tshark|\ wireshark|\ wireshark-common|\ wireshark-dbg|\ libcap2-bin) # libcap2-bin/wheezy is part of the minimal chroot and recommends libpam-cap # a conffile moved from libcap2-bin/squeeze to libpam-cap/wheezy log_debug apt-get install -yf libpam-cap ;; ogre-doc-nonfree) # #773059 - ogre-doc: unhandled symlink to directory conversion: /usr/share/doc/PACKAGE # package removed after lenny log_debug apt-get install -yf ogre-1.8-doc ;; phpgacl) # #682825 # package not in wheezy log_debug for dir in /usr/share/phpgacl/admin/templates_c do test ! -d "$dir" || chmod -c o-w "$dir" done ;; esac fi if [ "$PIUPARTS_DISTRIBUTION" = "jessie" ]; then # base-files only upgrades pristine /etc/nsswitch.conf if ! grep -q ^gshadow: /etc/nsswitch.conf ; then echo "Adding gshadow line to /etc/nsswitch.conf" sed -i '/^shadow:/a gshadow: files' /etc/nsswitch.conf fi fi if [ "$PIUPARTS_DISTRIBUTION" = "stretch" ]; then # fakeroot:i386 in jessie shipped these with permissions 0775 (#826318) for dir in /usr/share/man/nl /usr/share/man/nl/man1 do test ! -d "$dir" || chmod -c g-w "$dir" done fi piuparts/custom-scripts/scripts/post_distupgrade_force-unsafe-io0000777000000000000000000000000012567171321030042 2post_setup_force-unsafe-ioustar piuparts/custom-scripts/scripts/post_distupgrade_squeeze-backports0000755000000000000000000000062212567171321023343 0ustar #!/bin/sh set -e test "$PIUPARTS_DISTRIBUTION" = "squeeze-backports" || exit 0 # apt-get -t squeeze-backports dist-upgrade may pull in too many # packages that are not co-installable in squeeze-backports # so maintain a list of packages in the sid base system # that are in squeeze-backports, too, and don't cause problems PKGS="" PKGS="$PKGS insserv" apt-get -y -t squeeze-backports install $PKGS piuparts/custom-scripts/scripts/post_install_exceptions0000755000000000000000000000227313042113740021202 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } is_installed() { local pkg="$1" dpkg-query -s "$pkg" >/dev/null 2>&1 || return 1 local status="$(dpkg-query -W -f '${Status}' $pkg)" test "$status" != "unknown ok not-installed" || return 1 test "$status" != "deinstall ok config-files" || return 1 return 0 } case ${PIUPARTS_OBJECTS%%=*} in apt-listbugs) log_debug # when installed apt-listbugs is run on installations / upgrades # and will cause them to fail due to prompting # if packages being installed currently have RC bugs. # so disable it here. dpkg-divert --rename /usr/sbin/apt-listbugs ln -svf /bin/true /usr/sbin/apt-listbugs ;; esac if is_installed dkms; then # remember for post_purge_exceptions touch /var/run/piuparts-stamp-dkms-installed # if dkms built a kernel module, there might be a # leftover 'sleep' process from progress reporting sleep 3 fi for pgconf in /etc/postgresql/*/main/postgresql.conf do test -f "$pgconf" && test -r "$pgconf" || continue port=$(awk '/^port/ { print $3 }' "$pgconf") if [ "-n" $port ] && [ "$port" != "5432" ] then echo "Non-default port in $pgconf" fi done piuparts/custom-scripts/scripts/post_purge_exceptions0000755000000000000000000000212713042113740020654 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } case ${PIUPARTS_OBJECTS%%=*} in fai-nfsroot) log_debug rm -f /.THIS_IS_THE_FAI_NFSROOT ;; ltsp-client|\ ltsp-client-core) log_debug rm -f /etc/ltsp_chroot ;; amd64-libs|amd64-libs-dev) # leaves a superfluous empty line after purge log_debug sed -i '3{/^$/d}' /etc/ld.so.conf ;; localepurge) case ${PIUPARTS_DISTRIBUTION} in lenny*|squeeze*) ;; *) # reinstall packages where files might have been dropped log_debug EXTRA="" apt-get -u --reinstall --fix-missing install $(dpkg -S LC_MESSAGES 'man/??/man' | cut -d: -f1 | tr ', ' '\n' | sort -u) $EXTRA ;; esac ;; virtualbox-ext-pack) #802143: virtualbox-ext-pack leaves VBoxSVC processes running # these should terminate within 5 seconds if not used log_debug sleep 10 ;; esac if [ -f /var/run/piuparts-stamp-dkms-installed ]; then rm /var/run/piuparts-stamp-dkms-installed # if dkms built a kernel module, there might be a # leftover 'sleep' process from progress reporting sleep 3 fi piuparts/custom-scripts/scripts/post_remove_exceptions0000755000000000000000000000106113042113740021023 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } # cleanup from pre_remove_exceptions rm -fv /etc/apt/apt.conf.d/piuparts-allow-remove-essential case ${PIUPARTS_OBJECTS%%=*} in asclassic) case "$PIUPARTS_DISTRIBUTION" in lenny) # purging works in lenny ;; *) log_debug # postrm purge unconditionally removes some files that may be gone already sed -i 's/rm system.steprc menudefs.hook/rm -f system.steprc menudefs.hook/' /var/lib/dpkg/info/asclassic.postrm ;; esac ;; esac piuparts/custom-scripts/scripts/post_setup_dummy_users0000755000000000000000000000046013036653026021074 0ustar #!/bin/sh set -e case ${PIUPARTS_OBJECTS%%=*} in dpkg) # skip while creating the tarball exit 0 ;; esac for id in $(seq 100 149) $(seq 1000 1010) do getent group $id > /dev/null || groupadd --system -g $id dummy$id getent passwd $id > /dev/null || useradd --system -u $id -g 100 dummy$id done piuparts/custom-scripts/scripts/post_setup_forbid_home0000755000000000000000000000055713037404512020777 0ustar #!/bin/sh set -e case ${PIUPARTS_OBJECTS%%=*} in dpkg) # skip while creating the tarball exit 0 ;; esac case $PIUPARTS_DISTRIBUTION in lenny|squeeze) exit 0 ;; esac if [ -d /home ]; then echo "Disabling /home" mv /home /home.distrib dpkg-divert /home echo "This is a dummy file to prevent creating directories in /home" > /home chmod 000 /home fi piuparts/custom-scripts/scripts/post_setup_force-unsafe-io0000755000000000000000000000051712567171321021506 0ustar #!/bin/sh set -e if [ ! -f /etc/dpkg/dpkg.cfg.d/piuparts-force-unsafe-io ] then if dpkg --force-help | grep -q unsafe-io then echo "Enabling dpkg --force-unsafe-io." echo force-unsafe-io > /etc/dpkg/dpkg.cfg.d/piuparts-force-unsafe-io fi fi if [ ! -h /bin/sync ] then dpkg-divert --rename /bin/sync ln -sv true /bin/sync fi piuparts/custom-scripts/scripts/post_setup_minimize0000755000000000000000000000035613061276553020351 0ustar #!/bin/sh set -e # only while creating the tarball if [ "${PIUPARTS_OBJECTS%%=*}" = "dpkg" ] then case $PIUPARTS_DISTRIBUTION in jessie) dpkg --purge gcc-4.8-base ;; sid|stretch) dpkg --purge lsb-base tzdata ;; esac fi piuparts/custom-scripts/scripts/post_setup_squeeze-backports0000777000000000000000000000000012567171321031214 2post_distupgrade_squeeze-backportsustar piuparts/custom-scripts/scripts/pre_distupgrade_exceptions0000755000000000000000000000616613037404512021661 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } is_installed() { local pkg="$1" dpkg-query -s "$pkg" >/dev/null 2>&1 || return 1 local status="$(dpkg-query -W -f '${Status}' $pkg)" test "$status" != "unknown ok not-installed" || return 1 test "$status" != "deinstall ok config-files" || return 1 return 0 } dbconfig_enable_purge() { log_debug echo "$1 $1${2+/$2}/dbconfig-remove boolean true" | debconf-set-selections echo "$1 $1${2+/$2}/purge boolean true" | debconf-set-selections } if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "squeeze" ]; then case ${PIUPARTS_OBJECTS%%=*} in crm114) #562946 log_debug echo "crm114 crm114/forceupgrade boolean true" | debconf-set-selections ;; slapd) # squeeze postinst wants to restore a backup from lenny that was never created log_debug dpkg --purge slapd ;; esac # squeeze does not properly upgrade adduser.conf, so generate a new one if [ -f /etc/adduser.conf ]; then md5="$(md5sum /etc/adduser.conf | awk '{ print $1 }')" # 5b577c9cb18e4852fc7e45830d230ec1: adduser/lenny pristine # 28928669e36f1ab616dfda39af3d79a7: adduser/lenny + dpsyco-lib/lenny if [ "$md5" = "5b577c9cb18e4852fc7e45830d230ec1" ] || \ [ "$md5" = "28928669e36f1ab616dfda39af3d79a7" ] then rm -fv /etc/adduser.conf fi fi fi if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "wheezy" ]; then case ${PIUPARTS_OBJECTS%%=*} in cnews) # dpkg 1.16 does not like the bad cnews version number cr.g7-40.4 # cnews was removed after lenny log_debug dpkg --purge cnews ;; nfsbooted) # sysv-rc fails migration to dependency based boot sequence if nfsbooted is installed # nfsbooted was removed after lenny log_debug dpkg --purge nfsbooted ;; pdns-backend-mysql) # WORKSAROUND #698562: upgrade from squeeze => wheezy fails if database has not been created before the upgrade log_debug dpkg --purge pdns-backend-mysql ;; pdns-backend-pgsql) # WORKSAROUND #698562: upgrade from squeeze => wheezy fails if database has not been created before the upgrade log_debug dpkg --purge pdns-backend-pgsql ;; esac # WORKSAROUND #655969: lirc: prompting due to modified conffiles which where not modified by the user: /etc/lirc/hardware.conf if [ -f /etc/lirc/hardware.conf ]; then log_debug sed -i '/^DRIVER=/s/.*/DRIVER="UNCONFIGURED"/' /etc/lirc/hardware.conf fi fi if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "jessie" ]; then case ${PIUPARTS_OBJECTS%%=*} in ekeyd-uds) # ekeyd-uds was removed after squeeze, maintainer scripts are incompatible with udev/jessie log_debug dpkg --purge ekeyd-uds ;; otrs2) #707075: cannot automatically upgrade from wheezy to jessie due to mysql storage engine change if is_installed mysql-server; then log_debug dbconfig_enable_purge otrs2 dpkg --purge otrs2 fi ;; esac fi if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "experimental" ]; then case ${PIUPARTS_OBJECTS} in dnscache-run=1:1.05-9~exp2) #664848: breaks dns resolution in chroot if installed in a chroot log_debug echo "*** ABORT - Installation broke DNS in chroot ***" exit 1 ;; esac fi piuparts/custom-scripts/scripts/pre_distupgrade_foreign_architecture_i3860000755000000000000000000000135212762537303024445 0ustar #!/bin/sh set -e case "$PIUPARTS_DISTRIBUTION" in squeeze*) ;; *) exit 0 ;; esac case "$PIUPARTS_DISTRIBUTION_NEXT" in lenny*|squeeze*) exit 0 ;; esac test "$(dpkg --print-architecture)" = "amd64" || exit 0 go= case ${PIUPARTS_OBJECTS%%=*} in ia32-libs|ia32-libs-gtk) go=yes ;; lib32nss-mdns) go=yes ;; *) dpkg-query -s "ia32-libs" >/dev/null 2>&1 || exit 0 ;; esac case ${PIUPARTS_OBJECTS} in *=None) go=yes ;; esac case ${PIUPARTS_OBJECTS%%=*} in *wine*) go=yes ;; education-thin-client-server|education-workstation) go=yes ;; playonlinux) go=yes ;; esac test -n "$go" || exit 0 echo "Enabling foreign architecture i386 for $PIUPARTS_OBJECTS" apt-get update apt-get -y install apt dpkg dpkg --add-architecture i386 piuparts/custom-scripts/scripts/pre_distupgrade_zz_database-server0000755000000000000000000000700313061276553023273 0ustar #!/bin/sh set -e # Skip while creating the reference chroot. if [ "$PIUPARTS_PHASE" = "" ]; then exit 0 fi is_installed() { local pkg="$1" dpkg-query -s "$pkg" >/dev/null 2>&1 || return 1 local status="$(dpkg-query -W -f '${Status}' $pkg)" test "$status" != "unknown ok not-installed" || return 1 test "$status" != "deinstall ok config-files" || return 1 return 0 } retry_configure_pending() { echo "" echo "Upgrade didn't go smooth, trying dpkg --configure --pending" dpkg --configure --pending } CANDIDATES= CANDIDATES="$CANDIDATES default-mysql-server" CANDIDATES="$CANDIDATES postgresql" CANDIDATES="$CANDIDATES postgresql-8.3" CANDIDATES="$CANDIDATES postgresql-8.4" CANDIDATES="$CANDIDATES postgresql-9.1" CANDIDATES="$CANDIDATES postgresql-9.4" CANDIDATES="$CANDIDATES postgresql-9.6" case "$PIUPARTS_DISTRIBUTION_NEXT" in jessie) CANDIDATES="$CANDIDATES mysql-server-5.5" ;; esac case "$PIUPARTS_DISTRIBUTION_NEXT" in stretch) : # no mysql-server ;; *) CANDIDATES="$CANDIDATES mysql-server" ;; esac PACKAGES= SWITCH_TO_MARIADB= RETRY="false" # early upgrade runs into even more trouble for some packages ... case ${PIUPARTS_OBJECTS%%=*} in med-practice|audiolink) CANDIDATES="" ;; redmine) CANDIDATES="" ;; esac if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "stretch" ]; then # recommended/suggested/related package mysql-server gets removed during dist-upgrade case ${PIUPARTS_OBJECTS%%=*} in bacula-director-mysql|\ bacula-director-mysql-dbg) SWITCH_TO_MARIADB=yes # (new) transitively recommended package mariadb-server-x.y gets # configured (and the server started) too late (after ourselves) RETRY="retry_configure_pending" ;; bareos-database-mysql) SWITCH_TO_MARIADB=yes ;; icinga2-ido-mysql) SWITCH_TO_MARIADB=yes ;; openstack-cloud-services|\ openstack-cloud-identity) SWITCH_TO_MARIADB=yes ;; pdns-backend-mysql) SWITCH_TO_MARIADB=yes ;; phpmyadmin) SWITCH_TO_MARIADB=yes ;; pnopaste) SWITCH_TO_MARIADB=yes ;; redmine-mysql) SWITCH_TO_MARIADB=yes ;; tango-db|\ tango-db-dbg|\ tango-accesscontrol|\ tango-accesscontrol-dbg) SWITCH_TO_MARIADB=yes ;; yubikey-val) SWITCH_TO_MARIADB=yes ;; esac fi for pkg in $CANDIDATES do if is_installed "$pkg" then PACKAGES="$PACKAGES $pkg" fi done if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "stretch" ] && [ "$SWITCH_TO_MARIADB" = "yes" ]; then if is_installed "mysql-server" then PACKAGES="$PACKAGES default-mysql-server mysql-server-" fi fi test ! -x /usr/bin/pg_lsclusters || pg_lsclusters if [ -n "$PACKAGES" ] then echo "Upgrading ${PACKAGES# } early" apt-get update apt-get -y install $PACKAGES || $RETRY fi upgrade_pg_cluster() { local from=$1 local to=$2 if ! is_installed postgresql-${from} then return 0 fi if ! is_installed postgresql-${to} then echo "" echo "Installing postgresql-${to}" apt-get -y install postgresql-${to} fi if is_installed postgresql-${from} && is_installed postgresql-${to} then echo "" echo "Updating PostgreSQL Cluster from ${from} to ${to}" pg_dropcluster ${to} main --stop pg_upgradecluster -v ${to} ${from} main pg_dropcluster ${from} main fi } case "${PIUPARTS_DISTRIBUTION}:${PIUPARTS_DISTRIBUTION_NEXT}" in lenny:squeeze) upgrade_pg_cluster 8.3 8.4 ;; squeeze:wheezy) upgrade_pg_cluster 8.4 9.1 ;; wheezy:jessie) upgrade_pg_cluster 9.1 9.4 ;; jessie:stretch) upgrade_pg_cluster 9.4 9.6 ;; jessie:sid) upgrade_pg_cluster 9.4 9.6 ;; esac test ! -x /usr/bin/pg_lsclusters || pg_lsclusters piuparts/custom-scripts/scripts/pre_distupgrade_zz_upgrade_early0000755000000000000000000000157112762537303023052 0ustar #!/bin/sh set -e # Skip while creating the reference chroot. if [ "$PIUPARTS_PHASE" = "" ]; then exit 0 fi log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } # packages to upgrade early EARLY= if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "squeeze" ]; then if dpkg-query -s "octave3.0" >/dev/null 2>&1 then #696377 EARLY="$EARLY libblas3gf liblapack3gf" fi case ${PIUPARTS_OBJECTS%%=*} in libapt-rpm-pkg-dev) # libapt-rpm-pkg-dev no longer exists in squeeze # causing some packages to be "kept back" EARLY="$EARLY libreadline5-dev" ;; python-pydoctor) # work around #696376: "/usr/sbin/update-python-modules: /usr/bin/python: bad interpreter: No such file or directory" EARLY="$EARLY python-pydoctor" ;; esac fi if [ -n "$EARLY" ]; then log_debug echo "Upgrading early: $EARLY" apt-get update apt-get -y install $EARLY fi piuparts/custom-scripts/scripts/pre_install_database-server0000755000000000000000000001624313061276553021711 0ustar #!/bin/sh set -e # Install a database server for packages that require one during postinst # but only recommend or suggest (or maybe not even this) the server # package to allow for remote db servers. MYSQL= POSTGRESQL= SQLITE3= FUSIONFORGE= CLIENT= INSTALL= case ${PIUPARTS_OBJECTS%%=*} in acidbase) MYSQL=yes ;; auth2db) MYSQL=yes ; CLIENT=yes ;; auth2db-common) MYSQL=yes ; CLIENT=yes ;; auth2db-frontend) MYSQL=yes ; CLIENT=yes ;; b2evolution) MYSQL=yes ;; bacula-director-mysql) MYSQL=yes ;; bacula-director-mysql-dbg) MYSQL=yes ;; bacula-director-pgsql) POSTGRESQL=yes ;; bacula-director-pgsql-dbg) POSTGRESQL=yes ;; bandwidthd-pgsql) POSTGRESQL=yes ;; bareos) POSTGRESQL=yes ;; bareos-database-common) POSTGRESQL=yes ;; bareos-database-mysql) POSTGRESQL=yes ; MYSQL=yes ;; bareos-database-postgresql) POSTGRESQL=yes ;; bareos-database-sqlite3) POSTGRESQL=yes ; SQLITE3=yes ;; bareos-database-tools) POSTGRESQL=yes ;; bareos-director) POSTGRESQL=yes ;; bley) POSTGRESQL=yes ;; blootbot) MYSQL=yes ;; buddycloud-server) POSTGRESQL=yes ;; bugzilla3) MYSQL=yes ;; cacti) MYSQL=yes ;; cacti-cactid) MYSQL=yes ;; cacti-spine) MYSQL=yes ;; chado-utils) POSTGRESQL=yes ;; convirt) MYSQL=yes ;; docbookwiki) MYSQL=yes ;; dotclear) MYSQL=yes ;; dotlrn) POSTGRESQL=yes ;; drupal6) MYSQL=yes ;; drupal6-mod-*) MYSQL=yes ;; drupal6-thm-*) MYSQL=yes ;; drupal6-trans-ru) MYSQL=yes ;; drupal7) MYSQL=yes ;; drupal7-mod-*) MYSQL=yes ;; dtc-postfix-dovecot) MYSQL=yes ;; frontaccounting) MYSQL=yes ;; fossology-agents) POSTGRESQL=yes ;; fusionforge-plugin-*) FUSIONFORGE=yes ;; fusionforge-standard) POSTGRESQL=yes ;; fusioninventory-for-glpi) MYSQL=yes ;; gforge-db-postgresql) POSTGRESQL=yes ;; gforge-lists-mailman) POSTGRESQL=yes ;; gforge-plugin-extratabs) POSTGRESQL=yes ;; gforge-plugin-mediawiki) POSTGRESQL=yes ;; gforge-plugin-scmarch) POSTGRESQL=yes ;; gforge-plugin-scmcvs) POSTGRESQL=yes ;; gforge-plugin-scmdarcs) POSTGRESQL=yes ;; gforge-plugin-scmgit) POSTGRESQL=yes ;; gforge-plugin-scmhg) POSTGRESQL=yes ;; gforge-web-apache2) POSTGRESQL=yes ;; gforge-web-apache2-vhosts) POSTGRESQL=yes ;; glance) SQLITE3=yes ;; glance-api) SQLITE3=yes ;; glance-common) SQLITE3=yes ;; glance-registry) SQLITE3=yes ;; glpi) MYSQL=yes ;; gnuhealth-server) POSTGRESQL=yes ;; grr) MYSQL=yes ;; heat-api) SQLITE3=yes ;; heat-api-cfn) SQLITE3=yes ;; heat-api-cloudwatch) SQLITE3=yes ;; heat-common) SQLITE3=yes ;; heat-engine) SQLITE3=yes ;; icinga-idoutils) POSTGRESQL=yes ;; icinga-phpapi) MYSQL=yes ;; icinga-web) POSTGRESQL=yes ;; icinga-web-config-icinga) POSTGRESQL=yes ;; icinga-web-config-icinga2-ido-mysql) MYSQL=yes ;; icinga-web-config-icinga2-ido-pgsql) POSTGRESQL=yes ;; icinga-web-pnp) POSTGRESQL=yes ;; icinga2-ido-mysql) MYSQL=yes ;; icinga2-ido-pgsql) POSTGRESQL=yes ;; jclicmoodle) POSTGRESQL=yes ;; jffnms) MYSQL=yes ;; letodms) MYSQL=yes ;; letodms-webdav) MYSQL=yes ;; libchado-perl) POSTGRESQL=yes ;; libdspam7-dbg) POSTGRESQL=yes ;; libdspam7-drv-mysql) MYSQL=yes ;; libdspam7-drv-pgsql) POSTGRESQL=yes ;; mantis) MYSQL=yes ;; med-bio-dev) POSTGRESQL=yes ;; moodle) POSTGRESQL=yes ; MYSQL=yes ;; moodle-book) POSTGRESQL=yes ;; moodle-debian-edu-*) POSTGRESQL=yes ;; movabletype-opensource) SQLITE3=yes ;; mtop) MYSQL=yes ;; mythtv-database) MYSQL=yes ;; nagvis) MYSQL=yes ;; ndoutils-common) MYSQL=yes ;; ndoutils-nagios3-mysql) MYSQL=yes ;; neutron-common) SQLITE3=yes ;; neutron-server) SQLITE3=yes ;; neutron-*-agent) SQLITE3=yes ;; nginx-naxsi-ui) MYSQL=yes ;; obs-api) MYSQL=yes ;; ocsinventory-reports) MYSQL=yes ;; ocsinventory-server) MYSQL=yes ;; openacs) POSTGRESQL=yes ;; openstack-dashboard) SQLITE3=yes ;; openstack-dashboard-apache) SQLITE3=yes ;; otrs) POSTGRESQL=yes ;; otrs2) POSTGRESQL=yes case "$PIUPARTS_DISTRIBUTION" in lenny*|squeeze*) MYSQL=yes ;; #707075 esac ;; pdns-backend-mysql) MYSQL=yes ;; pdns-backend-pgsql) POSTGRESQL=yes ;; phpbb3) MYSQL=yes ;; phpbb3-l10n) MYSQL=yes ;; phpgacl) MYSQL=yes ;; phpmyadmin) MYSQL=yes ;; phpwiki) MYSQL=yes ;; pinba-engine-mysql) MYSQL=yes ;; pinba-engine-mysql-5.5) MYSQL=yes ;; pinba-engine-mysql-5.6) MYSQL=yes ;; piwigo) MYSQL=yes ;; pnopaste) MYSQL=yes ;; poker-web) MYSQL=yes ;; postfix-policyd) MYSQL=yes ;; postfixadmin) MYSQL=yes ;; prelude-manager) MYSQL=yes ;; prewikka) MYSQL=yes ;; pybit-web) POSTGRESQL=yes ;; python-django-horizon) SQLITE3=yes ;; python-quantum) SQLITE3=yes ;; python-quantumclient) SQLITE3=yes ;; quantum-common) SQLITE3=yes ;; quantum-plugin-*) SQLITE3=yes ;; quantum-server) SQLITE3=yes ;; quantum-*-agent) SQLITE3=yes ;; redmine) MYSQL=yes ;; redmine-mysql) MYSQL=yes ;; redmine-pgsql) POSTGRESQL=yes ;; redmine-plugin-botsfilter) MYSQL=yes ;; redmine-plugin-recaptcha) MYSQL=yes ;; roundcube) MYSQL=yes ;; roundcube-core) MYSQL=yes ;; roundcube-plugins) MYSQL=yes ;; roundcube-plugins-extra) MYSQL=yes ;; rsyslog-mysql) MYSQL=yes ;; rsyslog-pgsql) POSTGRESQL=yes ;; scuttle) MYSQL=yes ;; semanticscuttle) MYSQL=yes ;; serendipity) MYSQL=yes ;; simba) MYSQL=yes ;; spotweb) MYSQL=yes ;; sshproxy-backend-mysql) MYSQL=yes ;; steam) MYSQL=yes ;; sympa) MYSQL=yes ;; tango-accesscontrol) MYSQL=yes ;; tango-accesscontrol-dbg) MYSQL=yes ;; tango-db) MYSQL=yes ;; tango-db-dbg) MYSQL=yes ;; textpattern) MYSQL=yes ;; torrentflux) MYSQL=yes ;; tt-rss) MYSQL=yes ;; tuskar) SQLITE3=yes ;; tuskar-api) SQLITE3=yes ;; tuskar-common) SQLITE3=yes ;; tuskar-manager) SQLITE3=yes ;; typo3-dummy) MYSQL=yes ;; ukolovnik) MYSQL=yes ;; webcalendar) MYSQL=yes ;; webissues-server) MYSQL=yes ;; websimba) MYSQL=yes ;; wims-moodle) POSTGRESQL=yes ;; zabbix-frontend-php) MYSQL=yes ;; zabbix-proxy-mysql) MYSQL=yes ;; zabbix-proxy-pgsql) POSTGRESQL=yes ;; zabbix-server-mysql) MYSQL=yes ;; zabbix-server-pgsql) POSTGRESQL=yes ;; esac if [ "$FUSIONFORGE" = "yes" ]; then case "$PIUPARTS_DISTRIBUTION" in wheezy*|jessie|jessie/updates|jessie-updates|jessie-proposed-updates) POSTGRESQL=yes ;; *) INSTALL="$INSTALL fusionforge-db-local" ;; esac fi if [ "$MYSQL" = "yes" ]; then case "$PIUPARTS_DISTRIBUTION" in lenny*|squeeze*|wheezy*|jessie*) INSTALL="$INSTALL mysql-server ${CLIENT:+mysql-client}" ;; *) INSTALL="$INSTALL default-mysql-server ${CLIENT:+default-mysql-client}" ;; esac fi if [ "$POSTGRESQL" = "yes" ]; then INSTALL="$INSTALL postgresql" fi if [ "$SQLITE3" = "yes" ]; then INSTALL="$INSTALL sqlite3" fi if [ -n "$INSTALL" ]; then echo "Installing ${INSTALL# }..." apt-get -y install $INSTALL fi for pgconf in /etc/postgresql/*/main/postgresql.conf do test -f "$pgconf" && test -r "$pgconf" || continue port=$(awk '/^port/ { print $3 }' "$pgconf") if [ "-n" $port ] && [ "$port" != "5432" ] then echo "Non-default port in $pgconf" fi done exit 0 piuparts/custom-scripts/scripts/pre_install_exceptions0000755000000000000000000001540613061276553021022 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } case ${PIUPARTS_OBJECTS%%=*} in file-rc) case "$PIUPARTS_DISTRIBUTION" in lenny|squeeze*|stretch*|sid) # force installation and removal of essential package sysv-rc log_debug yes 'Yes, do as I say!' | apt-get -y --force-yes install file-rc ;; esac ;; sysvinit-core) # this will remove systemd-sysv log_debug apt-get -y install sysvinit-core ;; live-config-upstart|\ netscript-2.4-upstart|\ upstart) case "$PIUPARTS_DISTRIBUTION" in squeeze*|wheezy*) # force installation and removal of essential package sysvinit log_debug yes 'Yes, do as I say!' | apt-get -y --force-yes install upstart ;; esac ;; upstart-dconf-bridge|\ upstart-monitor) # switch init to upstart before installation apt-get install upstart ;; systemd-sysv) case "$PIUPARTS_DISTRIBUTION" in wheezy*) # force installation and removal of essential package sysvinit log_debug yes 'Yes, do as I say!' | apt-get -y --force-yes install systemd-sysv ;; esac ;; esac if [ "$PIUPARTS_DISTRIBUTION" = "wheezy-backports" ]; then case ${PIUPARTS_OBJECTS%%=*} in libreoffice-style-*) log_debug apt-get -y -t $PIUPARTS_DISTRIBUTION install ${PIUPARTS_OBJECTS%%=*} libreoffice-common- ;; esac fi # # the remaining exceptions are only for the initial package installation # if [ "$PIUPARTS_PHASE" != "install" ]; then exit 0 fi case ${PIUPARTS_OBJECTS%%=*} in samhain) # work around #749602 log_debug mkdir -p /var/state/samhain/ touch /var/state/samhain/samhain_file ;; fai-nfsroot) # fai-nfsroot refuses installation unless this file exist log_debug touch /.THIS_IS_THE_FAI_NFSROOT ;; education-thin-client|\ ltsp-client|\ ltsp-client-core) # ltsp-client-core refuses installation unless this file exist log_debug touch /etc/ltsp_chroot ;; ffgtk|\ roger-router|\ roger-router-cli) # postinst tries to add a printer with lpadmin log_debug dpkg-divert --rename /usr/sbin/lpadmin ;; freedombox-setup) log_debug # replaces /etc/motd by a symlink if [ -e /etc/motd ] && [ ! -L /etc/motd ] && [ ! -e /etc/motd.orig ]; then mv -v /etc/motd /etc/motd.orig fi ;; reseed) log_debug # postinst runs reseed which accesses the net dpkg-divert --rename /usr/sbin/reseed ln -s /bin/true /usr/sbin/reseed ;; bugzilla3) # checksetup.pl goes into infinite loop asking for them log_debug echo "bugzilla3 bugzilla3/bugzilla_admin_name string bz@local.host" | debconf-set-selections echo "bugzilla3 bugzilla3/bugzilla_admin_real_name string Bz" | debconf-set-selections echo "bugzilla3 bugzilla3/bugzilla_admin_pwd password bzbzbzbz" | debconf-set-selections ;; esac if [ "$PIUPARTS_DISTRIBUTION" = "wheezy" ]; then case ${PIUPARTS_OBJECTS%%=*} in ttf-root-installer) #774163: missing Depends: ca-certificates log_debug apt-get -y install ca-certificates ;; esac fi if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ] || \ [ "$PIUPARTS_DISTRIBUTION" = "squeeze-backports" ] || \ [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then case ${PIUPARTS_OBJECTS%%=*} in autopkgtest-xenlvm|\ clvm|\ cman|\ collectd|\ collectd-dbg|\ collectd-utils|\ dtc-xen|\ ganeti|\ ganeti2|\ gfs-tools|\ gfs2-tools|\ libcollectdclient0|\ libcollectdclient-dev|\ liblinux-lvm-perl|\ libsys-virt-perl|\ libvirt0|\ libvirt0-dbg|\ libvirt-dev|\ libvirt-ocaml|\ libvirt-ocaml-dev|\ libvirt-ruby|\ libvirt-ruby1.8|\ mozilla-virt-viewer|\ munin-libvirt-plugins|\ mylvmbackup|\ python-libvirt|\ redhat-cluster-suite|\ rgmanager|\ virtinst|\ virt-top|\ virt-viewer|\ xenwatch|\ lvm2) # work around lvm2 bug https://bugs.debian.org/603036 which is squeeze-ignore log_debug apt-get -y install udev ;; jhove|\ libbytecode-java|\ libcodemodel-java|\ libjavassist-java|\ libjdic-java|\ libjgroups-java|\ libjibx-java|\ libpj-java|\ libswingx-java) # these packages depend on sun-java* | java2-runtime # install default-jre to satisfy the java2-runtime dependency log_debug apt-get -y install default-jre ;; esac fi if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ]; then case ${PIUPARTS_OBJECTS%%=*} in bootcd-ia64) if [ "$PIUPARTS_TEST" = "distupgrade" ] && [ "$(uname -m)" = "x86_64" ]; then #622690: bootcd-ia64 has a Pre-Depends/Depends cycle that apt cannot resolve log_debug apt-get -y install bootcd-i386 fi ;; mini-buildd-bld|mini-buildd-rep) #632955, #656746 - time limit exceeded during install log_debug echo "*** ABORT - Installation would deadlock ***" exit 1 ;; esac fi if [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then # install undeclared dependencies case ${PIUPARTS_OBJECTS%%=*} in clamav-getfiles) #603082 log_debug apt-get -y install curl ;; cyrus-*-2.2|sa-learn-cyrus) #694254 db4.2-util exists in lenny only and is needed for upgrades to wheezy log_debug apt-get -y install db4.2-util ;; debian-edu-artwork-usplash) # postinst fails without update-initramfs log_debug apt-get -y install initramfs-tools ;; gforge-shell-postgresql|gforge-plugin-scmcvs|gforge-plugin-scmsvn) #604218 log_debug apt-get -y install libcap2 ;; otrs2) #561889 log_debug apt-get -y install libapache2-mod-perl2 ;; sdic-gene95) #478592 log_debug apt-get -y install bzip2 wget ;; tftpd-hpa) #522780 log_debug apt-get -y install update-inetd ;; ttf-beteckna) #502707 log_debug apt-get -y install defoma ;; ttf-mscorefonts-installer) log_debug apt-get -y install ca-certificates ;; esac # prefer inn over cnews (bad version number cr.g7-40.4) case ${PIUPARTS_OBJECTS%%=*} in newsx|post-faq) log_debug apt-get -y install inn ;; esac # work around postinst failures case ${PIUPARTS_OBJECTS%%=*} in ion3) # annoying debconf prompt log_debug echo "ion3 ion3/acknowledge-maybe-outdated boolean true" | debconf-set-selections echo "ion3 ion3/acknowledge-outdated boolean true" | debconf-set-selections ;; ipppd|ibod|isdnutils) #542156 log_debug test -e /dev/MAKEDEV || ln -sfv /sbin/MAKEDEV /dev/MAKEDEV ;; esac # allow lenny kernel image installation in chroot for flavor in amd64 openvz-amd64 vserver-amd64 xen-amd64 486 686 686-bigmem openvz-686 vserver-686 vserver-686-bigmem xen-686 do echo "linux-image-2.6.26-2-$flavor linux-image-2.6.26-2-$flavor/preinst/bootloader-initrd-2.6.26-2-$flavor boolean false" echo "linux-image-2.6.26-2-$flavor linux-image-2.6.26-2-$flavor/preinst/lilo-initrd-2.6.26-2-$flavor boolean false" done | debconf-set-selections # deterministic /bin/sh on upgrades echo "dash dash/sh boolean true" | debconf-set-selections fi piuparts/custom-scripts/scripts/pre_install_extras0000755000000000000000000000465713042113740020140 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts extra for package $PIUPARTS_OBJECTS" } # # these extras are only for the initial package installation # if [ "$PIUPARTS_PHASE" != "install" ]; then exit 0 fi case ${PIUPARTS_OBJECTS%%=*} in *dkms|\ ktap|\ zfs-*|\ zfsutils-linux) # Install kernel headers, so that dkms tries to build a module HEADERS=linux-headers FLAVOR=unknown FLAVOR_i386=686-pae case $PIUPARTS_DISTRIBUTION in lenny*|squeeze*) HEADERS=linux-headers-2.6 FLAVOR_i386=686 ;; esac case $(dpkg --print-architecture) in amd64) FLAVOR=amd64 ;; i386) FLAVOR=$FLAVOR_i386 ;; esac case ${PIUPARTS_OBJECTS} in *=None) # do not test discontinued packages - they won't support newer kernels ;; *) log_debug apt-get -y install $HEADERS-$FLAVOR ;; esac ;; firmware-ipw2x00) log_debug #echo "firmware-ipw2x00 firmware-ipw2x00/license/accepted boolean true" | debconf-set-selections ;; firmware-ivtv) log_debug #echo "firmware-ivtv firmware-ivtv/license/accepted boolean true" | debconf-set-selections ;; fonts-mathematica) log_debug #echo "fonts-mathematica mathematica-fonts/accept_license boolean true" | debconf-set-selections ;; mathematica-fonts|ttf-mathematica4.1) log_debug apt-get -y install wget #echo "mathematica-fonts mathematica-fonts/accept_license boolean true" | debconf-set-selections if [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then log_debug #echo "ttf-mathematica4.1 ttf-mathematica4.1/accept_license boolean true" | debconf-set-selections fi ;; steam|steam-devices|steamcmd) log_debug #echo "steam steam/question select I AGREE" | debconf-set-selections ;; zekr-quran-translations-en) log_debug #echo "zekr-quran-translations-en quran-translations-en/accepted-Zekr-Quran-Trans boolean true" | debconf-set-selections ;; esac if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ] || \ [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then case ${PIUPARTS_OBJECTS%%=*} in *sun-java5-*|*sun-java6-*) log_debug #echo "sun-java6-jre shared/accepted-sun-dlj-v1-1 boolean true" | debconf-set-selections ;; jajuk|\ liblucene2-java|\ libworldwind-java|\ omegat|\ robocode|\ solr-common|\ solr-tomcat5.5|\ weirdx|\ weka|\ worldwind) # these packages depend on sun-java* only log_debug #echo "sun-java6-jre shared/accepted-sun-dlj-v1-1 boolean true" | debconf-set-selections ;; esac fi piuparts/custom-scripts/scripts/pre_install_foreign_architecture0000755000000000000000000000072313036653026023024 0ustar #!/bin/sh set -e test "$PIUPARTS_PHASE" = "install" || exit 0 case "$PIUPARTS_DISTRIBUTION" in lenny*|squeeze*|wheezy*) exit 0 ;; esac FOREIGN= case ${PIUPARTS_OBJECTS%%=*} in nvidia-driver|\ nvidia-legacy-*xx-driver|\ lib*cuda1) FOREIGN="i386" ;; esac if [ -n "$FOREIGN" ] && [ "$FOREIGN" != "$(dpkg --print-architecture)" ]; then echo "Enabling foreign architecture $FOREIGN for $PIUPARTS_OBJECTS" dpkg --add-architecture $FOREIGN apt-get update fi piuparts/custom-scripts/scripts/pre_install_foreign_architecture_i3860000755000000000000000000000076012567171321023577 0ustar #!/bin/sh set -e test "$PIUPARTS_PHASE" = "install" || exit 0 case "$PIUPARTS_DISTRIBUTION" in lenny*|squeeze*) exit 0 ;; esac test "$(dpkg --print-architecture)" = "amd64" || exit 0 case ${PIUPARTS_OBJECTS%%=*} in ia32-libs|ia32-libs-gtk) ;; libwine-unstable|libwine-*-unstable|wine|wine-unstable) ;; boinc-nvidia-cuda|teamspeak-client) ;; lib32nss-mdns) ;; *) exit 0 ;; esac echo "Enabling foreign architecture i386 for $PIUPARTS_OBJECTS" dpkg --add-architecture i386 apt-get update piuparts/custom-scripts/scripts/pre_remove_40_find_missing_md5sums0000755000000000000000000000317113061276553023115 0ustar #!/bin/sh set -e is_installed() { local pkg="$1" dpkg-query -s "$pkg" >/dev/null 2>&1 || return 1 local status="$(dpkg-query -W -f '${Status}' $pkg)" test "$status" != "unknown ok not-installed" || return 1 test "$status" != "deinstall ok config-files" || return 1 return 0 } # skip the md5sum check if /usr/share/doc is pruned test ! -f /etc/dpkg/dpkg.cfg.d/piuparts-path-exclude || exit 0 for pkg in ${PIUPARTS_OBJECTS%%=*} do # skip check if the package is not installed is_installed "$pkg" || continue md5file="/var/lib/dpkg/info/$pkg.md5sums" test -f "$md5file" || md5file="/var/lib/dpkg/info/$pkg:$(dpkg --print-architecture).md5sums" if [ ! -f "$md5file" ]; then echo "MD5SUM FILE NOT FOUND FOR $pkg" continue fi f1=/var/run/f1.$$ sed -r 's%^[0-9a-f]{32} %/%' "$md5file" | sort >$f1 f2=/var/run/f2.$$ >$f2 dpkg -L "$pkg" | sed 's,\\,\\\\,' | sort | \ while read f ; do if [ "$f" != "${f#package diverts others to: }" ]; then : # skip diversion information elif [ "$f" != "${f#locally diverted to: }" ]; then : # skip diversion information elif [ -d "$f" ]; then : # ignore directories elif [ "$f" = "/home" ]; then : # ignore directory shipped by base-files that is # turned into a file by our post_setup_forbid_home script elif [ -L "$f" ]; then : # ignore links elif [ -z "${f%%/etc/*}" ]; then : # ignore files in /etc - probably conffiles elif [ ! -e "$f" ]; then echo "${pkg}: MISSING OBJECT $f" else echo "$f" >> $f2 fi done comm -13 $f1 $f2 | sed "s/^/${pkg}: FILE WITHOUT MD5SUM /" comm -23 $f1 $f2 | sed "s/^/${pkg}: MD5SUM WITHOUT FILE /" rm -f $f1 $f2 done piuparts/custom-scripts/scripts/pre_remove_40_find_obsolete_conffiles0000755000000000000000000000115013042113740023611 0ustar #!/bin/sh set -e for pkg in ${PIUPARTS_OBJECTS%%=*} do dpkg-query -W -f='${Conffiles}\n' $pkg | \ grep ' obsolete$' | \ while read file md5expected obs do info="OBSOLETE CONFFILE $file REGISTERED BY $pkg" query=$(dpkg-query -S "$file" || echo "???: $file") owner=${query%: ${file}} if [ "${owner%:*}" != "$pkg" ]; then info="${info} OWNER CHANGED TO $owner" fi if [ ! -f "$file" ]; then info="${info} (MISSING)" else md5=$(md5sum "$file" | awk '{ print $1 }') if [ "$md5expected" != "$md5" ]; then info="${info} (MODIFIED)" fi fi echo "$info" done done piuparts/custom-scripts/scripts/pre_remove_40_find_unowned_lib_links0000755000000000000000000000170112762537303023471 0ustar #!/bin/sh set -e for libdir in /lib /usr/lib /lib/*-gnu* /usr/lib/*-gnu* do test -d "$libdir" || continue for f in "$libdir"/* do test ! -d "$f" || continue test -L "$f" || continue rl=$(readlink "$f") test -n "${rl##/etc/alternatives/*}" || continue if ! dpkg-query -S "$f" >/dev/null 2>&1 then case "$f" in /lib/ld-lsb.so.?) # created by lsb-core continue ;; /usr/lib/libglide.so.2|\ /usr/lib/libglide3.so.3) # created by maintainer script continue ;; /usr/lib/libfmpich.so.12|\ /usr/lib/libmpich.so.12|\ /usr/lib/libmpichcxx.so.12|\ /usr/lib/libmpichf90.so.12|\ /usr/lib/libmpichfort.so.12) if [ -f /etc/alternatives/libmpi.so ] then # managed by mpi-default-dev via alternatives # to /usr/lib//lib${impl}.so # ldconfig adds the "real" soname links continue fi ;; esac echo "UNOWNED SYMLINK $f -> $rl" fi done done piuparts/custom-scripts/scripts/pre_remove_50_find_bad_permissions0000755000000000000000000000345513036653026023152 0ustar #!/bin/sh set -e case "$PIUPARTS_DISTRIBUTION" in lenny) case ${PIUPARTS_OBJECTS%%=*} in linpopup) # package removed after lenny for file in /var/lib/linpopup/messages.dat do test ! -f "$file" || chmod -c o-w "$file" done ;; esac ;; esac case "$PIUPARTS_DISTRIBUTION" in lenny|squeeze*) case ${PIUPARTS_OBJECTS%%=*} in citadel-server|citadel-dbg|citadel-mta|citadel-suite|bcron-run|capisuite|debbugs|raccess4vbox3|smartlist|sxid) #WORKSAROUND #684964: citadel-server: world writable config file: /etc/citadel/netconfigs/7 for file in /etc/citadel/netconfigs/7 /etc/citadel/refcount_adjustments.dat /etc/citadel/citadel.control do test ! -f "$file" || chmod -c o-w "$file" done ;; esac ;; esac case "$PIUPARTS_DISTRIBUTION" in wheezy*) case ${PIUPARTS_OBJECTS%%=*} in citadel-server|citadel-dbg|citadel-mta|citadel-suite|bcron|bcron-run|capisuite|debbugs|exmh|nmh|raccess4vbox3|smartlist|xlbiff) #WORKSAROUND #684964: citadel-server: world writable config file: /etc/citadel/netconfigs/7 for file in /etc/citadel/netconfigs/7 /etc/citadel/refcount_adjustments.dat /var/lib/citadel/data/refcount_adjustments.dat do test ! -f "$file" || chmod -c o-w "$file" done ;; esac ;; esac case ${PIUPARTS_OBJECTS%%=*} in gpe-tetris|gpe) #WORKSAROUND #684178: gpe-tetris: creates world writable directory /var/games/gpe # package removed after wheezy for file in /var/games/gpe/gpe-tetris.dat do test ! -f "$file" || chmod -c o-w "$file" done for dir in /var/games/gpe do test ! -d "$dir" || chmod -c o-w "$dir" done ;; esac # find world writables without sticky bit BADPERMS=$(find / -mount ! -type l ! -type c ! -type p ! -type s -perm -o+w ! -perm -1000) if [ -n "$BADPERMS" ]; then echo "ERROR: BAD PERMISSIONS" ls -lad $BADPERMS exit 1 fi piuparts/custom-scripts/scripts/pre_remove_50_find_missing_copyright0000755000000000000000000000251512762537303023532 0ustar #!/bin/sh set -e is_installed() { local pkg="$1" dpkg-query -s "$pkg" >/dev/null 2>&1 || return 1 local status="$(dpkg-query -W -f '${Status}' $pkg)" test "$status" != "unknown ok not-installed" || return 1 test "$status" != "deinstall ok config-files" || return 1 return 0 } failed= for pkg in ${PIUPARTS_OBJECTS%%=*} do # ignore failures for some old packages with many rdepends ignore= case "${pkg}_${PIUPARTS_DISTRIBUTION}" in gij_lenny) ignore=1 ;; gnumeric-common_lenny) ignore=1 ;; #554201 libuim6_lenny) ignore=1 ;; #554204 libuim-data_lenny) ignore=1 ;; #554200 mozilla-plugin-vlc_lenny) ignore=1 ;; #687657 postgresql-8.3-plsh_lenny) ignore=1 ;; # removed after lenny vlc_lenny) ignore=1 ;; #687657 cdd-common_squeeze) ignore=1 ;; #692946 libfbclient2_squeeze) ignore=1 ;; #692948 libcucul0_wheezy) ignore=1 ;; # removed esac # skip check if the package is not installed is_installed "$pkg" || continue docdir="/usr/share/doc/$pkg" copyright="$docdir/copyright" if [ ! -f "$copyright" ] then if [ -n "$ignore" ]; then echo "ignoring failure of $pkg on $PIUPARTS_DISTRIBUTION" else failed="$failed $copyright" fi echo "MISSING COPYRIGHT FILE: $copyright" echo "# ls -lad $docdir" ls -lad "$docdir" || true echo "# ls -la $docdir/" ls -la "$docdir/" || true fi done piuparts/custom-scripts/scripts/pre_remove_exceptions0000755000000000000000000000372313061276553020650 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } case ${PIUPARTS_OBJECTS%%=*} in file-rc) case "$PIUPARTS_DISTRIBUTION" in lenny|squeeze*) log_debug # removal wont work if sysv-rc isn't reinstalled yes 'Yes, do as I say!' | apt-get -y --force-yes install sysv-rc ;; esac ;; isdnlog|isdnutils) #WORKSAROUND #431855: fails with "There are still files in /etc/isdn/ that you may want to remove manually." if [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then log_debug rm -fv /etc/isdn/* fi ;; apt-listbugs) # cleanup from post_install_exceptions log_debug rm -f /usr/sbin/apt-listbugs dpkg-divert --remove --rename /usr/sbin/apt-listbugs ;; ffgtk|\ roger-router|\ roger-router-cli) # cleanup from pre_install_exceptions log_debug dpkg-divert --remove --rename /usr/sbin/lpadmin ;; freedombox-setup) log_debug # cleanup from pre_install_exceptions if [ -e /etc/motd.orig ]; then if [ -L /etc/motd ]; then rm -v /etc/motd fi if [ ! -e /etc/motd ]; then mv -v /etc/motd.orig /etc/motd fi fi ;; reseed) log_debug # cleanup from pre_install_exceptions test ! -L /usr/sbin/reseed || rm /usr/sbin/reseed dpkg-divert --remove --rename /usr/sbin/reseed ;; esac case "$PIUPARTS_DISTRIBUTION" in stretch*|sid) case ${PIUPARTS_OBJECTS%%=*} in init|\ education-thin-client|\ ltsp-client|\ ltsp-client-core) log_debug # requires removal of essential packages if [ ! -f /etc/apt/apt.conf.d/piuparts-allow-remove-essential ] then echo "Enabling APT::Get::allow-remove-essential" echo 'APT::Get::allow-remove-essential "true";' >> /etc/apt/apt.conf.d/piuparts-allow-remove-essential fi ;; esac ;; esac # Allow removal of the kernel running on the host from the chroot. UNAME_R="$(uname -r)" echo "linux-image-$UNAME_R linux-image-$UNAME_R/prerm/removing-running-kernel-$UNAME_R boolean false" | debconf-set-selections piuparts/custom-scripts/scripts/pre_test_root_password0000755000000000000000000000034112567171321021044 0ustar #!/bin/sh set -e # sudo refuses removal if no root password is set, so set one # do this unconditionally, as there are quite some packages depending on sudo # (and since its harmless and fast) yes "yes" 2>/dev/null | passwd piuparts/custom-scripts/scripts-apt-first/0000755000000000000000000000000012762537303016220 5ustar piuparts/custom-scripts/scripts-apt-first/pre_distupgrade_zz_apt_first0000755000000000000000000000007012762537303024122 0ustar #!/bin/sh set -e apt-get update apt-get -y install apt piuparts/custom-scripts/scripts-debug-packagemanager/0000755000000000000000000000000012567171321020316 5ustar piuparts/custom-scripts/scripts-debug-packagemanager/pre_distupgrade_debug_packagemanager0000755000000000000000000000034012567171321027576 0ustar #!/bin/sh set -e if [ ! -f /etc/apt/apt.conf.d/piuparts-debug-packagemanager ] then echo "Enabling Debug::pkgPackageManager" echo 'Debug::pkgPackageManager "true";' >> /etc/apt/apt.conf.d/piuparts-debug-packagemanager fi piuparts/custom-scripts/scripts-debug-problemresolver/0000755000000000000000000000000012567171321020612 5ustar piuparts/custom-scripts/scripts-debug-problemresolver/pre_distupgrade_debug_problemresolver0000755000000000000000000000052012567171321030366 0ustar #!/bin/sh set -e if [ ! -f /etc/apt/apt.conf.d/piuparts-debug-problemresolver ] then echo "Enabling Debug::pkgProblemResolver" echo 'Debug::pkgProblemResolver "true";' >> /etc/apt/apt.conf.d/piuparts-debug-problemresolver echo 'Debug::pkgProblemResolver::ShowScores "true";' >> /etc/apt/apt.conf.d/piuparts-debug-problemresolver fi piuparts/custom-scripts/scripts-debug-problemresolver/pre_install_debug_problemresolver0000755000000000000000000000057512567171321027533 0ustar #!/bin/sh set -e test "$PIUPARTS_TEST" = "install" || exit 0 if [ ! -f /etc/apt/apt.conf.d/piuparts-debug-problemresolver ] then echo "Enabling Debug::pkgProblemResolver" echo 'Debug::pkgProblemResolver "true";' >> /etc/apt/apt.conf.d/piuparts-debug-problemresolver echo 'Debug::pkgProblemResolver::ShowScores "true";' >> /etc/apt/apt.conf.d/piuparts-debug-problemresolver fi piuparts/custom-scripts/scripts-debug-purge/0000755000000000000000000000000013036653026016511 5ustar piuparts/custom-scripts/scripts-debug-purge/post_remove_postrm_set-x0000755000000000000000000000076213036653026023532 0ustar #!/bin/sh set -e set_minus_x() { local script=$1 if [ -f $script ]; then if head -n 1 $script | grep -qE '/bin/(ba)?sh' ; then echo "DEBUG PURGE: enabling 'set -x' in ${script##*/}" sed -i '2 i set -x' $script else echo "Unsupported script type in $script:" head -n 1 $script fi fi } arch=$(dpkg --print-architecture) for target in ${PIUPARTS_OBJECTS%%=*} do pkg=${target} set_minus_x /var/lib/dpkg/info/$pkg.postrm set_minus_x /var/lib/dpkg/info/$pkg:$arch.postrm done piuparts/custom-scripts/scripts-debug-remove/0000755000000000000000000000000013036653026016664 5ustar piuparts/custom-scripts/scripts-debug-remove/pre_remove_prerm_postrm_set-x0000755000000000000000000000111713036653026024706 0ustar #!/bin/sh set -e set_minus_x() { local script=$1 if [ -f $script ]; then if head -n 1 $script | grep -qE '/bin/(ba)?sh' ; then echo "DEBUG REMOVE: enabling 'set -x' in ${script##*/}" sed -i '2 i set -x' $script else echo "Unsupported script type in $script:" head -n 1 $script fi fi } arch=$(dpkg --print-architecture) for target in ${PIUPARTS_OBJECTS%%=*} do pkg=${target} set_minus_x /var/lib/dpkg/info/$pkg.prerm set_minus_x /var/lib/dpkg/info/$pkg.postrm set_minus_x /var/lib/dpkg/info/$pkg:$arch.prerm set_minus_x /var/lib/dpkg/info/$pkg:$arch.postrm done piuparts/custom-scripts/scripts-jessie/0000755000000000000000000000000012567171321015566 5ustar piuparts/custom-scripts/scripts-jessie/post_setup_jessie-fake-essential0000755000000000000000000000133212567171321024153 0ustar #!/bin/sh set -e # The following issues won't be fixed in jessie: # - unconditional use of ucf during postrm purge # so add these packages to the "fake" essential set. USED_DURING_PURGE="ucf" # For purging configuration from /var/lib/systemd/ INITSYSTEMHELPERS="init-system-helpers" case ${PIUPARTS_DISTRIBUTION} in lenny*|squeeze*|wheezy*) # package does not exist INITSYSTEMHELPERS="" ;; esac case ${PIUPARTS_OBJECTS%%=*} in dpkg) # don't install fake essential packages while creating the tarball exit 0 ;; ucf|init-system-helpers) # allow testing of the fake essential packages exit 0 ;; esac echo "*** Adding fake essential packages ***" apt-get install -yf $USED_DURING_PURGE $INITSYSTEMHELPERS piuparts/custom-scripts/scripts-leftovers/0000755000000000000000000000000013036653026016314 5ustar piuparts/custom-scripts/scripts-leftovers/post_purge_manual_cleanup0000755000000000000000000000102312762537303023475 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } remove_ssl_cert() { for c in /etc/ssl/certs/*.0 ; do if [ -L "$c" ] && [ "$(readlink "$c")" = "$1" ]; then rm -fv "$c" fi done rm -fv /etc/ssl/certs/$1 rmdir --ignore-fail-on-non-empty /etc/ssl/certs } case ${PIUPARTS_OBJECTS%%=*} in uw-imapd) log_debug remove_ssl_cert imapd.pem ;; ipopd) log_debug remove_ssl_cert ipop3d.pem ;; smartlist) log_debug rm -fv /etc/aliases.????-??-??.??:??:?? ;; esac piuparts/custom-scripts/scripts-leftovers/post_remove_cleanup0000755000000000000000000000043712567171321022320 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } case ${PIUPARTS_OBJECTS%%=*} in dovecot-core) # #330519 - does not remove certificates on purge log_debug rm -fv /etc/dovecot/dovecot.pem /etc/dovecot/private/dovecot.pem ;; esac piuparts/custom-scripts/scripts-leftovers/post_setup_fake-essential0000755000000000000000000000156512567171321023432 0ustar #!/bin/sh set -e # If dbconfig-common was already purged, packages that have used # dbconfig-common in postinst will leave configuration files in # /etc/dbconfig-common DBCONFIG="dbconfig-common" # For purging configuration from /var/lib/systemd/ INITSYSTEMHELPERS="init-system-helpers" case ${PIUPARTS_DISTRIBUTION} in lenny*|squeeze*|wheezy*) # package does not exist INITSYSTEMHELPERS="" ;; esac # openssl may be used during purge to compute the hash for a # certificate, otherwise files in /etc/ssl/certs can't be removed. OPENSSL="" case ${PIUPARTS_OBJECTS%%=*} in dpkg) # skip while creating the tarball exit 0 ;; dbconfig-common|init-system-helpers) # allow testing of the fake essential packages exit 0 ;; stone) OPENSSL="openssl" ;; esac echo "*** Adding fake essential packages ***" apt-get install -yf $DBCONFIG $INITSYSTEMHELPERS $OPENSSL piuparts/custom-scripts/scripts-leftovers/pre_remove_preseed_cleanup0000755000000000000000000000562013036653026023626 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } dbconfig_enable_purge() { log_debug echo "$1 $1${2+/$2}/dbconfig-remove boolean true" | debconf-set-selections echo "$1 $1${2+/$2}/purge boolean true" | debconf-set-selections } # # enable extended purge mode that is available in some packages # but disabled by default because it might remove valuable user or # application data # case ${PIUPARTS_OBJECTS%%=*} in cvsd) log_debug echo "cvsd cvsd/remove_chroot boolean true" | debconf-set-selections ;; docbookwiki) log_debug echo "docbookwiki docbookwiki/purge_books boolean true" | debconf-set-selections ;; ifetch-tools) log_debug echo "ifetch-tools ifetch-tools/purge boolean true" | debconf-set-selections ;; mlmmj) #668752 log_debug echo "mlmmj mlmmj/remove-on-purge boolean true" | debconf-set-selections ;; pluxml) log_debug echo "pluxml pluxml/system/purgedata boolean true" | debconf-set-selections ;; ironic-common) dbconfig_enable_purge ironic-common ;; movabletype-opensource) dbconfig_enable_purge movabletype-opensource ;; nova-common) dbconfig_enable_purge nova-common ;; otrs2) dbconfig_enable_purge otrs2 ;; pdns-backend-sqlite) dbconfig_enable_purge pdns-backend-sqlite ;; pdns-backend-sqlite3) dbconfig_enable_purge pdns-backend-sqlite3 ;; redmine) dbconfig_enable_purge redmine instances/default ;; request-tracker3.8) dbconfig_enable_purge request-tracker3.8 ;; request-tracker4) dbconfig_enable_purge request-tracker4 ;; esac if [ -d /var/lib/mysql ]; then log_debug echo "Enabling MySQL database purge." echo "mysql-server-5.1 mysql-server-5.1/postrm_remove_databases boolean true" | debconf-set-selections echo "mysql-server-5.5 mysql-server-5.5/postrm_remove_databases boolean true" | debconf-set-selections echo "mysql-server-5.6 mysql-server-5.6/postrm_remove_databases boolean true" | debconf-set-selections echo "mysql-server-5.7 mysql-server-5.7/postrm_remove_databases boolean true" | debconf-set-selections # wrong template name is used in jessie echo "mariadb-server-10.0 mysql-server-10.0/postrm_remove_databases boolean true" | debconf-set-selections echo "mariadb-server-10.0 mariadb-server-10.0/postrm_remove_databases boolean true" | debconf-set-selections echo "mariadb-server-10.1 mariadb-server-10.1/postrm_remove_databases boolean true" | debconf-set-selections echo "percona-xtradb-cluster-server-5.5 mysql-server-5.1/postrm_remove_databases boolean true" | debconf-set-selections fi if [ -d /var/lib/ldap ]; then log_debug echo "Enabling LDAP database purge." echo "slapd slapd/purge_database boolean true" | debconf-set-selections fi if [ -d /var/lib/cyrus ] || [ -d /var/spool/cyrus ] || [ -d /var/spool/sieve ]; then log_debug echo "Enabling Cyrus spool purge." echo "cyrus-common cyrus-common/removespools boolean true" | debconf-set-selections fi exit 0 piuparts/custom-scripts/scripts-log-alternatives/0000755000000000000000000000000013061276553017567 5ustar piuparts/custom-scripts/scripts-log-alternatives/post_install_log_alternatives0000755000000000000000000000041513061276553025652 0ustar #!/bin/sh set -e [ ! -e /usr/bin/update-alternatives.orig ] && exit 0 rm /usr/bin/update-alternatives dpkg-divert --rename --remove /usr/bin/update-alternatives if [ -f /tmp/log-alternatives.log ]; then cat /tmp/log-alternatives.log rm /tmp/log-alternatives.log fi piuparts/custom-scripts/scripts-log-alternatives/pre_install_log_alternatives0000755000000000000000000000100213061276553025444 0ustar #!/bin/sh set -e # Do nothing if the script already ran. # The pre_install step can be run multiple times. [ -e /usr/bin/update-alternatives.orig ] && exit 0 dpkg-divert --divert /usr/bin/update-alternatives.orig --rename /usr/bin/update-alternatives cat >/usr/bin/update-alternatives <<'EOT' #!/bin/sh echo "LOG-ALTERNATIVES: dpkg=${DPKG_MAINTSCRIPT_PACKAGE}: piuparts=${PIUPARTS_OBJECTS}: $0 $@" >>/tmp/log-alternatives.log exec /usr/bin/update-alternatives.orig "$@" EOT chmod +x /usr/bin/update-alternatives piuparts/custom-scripts/scripts-no-usr-share-doc/0000755000000000000000000000000012762537303017375 5ustar piuparts/custom-scripts/scripts-no-usr-share-doc/post_setup_disable_usr_share_doc0000755000000000000000000000256512762537303026123 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } case ${PIUPARTS_OBJECTS%%=*} in dpkg) # skip while creating the tarball exit 0 ;; esac # clear out /usr/share/doc # but keep the directories (and symlinks) as well as the copyright files cat << EOF > /etc/dpkg/dpkg.cfg.d/piuparts-path-exclude path-exclude=/usr/share/doc/*/* path-include=/usr/share/doc/*/copyright EOF # switching init systems forth and back will clean out /usr/share/doc # reinstalling the affected packages beforehand makes the files disappear # before the snapshot of the reference system is created CANDIDATES="systemd systemd-sysv sysv-rc" CANDIDATES=$(dpkg-query -W $CANDIDATES | awk '{ if ($2) { print $1 } }') if [ -n "$CANDIDATES" ]; then echo "Reinstalling $(echo $CANDIDATES)..." # workaround apt bug #770291 - do it one by one, not all at once for package in $CANDIDATES do apt-get -u --reinstall install $package done fi case ${PIUPARTS_OBJECTS%%=*} in localepurge) case ${PIUPARTS_DISTRIBUTION} in lenny*|squeeze*) ;; *) # reinstall packages that will be reinstalled after purge # to not record their /usr/share/doc content that is about to disappear log_debug EXTRA="" apt-get -u --reinstall --fix-missing install $(dpkg -S LC_MESSAGES 'man/??/man' | cut -d: -f1 | tr ', ' '\n' | sort -u) $EXTRA ;; esac ;; esac piuparts/custom-scripts/scripts-squeeze/0000755000000000000000000000000013037404512015757 5ustar piuparts/custom-scripts/scripts-squeeze/post_distupgrade_squeeze-fake-essential0000755000000000000000000000050512567171321025725 0ustar #!/bin/sh set -e test "$PIUPARTS_DISTRIBUTION" = "squeeze" || exit 0 case ${PIUPARTS_OBJECTS%%=*} in kmplayer-plugin|kde-core|konqueror|konq-plugins|kwin-baghira|mozart-doc|smb4k|strigi-applet|texlive-full) # work around #601961: apt: wrongly thinks install-info is essential apt-get -y install install-info ;; esac piuparts/custom-scripts/scripts-squeeze/post_remove_exceptions_lenny0000755000000000000000000000057012567171321023725 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } case "$PIUPARTS_DISTRIBUTION" in lenny) : ;; *) exit 0 ;; esac case ${PIUPARTS_OBJECTS%%=*} in postgis) #WORKSAROUND #560409: postgis can't be purged if postgresql # is not installed due to missing directory log_debug mkdir -p /usr/lib/postgresql/8.3/lib ;; esac piuparts/custom-scripts/scripts-squeeze/post_remove_exceptions_squeeze0000755000000000000000000000061012762537303024257 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } case "$PIUPARTS_DISTRIBUTION" in squeeze) : ;; *) exit 0 ;; esac case ${PIUPARTS_OBJECTS%%=*} in libblkid-dev) #775350 - unhandled symlink to directory conversion if [ ! -f /usr/share/doc/libblkid1/copyright ]; then log_debug apt-get install --reinstall libblkid1 fi ;; esac piuparts/custom-scripts/scripts-squeeze/post_setup_squeeze-fake-essential0000755000000000000000000000313413037404512024545 0ustar #!/bin/sh set -e # The following issues won't be fixed in squeeze: # - unconditional use of deluser during postrm purge # - unconditional use of ucf during postrm purge # so add these packages to the "fake" essential set. USED_DURING_PURGE="adduser ucf" FAILS_TO_REMOVE= case ${PIUPARTS_OBJECTS%%=*} in dpkg) # don't install fake essential packages while creating the tarball exit 0 ;; adduser|ucf) # allow testing of the fake essential packages exit 0 ;; gosa-desktop|kde-core|kde-full|kde-plasma-desktop|kde-standard|kdebase-apps|konqueror|konq-plugins|mozart-doc|texlive-full) # work around #601961: apt: wrongly thinks install-info is essential case ${PIUPARTS_DISTRIBUTION} in squeeze|squeeze-lts) FAILS_TO_REMOVE="$FAILS_TO_REMOVE install-info" ;; esac ;; docbookwiki) USED_DURING_PURGE="$USED_DURING_PURGE mysql-client" ;; phpbb3) USED_DURING_PURGE="$USED_DURING_PURGE dbconfig-common" ;; prelude-manager) #WORKSAROUND #660455 USED_DURING_PURGE="$USED_DURING_PURGE dbconfig-common" ;; drupal6|moodle|moodle-book|moodle-debian-edu-theme|scuttle) if [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then USED_DURING_PURGE="$USED_DURING_PURGE wwwconfig-common" fi ;; octave-audio|octave-symbolic|octave-vrml) if [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then USED_DURING_PURGE="$USED_DURING_PURGE octave3.0" fi ;; ttf-beteckna) #502707 if [ "$PIUPARTS_DISTRIBUTION" = "lenny" ]; then USED_DURING_PURGE="$USED_DURING_PURGE defoma" fi ;; esac echo "*** Adding fake essential packages ***" apt-get install -yf $USED_DURING_PURGE $FAILS_TO_REMOVE piuparts/custom-scripts/scripts-sysvinit/0000755000000000000000000000000012762537303016177 5ustar piuparts/custom-scripts/scripts-sysvinit/pre_distupgrade_zz_sysvinit0000755000000000000000000000017712762537303024006 0ustar #!/bin/sh set -e if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "jessie" ]; then apt-get update apt-get -y install sysvinit-core fi piuparts/custom-scripts/scripts-unused-examples/0000755000000000000000000000000013036653026017422 5ustar piuparts/custom-scripts/scripts-unused-examples/post_chroot_unpack_key_setup0000755000000000000000000000056413036653026025351 0ustar #!/bin/sh # we rely on gnupg and wget being available, # make sure to use "--include=gnupg,wget" in your debootstrap cmdline echo "Setting up https://example.com/internal_key.asc for apt-get usage." wget -O - 'https://example.com/internal_key.asc' | apt-key add - echo "Running apt-get update to have a verified and working Debian repository available." apt-get update piuparts/custom-scripts/scripts-wheezy/0000755000000000000000000000000012567171321015617 5ustar piuparts/custom-scripts/scripts-wheezy/post_setup_wheezy-fake-essential0000755000000000000000000000104612567171321024237 0ustar #!/bin/sh set -e # The following issues won't be fixed in wheezy: # - unconditional use of deluser during postrm purge # - unconditional use of ucf during postrm purge # so add these packages to the "fake" essential set. USED_DURING_PURGE="adduser ucf" case ${PIUPARTS_OBJECTS%%=*} in dpkg) # don't install fake essential packages while creating the tarball exit 0 ;; adduser|ucf) # allow testing of the fake essential packages exit 0 ;; esac echo "*** Adding fake essential packages ***" apt-get install -yf $USED_DURING_PURGE piuparts/custom-scripts/scripts-wheezy/pre_distupgrade_wheezy0000755000000000000000000000416412567171321022326 0ustar #!/bin/sh set -e log_debug() { echo "Debug: piuparts exception for package $PIUPARTS_OBJECTS" } # Work around bug where apt/squeeze prefers to 'keep back' some # upgradable packages instead of removing some obsolete ones. FORCEREMOVE= if [ "$PIUPARTS_DISTRIBUTION_NEXT" = "wheezy" ] || \ [ "$PIUPARTS_DISTRIBUTION_NEXT" = "wheezy-proposed" ] then case ${PIUPARTS_OBJECTS%%=*} in gnustep|\ gnustep-games|\ gnustep-back0.18-cairo|\ gnustep-back0.18-art|\ gnustep-back-dbg|\ libgnustep-base1.20-dbg|\ addressmanager.app|\ biococoa.app|\ easydiff.app|\ gnumail.app|\ gnumail.app-dbg|\ mines.app|\ projectmanager.app|\ INVALID) FORCEREMOVE="libobjc2" ;; libahven17.0|\ libalog0.3-full|\ libalog1-full-dev|\ libalog-full-dbg|\ libapq3.0|\ libapq-postgresql3.0|\ libaws2.7|\ libaws2.7-dev|\ libaws-dbg|\ libflorist2009|\ libflorist-dbg|\ libgmpada1|\ libgmpada1-dbg|\ libgnomeada2.14.2|\ libgnomeada2.14.2-dbg|\ libgnomeada2.14.2-dev|\ libgtkada2.14.2|\ libgtkada2.14.2-dbg|\ libgtkada2.14.2-dev|\ libgtkada-gl2.14.2|\ libgtkada-glade2.14.2|\ libgtkada2-bin|\ liblog4ada0|\ liblog4ada-dbg|\ libnarval1.10.1|\ libnarval1-dev|\ libnarval-dbg|\ libpcscada0.6|\ libplplot-ada|\ libplplot-dev|\ libpolyorb2|\ libtemplates-parser11.5|\ libtexttools4|\ libtexttools4-dbg|\ libtexttools-doc|\ libxmlada3.2|\ libxmlezout0|\ libxmlezout-dbg|\ cl-plplot|\ gprbuild|\ narval-doc|\ narval-generic-actors|\ narval-servers|\ narval-tests-actors|\ narval-utils|\ polyorb-servers|\ INVALID) FORCEREMOVE="gnat-4.4-base" ;; dolfin-bin|\ dolfin-dev|\ dolfin-doc|\ petsc-dev|\ python-dolfin|\ INVALID) # only affects i386 FORCEREMOVE="libjpeg62-dev" ;; science-physics) # #706111 FORCEREMOVE="tessa" ;; firestarter|\ gnome-netstatus-applet|\ gnome-utils|\ INVALID) FORCEREMOVE="gconf2" ;; esac fi if [ -n "$FORCEREMOVE" ]; then if dpkg-query -s $FORCEREMOVE >/dev/null 2>&1 ; then log_debug echo "Forcibly removing $FORCEREMOVE for smoother upgrade" dpkg -r --force-depends $FORCEREMOVE fi fi piuparts/debian/0000755000000000000000000000000013061276553011045 5ustar piuparts/debian/.gitignore0000644000000000000000000000014512567171321013032 0ustar files piuparts.debhelper.log piuparts.postinst.debhelper piuparts.prerm.debhelper piuparts.substvars piuparts/debian/changelog0000644000000000000000000036750613061276553012740 0ustar piuparts (0.76) unstable; urgency=medium [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /usr/share/texmf/ls-R - Remove from ignored_files: - /usr/share/keyrings/debian-archive-removed-keys.gpg~ - Do not require --scriptsdir for proper operation. (Closes: #855038) - Make install_over_symlink tests fail by default, add --warn-on-install-over-symlink option to demote this to a warning. - Add --install-suggests option. - Fall back to generating the reference chroot state on-the-fly if the --end-meta file is missing. - Make logged command lines shell-safe with pipes.quote(). * piuparts.conf: - Use --warn-on-install-over-symlink for tests ending before stretch. - Enable scripts-log-alternatives for [sid] and [stretch]. (See: #850917) - Add new suite: wheezy222testing, upgrading wheezy -> jessie -> stretch. * piuparts.conf.anbe: Add some more example sections. * distros.conf: Add support for sid-debug, stretch-debug. * piuparts-slave.py: - Replace literal ESC (0x1b) chars with string '[ESC]' for easier copy+paste from logfiles. - Make logged command lines shell-safe with pipes.quote(). * piuparts-analyze.py: - Also process untestable/*.log. * instances/*: Generate the config files piuparts.conf.pejacevic, piuparts.conf.piu-slave-bm-a, piuparts.conf.piu-slave-ubc-01 (new) and piuparts.conf.piu-slave-1und1-01 (unused) from a common template to keep the settings in sync, since they should only differ in mirror and arch. * Clean up leftover /etc/piuparts/scripts/post_setup_experimental. (Closes: #854317) * scripts/post_setup_minimize: - Exclude gcc-4.8-base from jessie chroot. * scripts/pre_install_database-server: Add/fix some database requirements. * scripts/pre_distupgrade_zz_database-server: Switch from mysql-server to default-mysql-server (aka mariadb-server) for selected packages on upgrades to stretch, otherwise they will lose their database server. Try 'dpkg --configure --pending' after failed upgrades in case the database server was started too late - after a client tried to access it. * scripts/pre_remove_40_find_missing_md5sums: Correctly escape backslashes in filenames. * scripts/pre_remove_exceptions: Handle removal of more packages pulling in 'init'. * Add/update exceptions for ancient packages in scripts/is_testable_uninstallable, scripts/pre_install_exceptions, and scripts/post_distupgrade_exceptions. * master-bin/detect_piuparts_issues: Recognize more cases of full disks. * master-bin/detect_network_issues: Recognize unauthenticated packages. * Add known_problems/problems_and_no_force_issue.conf to recognize non-fatal failures when attempting to remove essential packages. * Add bug template for failing to upgrade from testing to sid. * Add bug template for shipping broken symlinks. [ Holger Levsen ] * piuparts.py: - Apply patch from Maxime Lareo to allow usage of --keep-sources-list without --basetgz. (Closes: #841157) - Add to ignored_patterns: + /srv/.* (Closes: #848186) - Add my copyright note according to git blame. * Add python (>= 2.7) to the depends of piuparts-master and piuparts-slave as dh-python has been changed to only add python depends if it finds python code in PATH. Thanks lintian. * Replace httpredir.debian.org with deb.debian.org everywhere: with deb.d.o apt will use SRV records to go directly to the mirror, with httpredir.d.o there will be http redirects for every file. * slave_run: Fix bug which prevented a different number of slaves to be started than defined in the configuration. Also add a check to ensure that this optional argument is an integer and within the defined maximum number of slaves. * slave_stop: New script to stop all piuparts-slaves on a host. * slave_stop.8: Add new manpage for this new script, also update slave_run.8 to mention slave_stop.8. * Update README_pejacevic.txt and README_server.txt. * Drop instances/piuparts.conf.lamarr and …/piuparts.conf.goldwasser. * htdocs/index.tpl: - Announce 2nd piuparts-slave, explain this is a three host setup now, mention that pejacevic is a four core system and other news. - Make "contribute" a link. * archive_old_logs: use xargs -P $NUM_CPU and -n 1 to compress old logs in parallel. * update-piuparts-(master|slave)-setup: touch $PIUPARTS_TMPDIR/.nobackup to exclude it from backups made by DSA. * conf/crontab-master.in: run piuparts-report twice a day instead of trying four times a day, which hasn't worked out for a long time. * generate_daily_report.in: measure runtime of this script and the scripts started by it. * detect_(network|piuparts|archive)_issues.in: measure runtimes too. * reschedule_oldest_logs.in: measure runtime. [ Michael Stapelberg ] * Add support for logging update-alternatives calls using optional custom-scripts/scripts-log-alternatives/ (Closes: #850917) -- Holger Levsen Sun, 12 Mar 2017 17:55:07 +0100 piuparts (0.75) unstable; urgency=medium [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /etc/network/ + /etc/php/7.0/ and selected subdirs + /var/lib/apache2/ - Run panic handlers in reverse order. - Add --shell-on-error option to simplify debugging of piuparts failures. Starts an interactive shell in the chroot if an error occurred. - Unmount /proc/sys/fs/binfmt_misc only if mounted. - Query the apt version in the chroot. (Closes: #851517) - Use apt_pkg.version_compare(). * piuparts.conf: - Add new suite: stretch-rcmd (test with --install-recommends). - Add new suite: jessie2stretch-rcmd (test with --install-recommends). - Add new suite: stretch-security. - Add new suite: stretch-pu (test packages in stretch-proposed-updates). - Add new suite: stretch2proposed. - Reduce delay before rescheduling untestable packages. * piuparts-analyze.py: - Adjust debianbts syntax to be compatible with both jessie and stretch. - Improve exception handling. * Handle more database setup issues. * Add/adjust some exceptions for ancient packages. * scripts/pre_install_extras: Add examples how to preseed license acceptance for non-free and downloader packages. * scripts/pre_distupgrade_zz_database-server: Run pg_upgradecluster during dist-upgrades. * scripts/post_distupgrade_hack_debsums: Limit to squeeze->wheezy and merge into scripts/post_distupgrade_exceptions. * scripts/{pre,post}_remove_exceptions: Support removal of essential packages with apt-get from stretch onwards. * known_problems/piuparts-depends-dummy_error.conf: Recognize the failure from 'apt-get install piuparts-depends-dummy.deb'. * known_problems/problems_and_no_force_error.conf: Recognize the apt 1.1 message on attempted removal of essential packages. * master-bin/detect_piuparts_issues: Recognize more cases of full disks. -- Holger Levsen Wed, 25 Jan 2017 13:19:30 +0100 piuparts (0.74) unstable; urgency=medium [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /etc/modprobe.d/ - Add new class of is_testable_* custom scripts: A non-zero return value makes the test succeed without actually installing the package. Use for packages that cannot be tested with piuparts by design or are broken in an unfixable way. (Closes: #682739) - Ensure /etc/resolv.conf in the chroot has permissions 0644 to match a possible recreation of that file by resolvconf. (Closes: #660350) - Increase output limit to 8 MB. (Closes: #794331) - Increase single command time limit to 60 minutes. * piuparts.conf: - New section [sid-strict], testing with --install-remove-install and failing on leftover files. (Closes: #604406) - Only warn on leftover files for [sid] and use the same settings for both [sid] and [stretch], since these are now used by britney to block testing migration in case of regressions. * piuparts.conf.anbe: Add some more example sections. * piuparts-slave.py: - Increase test timeout to 90 minutes. * piuparts-analyze.py: - Also consider bugs filed against source packages. * piuparts-reports.py: - Mark arch:all packages with an :all suffix in package lists. - Report skipped tests. * Makefile: Add check-whitespace target looking for trailing whitespace etc. * scripts/pre_install_database-server: Use default-mysql-* from stretch onwards. * scripts/post-{install,purge}-exceptions: Wait for dkms progress indicator to terminate. * scripts/post_distupgrade_exceptions: Work around some jessie->stretch issues. * scripts/is_testable_uninstallable: New custom script to exclude some uninstallable (or similarly broken) packages from being tested. * scripts/post_setup_minimize: New custom script to minimize chroots. - Remove tzdata and lsb-base from sid and stretch. * scripts/post_setup_dummy_users: New custom script to create dummy groups and users that block ids that are likely being used by the host system. * scripts/post_{setup,distupgrade}_experimental: Removed, obsolete since restoring the initial package selection supports downgrades (0.64). * custom-scripts/scripts-debug-{remove,purge}/: Improved debugging of maintainer scripts during package removal. * Overhaul some more custom scripts. * known_problems/debsums_mismatch_*.conf: Report missing symlinks and directories. * known_problems/needs_rebuild_issue.conf: Report "called emacs-package-install as a new-style add-on, but has no compat file" * known_problems/test_was_skipped_issue.conf: New known problem for skipped tests. * master-bin/detect_network_issues: Detect "E: The repository '.*' does no longer have a Release file." * master-bin/detect_piuparts_issues: Detect 'invoke-rc.d: initscript mysql, action "restart" failed.' * master-bin/reschedule_piuparts_tests: New experimental helper script for rescheduling piuparts tests based on detected known problems. [ Holger Levsen ] * Apply patch by Antonio Ospite to workaround #847788 by force-unmounting /proc/sys/fs/binfmt_misc - some java packages trigger update-binfmts, which mounts /proc/sys/fs/binfmt_misc but never unmounts it. (Closes: #704037) - Thanks Antonio! * Add build-depends to dh-python, thanks lintian. * Bump copyright years. [ Sean Whitton ] * piuparts.py: use `apt-get install ./foo.deb` instead of `dpkg -i ./foo.deb && apt-get install -yf` for more reliable dependency resolution. (Closes: #825487) -- Holger Levsen Sun, 15 Jan 2017 11:59:00 +0100 piuparts (0.73) unstable; urgency=medium * Add new option, --hard-link, and do *not* use it by default. (Closes: #841025) -- Holger Levsen Tue, 06 Dec 2016 18:14:19 +0100 piuparts (0.72) unstable; urgency=medium [ Holger Levsen ] * instances/piuparts.conf.pejacevic: test jessie2bpo and jessie2bpo2stretch. * piuparts.py: - Fix devpts mounting to use the correct gid=5,mode=620 mount options. Patch taken from Adam Conrad. (Closes: #829146) - explain (as a code comment) why /root/.rnd shouldn't be ignored. (Closes: #750099) - Add /var/log/apt/eipp.log.xz to ignored files. (Closes: #830527) * custom-scripts/scripts/pre_remove_exceptions: force removal of usrmerge package. (Closes: #798374) [ Stefano Rivera ] * Don't test the current Debian release status, tracking that is distro-info-data's problem. (Closes: #827411) -- Holger Levsen Wed, 03 Aug 2016 11:24:56 +0200 piuparts (0.71) unstable; urgency=medium [ Andreas Beckmann ] * piuparts.py: - Define namedtuple FileInfo globally s.t. it can be pickled. - New option --no-check-valid-until to set apt option Acquire::Check-Valid-Until=false for testing archived releases. - Clear all locale settings from the environment before running commands in the chroot. * piuparts.conf: - Use --no-check-valid-until for squeeze. - New section [wheezy-security] (also used by wheezy LTS). - New section [jessie-security]. * distros.conf: squeeze-* has been archived/removed. * piuparts-analyze.py: - Skip logfile on HTTPError while querying the BTS. * piuparts-reports.py: - Skip section on HTTPError while downloading Packages. * post_setup_forbid_home: Adjust for basefiles now shipping /home/ * Add/adjust some exceptions. [ Holger Levsen ] * Bump standards version to 3.9.8, no changes needed. -- Holger Levsen Thu, 05 May 2016 14:03:30 +0200 piuparts (0.70) unstable; urgency=medium [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /etc/sysctl.d/99-sysctl.conf (dangling symlink to ../sysctl.conf) + /var/log/btmp - Canonicalize mountpoint inside the chroot to avoid mounting over symlinks and possibly outside the chroot. - --install-remove-install runs 'apt-get install --reinstall $pkg', too. * post_distupgrade_base_cleanup: Remove libprocps3 in stretch. * pre_remove_40_find_unowned_lib_links: Handle another mpi alternative. * Add bug template for maintainer scripts incorrectly complaining about being called with unknown arguments. [ Holger Levsen ] * conf/distros.conf and instances/piuparts.conf.pejacevic: move squeeze to archive.debian.org, stop testing squeeze only suites - but keep testing upgrades to wheezy suites. * Fix typo in piuparts_slave_join.8.txt, thanks lintian. -- Holger Levsen Tue, 08 Mar 2016 13:00:22 +0100 piuparts (0.69) unstable; urgency=medium * piuparts.py: add to ignored_files: - /var/lib/pkcs11proxyd/ (Closes: #810703) * piuparts.1.txt: Fix typos, thanks to Дилян Палаузов and lintian. * debian/control: - Bump standards version to 3.9.7, no changes needed. - Use more secure https URL for Vcs-Git: header, thanks lintian. - Use /git/ not /cgit/ in Vcs-* headers. * debian/copyright: bump years for my contributions to 2016. -- Holger Levsen Sat, 13 Feb 2016 12:46:10 +0100 piuparts (0.68) unstable; urgency=medium * Dedicated to the memory of Ian Murdock. Thank you very much for starting and shaping Debian, Ian! The world would be very different today without your work and you will never be forgotten. [ Andreas Beckmann ] * piuparts.py: - Simplify mounting/unmounting into the chroot. - Mount a 64 MB tmpfs onto /dev/shm in the chroot. (Closes: #793487) [ Sven Haardiek ] * Add option "--single-packages" to test every package specified via path or package name individually. (Closes: #810237) [ Michael Prokop ] * piuparts.py: - Execute dist-upgrade also when using --existing-chroot option. (Closes: #798266) -- Holger Levsen Tue, 02 Feb 2016 10:33:40 +0100 piuparts (0.67) unstable; urgency=medium [ Holger Levsen ] * Add wheezy-pu to the suites being tested. (Closes: #800094) [ Andreas Beckmann ] * piuparts.py: - Ensure /etc/mtab exists in the chroot. - Record and compare file owner and group names instead of uid/gid. * Add support for adequate tag ldd-failure. (Closes: #793108) * scripts/post_distupgrade_exceptions: - Fix the /etc/nsswitch.conf wheezy -> jessie upgrade handling. * Update custom scripts handling databases. * Add bug template for wheezy -> jessie -> stretch upgrades. -- Holger Levsen Sat, 17 Oct 2015 16:32:06 +0200 piuparts (0.66) unstable; urgency=medium * Add python-lzma to build-depends, as the tests need it now. -- Holger Levsen Wed, 26 Aug 2015 00:59:42 +0200 piuparts (0.65) unstable; urgency=medium [ Andreas Beckmann ] * piuparts.py: - Create sources.list entries for --testdebs-repo with '[ trusted=yes ]' to avoid needing --do-not-verify-signatures which would act globally. - Add --fake-essential-packages option to include packages in the base chroot that are not removed after the test. These are available during purge and for checking against mistreatment. - Increase permitted command output volume from 3 MB to 4 MB (for daptup). - Only ignore 'dpkg -i' failures that can be fixed by 'apt-get -f install'. (Closes: #794628) - Set Architecture of piuparts-depends-dummy to match the package to be tested. * piuparts.conf.anbe: Add some more example sections. * piupartslib/packagesdb.py: - Put package in 'dependency-does-not-exist' state if a pass/ log exists but a dependency has been removed from the archive since then. - Prepare improvements for handling foreign arch dependencies. * piupartslib/open_packages_url(): Support xz-compressed and uncompressed Packages files. * piuparts-master-backend.py: - Catch URLError, log it and abort without backtrace. * piuparts-common: Add Depends: python-lzma. * scripts/pre_remove_40_find_unowned_lib_links: Ignore some symlinks caused by mpi-default-dev and libglide2/libglide3. * Add/adjust some exceptions for ancient packages. -- Holger Levsen Tue, 25 Aug 2015 08:23:51 +0000 piuparts (0.64) unstable; urgency=medium [ Holger Levsen ] * Add FancyIndexing to piuparts-master.conf and enable apache syntax highlighting in vim. (Thanks DSA!) * Use httpredir.debian.org instead of http.debian.net. (Closes: #783617) * Add support for adequate tag missing-pkgconfig-dependency. (Closes: #783528) * Use revdate: in the asciidoc headers of manpages instead of DATE in the body to achieve reproducible builds and set the revdate: to the last modification date of each manpage. [ Andreas Beckmann ] * piuparts.py: - Use 'dpkg-query -W' instead of 'dpkg --get-selections' to record architecture and version information for installed packages, too. - Restore the selections to the package versions from the reference chroot in scenarios like sid+experimental where more than one version could be available. Use --force-yes since this may involve downgrades. * piuparts.conf: - New global setting: basetgz-sections, used by piuparts-slave only. - Switch to separate tarball sections. * piuparts-slave.py: - Add support for special sections that only create/refresh reference basetgz tarballs without testing packages. The basetgz can be shared between multiple sections without being affected by their flags. -- Holger Levsen Fri, 12 Jun 2015 13:42:52 +0200 piuparts (0.63) unstable; urgency=medium [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /etc/clamav/ + /etc/lighttpd/ + /etc/lighttpd/conf-available/ - Increase verbosity when reporting things installed over directory symlinks: report all symlinks and their targets. - Reinstall base system packages missing upon removal with --no-install-recommends. - Mount /dev/pts into the chroot. Wanted by recent apt. - Fix usage of schroot with chroot names without ':'. (Closes: #773874) - Resolve distro aliases before setting PIUPARTS_DISTRIBUTION. - Fix running debootstrap if --do-not-verify-signatures is given. (Closes: #781673) * piuparts.conf: - Enable --warn-on-debsums-errors for all jessie tests. - Add new suite: jessie-pu (test only packages in jessie-proposed-updates). - Add new suite: jessie-rcmd (test with --install-recommends). - Add new suite: wheezy2jessie-rcmd (test with --install-recommends). - Add new suite: stretch. - Add new suite: jessie2stretch. * piuparts.conf.anbe: Add many more example sections. * distros.conf: Reactivate 'testing' and 'stable' aliases. * piupartslib/packagesdb.py: - Consider logfile age in addition to the time waiting for recycling. * piuparts-slave.py: - Handle Ctrl-C while downloading Packages files and flush/unreserve, too. - Catch another exception that may happen while talking to the master. * detect_archive_issues: Move archive_issues.txt from htdocs/ to master/. * detect_piuparts_issues: Detect transient apt-cache failures. * generate_daily_report: Remove obsolete .html files (e.g. summaries for removed source packages or obsolete maintainer addresses) after 30 days. * reschedule_oldest_logs: - Fix gradual oldest-first recycling if nearly all logs are candidates. - Recycle/expire pass/ logs only if the section is (recycle-)idle. * scripts-apt-first/pre_distupgrade_zz_apt_first: New custom script for upgrading apt first and having the new apt perform the remaining upgrade. * scripts-sysvinit/pre_distupgrade_zz_sysvinit: New custom script for doing wheezy->jessie upgrade tests while keeping sysvinit as init if possible. * scripts/pre_remove_40_find_unowned_lib_links: Ignore /lib/ld-lsb.so.[123] links created by lsb-core. * scripts-no-usr-share-doc: Add exception for localepurge. * post_distupgrade_base_cleanup: New custom script for generalized libdbX.Y exceptions. * Update exceptions for squeeze-lts. * Add/update exceptions for many packages. * Add/update some bug templates. * known_problems/{initdscript_lsb_header_issue,insserv_error}.conf: Report "service.*already provided", too. [ Holger Levsen ] * piuparts.conf.pejacevic: add new suite: jessie2proposed. * piuparts.conf.piu-slave-1und1-01: merge changes from pejacevic config but replace amd64 with i386. * crontab-master.in: update webpages every six hours. * piuparts-reports.py: - Link to the new "Debian Package Tracker" (tracker.debian.org) instead to the old "Package Tracker System" (PTS). - Switch links to lintian.debian.org to https. * crontab-slave.in: try to start slave every hour to make sure it's always running. * Use "autopep8 --max-line-length=160" to re-format all .py files, then also use "-a" on a 2nd run. * test_config.py: update for Jessie being stable and Stretch being the new testing suite. * debian/copyright: Remove space from W3C FreeUse licence name, thanks lintian. * Update piuparts.py, piuparts.1.txt and README_server.txt due to the fact that jessie is the new stable and testing is stretch now. -- Holger Levsen Mon, 27 Apr 2015 16:46:29 +0200 piuparts (0.62) unstable; urgency=medium * Default to http://http.debian.net/debian instead of the deprecated http://cdn.debian.net service as default mirror. (Closes: #774471) -- Holger Levsen Tue, 06 Jan 2015 10:36:39 +0100 piuparts (0.61) unstable; urgency=medium [ Michael Prokop ] * Support new scripts directive 'post_chroot_unpack', useful for adding apt-keys for custom apt repositories. (Closes: #767485) * Add custom-scripts/scripts-unused-examples/post_chroot_unpack_key_setup. [ Holger Levsen ] * piuparts.py: Add "/var/cache/apt/archives/partial/" to ignored_files. (Closes: #767489) * piuparts.py and piuparts-report: Detect four more (in)adequate issues: - broken-binfmt-detector - broken-binfmt-interpreter - missing-alternative - program-name-collision (Closes: #767499) * distros.conf: Remove unneeded URI for squeeze-lts, it just uses the default repository. * instances/pejacevic+slaves: Do not reschedule passed squeeze logs anymore. [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /etc/modules-load.d/modules.conf (dangling symlink to ../modules) - Place an up-to-date copy of the host's /etc/resolv.conf (with comments stripped and whitespace normalized) in the chroot. * piuparts.conf: - Add flags-start-jessie, flags-end-jessie global variables. - Use --scriptsdir .../scripts-jessie for tests ending in jessie. - Enable --warn-on-debsums-errors for wheezy2jessie upgrades. * piuparts.conf.anbe: Add many more example sections. * distros.conf: - Add *updates, *backports, *proposed for jessie. - Switch 'testing' alias to 'stretch', but deactivate it for stable. - Add 'stable' alias pointing to 'jessie', deactivated as well. * piupartslib/dwke.py: - Fix parsing of known_problems/*.conf that have EXCLUDE_PATTERN. * piuparts-slave.py: - Avoid a busy loop by suspending a section for 5 minutes if basetgz creation is in progress or failed. * Add "AddDefaultCharset utf-8" to the apache2 configuration snipplet example, so that logfiles are displayed as UTF-8. * master-bin/detect_piuparts_issues: Another partial write error from tar. * pre_install_foreign_architecture: New script to enable foreign architectures for testing cross toolchain packages. * pre_remove_40_find_unowned_lib_links, unowned_lib_symlink_*.conf: New script and reports for unowned symlinks in [/usr]/lib[/]. * post_distupgrade_exceptions: Handle libdb5.1 removal in jessie. * scripts-jessie/post_setup_jessie-fake-essential: New custom script. - Make ucf fake-essential. (ucf may be used unconditionally during purge.) - Make init-system-helpers fake-essential. (Systemd service link cleanup.) * Rework the file-rc/sysvinit/upstart/systemd specific exceptions. * Remove debugging output for resolvconf. * Update custom scripts handling databases, experimental, ... and some more. * Update bug templates. -- Holger Levsen Wed, 03 Dec 2014 19:55:26 +0100 piuparts (0.60) unstable; urgency=medium [ Leo Iannacone ] * piuparts.py: when using the --existing-chroot option, instead of copying files, use hard links if the original chroot and destination directory are on the same device. (Closes: #754878) [ Iain Lane ] * Use python-distro-info to get the Debian and Ubuntu devel releases. * tests: Remove the Ubuntu devel release test. [ Holger Levsen ] * Update README_server.txt and README_pejacevic.txt after setting up a piuparts master/slave system from packages myself. * Drop conf/piuparts-master.sudoers from debian/piuparts-master.examples and include all commands piupartss needs to run as root in conf/piuparts-slave.sudoers. * Update instances/piuparts.conf* to match what is in use for piuparts.debian.org today. Add two new systems, goldwasser and lamarr, as examples taken from piuparts development. * Add "AddType text/plain .log" to the apache2 configuration snipplet example, so that logfiles are displayed in the browser. * Bump standards version to 3.9.6, no changes needed. * Replace all occurrences of old gitweb Alioth URLs with cgit ones. -- Holger Levsen Sun, 26 Oct 2014 20:07:24 +0100 piuparts (0.59) unstable; urgency=low [ Holger Levsen ] * piuparts.py: - mount selinuxfs read-only and on new location when possible. Thanks to Laurent Bigonville for the patch! (Closes: #682068) - Add to ignored_files: + /etc/sub{g,u}id[-] introduced by shadow 1:4.2-1. * piuparts-report.py: - create https URLs in source_summary_url(). This should probably be made configurable. - fix log directory creation. (Closes: #746722) - fix spelling of "ditto". (Closes: #746724) - create graph of rc and non-rc bugs filed (and fixed) since the beginning of piuparts testing and display it on piuparts.d.o welcome page. - move counts.txt from htdocs to master directory. * master-bin/gather_bts_stats: differate between rc and non-rc bugs. * Makefile: - Don't install crontabs and sudoers file anymore, except when installing from git. - Also cleanup python bytecode from tests. * Packaging: - install cron files for piupartsm and piupartss users in /usr/share/doc/piuparts-(master|slave)/examples/ (Closes: #746721) - split sudoers.d into master and slave and provide them in /examples/ too. - install piuparts.conf.sample in /usr/share/doc/piuparts-master/examples/ - install slave_run and slave_join in /usr/bin/ (Closes: #746718) - piuparts-slave.postinst: do not attempt to add the slave user's ssh public key to the master user's authorized key. The slave package should maybe recommend the master package, but definitly not pre-depend on it, which would be a requirement to make this work. So document this instead. * Updates various pieces of documentation, rename README_1st to README. Explicitly state what will break when using cdebootstrap with pbuilder. (Closes: #709348) * Update various custom-scripts (usually to workaround problems in other packages). * Switch all links to bugs.debian.org, www.debian.org, wiki.debian.org, packages.debian.org, anonscm.debian.org and lists.debian.org, db.debian.org, qa.debian.org, lists.alioth.debian.org and udd.debian.org to https. [ David Steele ] * dependencyparser.py - Be more lenient about the multi-arch modifiers accepted and stripped by the parser. * piuparts-report.py: - Fix indentation of generate_global_summary() - Refactor sections_by_precedence() out of generate_global_summary(). - Restore (a copy of) the counts.txt file to the web view, in
/counts.txt. - Also add a link to counts.txt to the section page. * Remove oldstable distribution results from the JSON summary. - Also remove stable distribution results. -- Holger Levsen Mon, 14 Jul 2014 13:14:19 +0000 piuparts (0.58) unstable; urgency=low [ Holger Levsen ] * debian/copyright: update years. * piuparts.py: - ignore multiarch definition in_default_debian_mirrors() (Closes: #711370) - Add to ignored_files: + /etc/machine-id (Closes: #619244) * custom-scripts: add debugging output for resolvconf. (See #660350) * Adapt and rename conf/piuparts.apache to piuparts-master.conf as per apache2.4 transition. (Closes: #712384) * piuparts.1.txt: explain that --do-not-verify-signatures also causes APT::Get::AllowUnauthenticated to be set accordingly in the chroots. * Switch ubuntu default to 'trusty'. (Closes: #744848). Then switch it to 'utopic'... * piuparts-report: Add link to ci.debian.net to navigation menu. * Replace http://piuparts.debian.org with https everywhere. Thanks to DSA for all their support running piuparts.debian.org and for everything else! [ Andreas Beckmann ] * piupartslib/packagesdb.py: - Add missing PackagesDB.get_version(package_name) method, needed by piuparts-report. - Add PackagesDB.get_source(package_name) method, for easy lookup of source package names. * piuparts-report.py: - Improve integration of known problem reports generation. * master-bin/detect_well_known_errors.py: - Classify unclassified failures as unclassified_failures.conf. [ David Steele ] * piuparts-report.py: Store test summary.json output at the section and global level, using a new piupartslib/pksummary.py module. The new Config.get_std_distro() provides a standard default distro name for summaries. (Closes: #740386) * Add python-distro-info to piuparts-common's depends and to build-depends as it's used in the tests too. -- Holger Levsen Thu, 01 May 2014 00:23:42 +0200 piuparts (0.57) unstable; urgency=low [ Ivo De Decker ] * piuparts-report.py: Use source version from final_distro. (Closes: #733900) [ Matthias Schmitz ] * Rearrange old unittests.py, use nosetests instead: - Move unittests.py to tests/ subdirectory, - Split out dependencyparser test to an extra file, - Add configuration for nosetests, - Use nosetests in Makefile target "check". * piuparts.py: - Add unittest for DefaultsFactory / Defaults. - Use pymox to mock the piuparts.guess_flavor() function as it would run lsb_release in a subprocess. - Move broken_symlink unittest out of piuparts.py into tests/test_piuparts.py so nosetests can run this test and detect coverage. * Add tests/README documenting the naming of the test classes and functions. * Re-enable unittests during package build - Don't create coverage report during pkg build. - Add to Build-Depends: python-debian, python-apt, python-nose, python-debianbts, python-yaml, and python-mox3. [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /etc/group.org + /etc/passwd.org + /etc/shadow.org - Create piuparts-depends-dummy.deb with gzip compression for installability with older dpkg versions. - Exclude virtual packages provided by the real package (but not the dummy package) from the Conflicts of piuparts-depends-dummy.deb. - Check the status of the packages after installing the supplied .debs. Based on an initial patch by Julien Cristau. (Closes: #718544, #736743) * piupartslib/packagesdb.py: - load_alternate_versions_from_packages_urls(): New method to load alternate version numbers from a different set of URLs (e.g. final_distro). - get_best_package_state(): New method that considers successfully tested providers of a virtual package if the real package of the same name failed. Used for dependency resolution. - No longer special-case packages as essential-required and test them like normal packages (no-op test plus adequate run). (Closes: #735907) - Improve ordering of packages to be tested/recycled. * piupartslib/dwke.py: - Factored out classes Problem and FailureManager and some helpers from detect_well_known_errors.py * piuparts-master-backend.py: - Improve master.log verbosity. * piuparts-report.py: - Mark essential-required as obsolete and hide it. - Integrate the reporting part from detect_well_known_errors.py. - Archive outdated logs even if the current version is untested. * master-bin/detect_well_known_errors.py: - Only update .kpr files, the reporting part was integrated into piuparts-report. * scripts/post_{setup,distupgrade}_force-unsafe-io: New custom script to automatically enable dpkg --force-unsafe-io once that is supported. Also divert /bin/sync and replace it with a link to /bin/true. Syncing a throwaway chroot is useless, but affects on the host's performance. * scripts/pre_install_extras: New custom script to install a linux-headers package before testing a *-dkms package, s.t. dkms tries to build a kernel module. * known_problems/module_build_error_issue.conf: New known problem for dkms module build failures. * known_problems/piuparts-depends-dummy_{error,issue}.conf: New known problem for failing to install and fixup piuparts-depends-dummy.deb. * Add more piuparts exceptions. * Handle lib32nss-mdns like ia32-libs[-gtk]. [ Holger Levsen ] * piuparts.py: - Report adequate version used. -- Holger Levsen Fri, 14 Feb 2014 13:37:23 +0100 piuparts (0.56) unstable; urgency=low [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /etc/apt/apt.conf.d/01autoremove-kernels - Use temporary files while creating basetgz and rename afterwards. - Remove the package to be tested separately before removing its Depends and Recommends. - --allow-database now also enables firebird2.5-super. * piuparts.conf: - New per-section settings: + components * piupartslib/__init__.py: - open_packages_url(): Try Packages.gz if Packages.bz2 is not available. (Closes: #711157) - Add new class DecompressedStream. - open_packages_url(): Use DecompressedStream to transparently decompress a Packages.{bz2,gz} file while downloading and reading the content without requiring to store the complete file (compressed and/or decompressed) in memory. * piupartslib/packagesdb.py: - PackagesFile: Allow restricting to a set of package names. * piuparts-slave.py: - Use locking to prevent concurrent basetgz creation. (Closes: #708287) - Use the "components" setting to restrict the archive areas being used in the generated sources.list. - Reduce memory footprint by discarding unneeded Packages file parts. * piuparts-report.py: - Report URLs of all Packages files used for a section. - Avoid reporting duplicate dependencies after stripping versioning. - Fix some invalid HTML constructs. - Sort maintainer summary by state: failing packages on top. * Improve exception handling. * known_problems: Upgrade adequate issue 'missing-copyright-file' from boring to normal and make pre_remove_50_find_missing_copyright no longer generate fatal errors. * Add support for new adequate tags missing-symbol-version-information and symbol-size-mismatch. * reschedule_oldest_logs: Report number of logfiles in recycle/. * Add more piuparts exceptions to work around various failures in multiple packages. Add a known_problem report for packages using exceptions. * bug-templates: Recommend to use the new dpkg-maintscript-helper commands dir_to_symlink/symlink_to_dir that were added in dpkg 1.17.2. (Closes: #720185) [ Holger Levsen ] * piuparts-report.py: use descriptive link targets for (in)adequate issues instead of mere tag names, thanks to Jakub Wilk for the wordings. * Bump standards version to 3.9.5, no changes needed. * Bump compat level to 9, thus build-depend on debhelper > 9.20120909~. -- Holger Levsen Mon, 16 Dec 2013 12:59:14 +0100 piuparts (0.55) unstable; urgency=low [ Ivo De Decker ] * piuparts-report.py: Add sources.yaml and section.yaml exports. [ Holger Levsen ] * Add file CONTRIBUTING, thanks to Vincent McIntyre for it! (Closes: #721348) * piuparts-report: Add "contact us" link. * piuparts.py: Explain at the very beginning of the generated log that contact information is part of the FAQ. [ Andreas Beckmann ] * Add bug templates for wheezy. * scripts-leftovers: Improve cleanup in /var/lib/systemd with fake-essential init-system-helpers. (Closes: #719869) -- Holger Levsen Thu, 03 Oct 2013 15:31:38 +0200 piuparts (0.54) unstable; urgency=low [ Holger Levsen ] * known_problems: fix some adequate patterns. * piuparts-master: recommend apache2 | httpd. * README_server.txt: be more verbose about configuring sudo. (Closes: #711155) * piuparts.py and piuparts-reports.py: add "incompatible-licenses" (detected since adequate 0.7) and "ldd" to inadequate_tags and known_problems. [ David Steele ] * piupartslib/packagesdb.py - The dependency calculation for rdeps is reworked to match that used elsewhere, to allow package reserve() to drive waiting-for-dep to drive to 0 more quickly. (Closes: #705977) - Reworked rdep metrics (block_count, rrdep_count, waiting_count) to calculate on-demand. Access methods are moved from Package to PackagesDB. - Add a metric for the worst-case reverse-dependency chain length. - Weight the reserved packages returned by this additional factor. [ Andreas Beckmann ] * piuparts.py: add to ignored_files: /etc/hosts -- Holger Levsen Tue, 30 Jul 2013 11:05:32 +0200 piuparts (0.53) unstable; urgency=low [ Andreas Beckmann ] * *.py: Reindent and reformat, slowly moving towards PEP 8 style. * piuparts.py: - Do not run adequate checks on possibly disappearing packages. - Add --allow-database option that enables starting MySQL and PostgreSQL database servers in the chroot. (Closes: #662633) - Relax the check for running processes if --allow-database is given. * distros.conf: - Add "candidates" key to merge multiple Packages files into one virtual distro. - Define wheezy-proposed and squeeze-proposed. * piupartslib/conf.py: - Add get_{packages,sources}_urls() that resolve "candidates". * piupartslib/packagesdb.py: - Add load_packages_urls() methods. * piuparts-{master-backend,slave,report}.py: - Switch to load_*_urls(get_*_urls()) for loading Packages/Sources files, adding support for arbitrarily composed virtual distros. * Makefile: - Stop installing python modules in $libdir/python2.6/dist-packages. * debian/control: - Bump Build-Depends: python (>= 2.7). - piuparts-slave: Add Depends: sudo. (Closes: #710350) * New known problem: Packages with missing or incomplete .md5sums files. * piuparts-master.deb: Ship some piuparts.conf examples. * detect_piuparts_issues: Detect problems starting MySQL or PostgreSQL. * pre_install_database-server: New script to install a database server for packages that require one during postinst but only recommend or suggest (or maybe not even this) the server package to allow for remote db servers. * pre_distupgrade_zz_database-server: New script to upgrade (running) database servers before all other packages. This ensures the database server is running again before the dependent packages (that may declare 'insufficient' dependencies on a 'local' database server) are upgraded. [ David Steele ] * Update python Requires to 2.7 (for argparse) * Edit piuparts_slave_* man pages for more information and format. * Specify the manpage doctype for AsciiDoc man sources. [ Holger Levsen ] * piuparts.1.txt, README.txt and README_pejacevic.txt: proof-read and cleaned up. * Move master-slave related documentation from README.txt to README_server.txt and leave a pointer in the former. * Rename README.txt to README_1st.txt. * piuparts-master.deb: - Ship README_server.txt. - Recommend devscripts and libsoap-lite-perl (for gather_bts_stats). * piuparts-report.py: Add a link to README_server.html. * master-bin/gather_bts_stats: write bts_stats.txt to master-directory. * conf/crontab-master: run gather_bts_stats daily. * master-bin/prepare_backup: backup bts_stats.txt and do not backup sources.txt. * piuparts.py: categorize adequate results using inadequate_tags, boring_tags, ignored_tags and unknown. Show adequate exit code if non-zero. * Add known_problems/*inadequate*.conf for all adequate tags we care about. * htdocs/index.tpl: Improve description about piuparts.debian.org. (Closes: #700848) [ Andrew Starr-Bochicchio ] * piuparts.py: - Move default keyring to distro defaults. (Closes: #710346) - When distro defaults are explicitly specified, use the distro default mirrors (Closes: #710347). -- Holger Levsen Sun, 09 Jun 2013 12:57:52 +0200 piuparts (0.52) unstable; urgency=low [ Andreas Beckmann ] * piuparts.py: - Really disable PDiffs. (Closes: #708241) * piuparts-master.in: - Use timeout to prevent master from getting stuck. (Closes: #701618) * piuparts-slave.py: - Report upgrade tests involving downgrades as untestable. - Always pass --arch to piuparts. * piuparts-analyze.py: - Consider archived bugs, too. (Closes: #669966) * piuparts-report.py: - Switch from python-rpy to python-rpy2. (Closes: #705975) - rpy2: Use a noninteractive grDevice. * Makefile: - Switch default target from 'install' to 'build'. * scripts-no-usr-share-doc/post_setup_disable_usr_share_doc: New custom script to instruct dpkg to skip installing most files in /usr/share/doc. This catches maintainer scripts relying on /usr/share/doc content. * New bug template: postinst_uses_usr_share_doc. * piuparts-master.deb: - Switch Depends: python-rpy to Depends: python-rpy2. [ Holger Levsen ] * piuparts.py: - Run adequate (if installed) after installation test. New options --no-adequate and --fail-if-inadequate. (Closes: #703902) - Workaround #709372 and ignore broken symlinks found by adequate. - Do not print a "Guessed: debian" line on every run. It's the default anyway. (Closes: #703058) * piuparts-report: - Add templates for known_problems/inadequate_issue and inadequate_error. - Add link to the piuparts source at the top of the navigation menu on the left. - Add links to Debian QA group and jenkins.debian.net at the bottom of that navigation menu. * debian/control: Add adequate to piuparts' recommends. * piuparts-slave.postrm: Also remove /var/cache/piuparts/tmp on purge. * conf/crontab-master: Run generate_daily_report twice a day. * piuparts-slave.deb: Add piuparts_slave_run.8 and piuparts_slave_join.8 manpages. -- Holger Levsen Mon, 27 May 2013 18:10:43 +0200 piuparts (0.51) unstable; urgency=low [ Andreas Beckmann ] * Mark all python code as utf-8 encoded. * piuparts.py: - In install-purge tests support installing extra packages before the package to be tested via --extra-old-packages. - Add --arch option to run the test for the requested instead of the default architecture. - Disable using PDiffs in order to save CPU instead of bandwidth to a local mirror/proxy. (Closes: #708241) * piuparts.conf: - New global settings: + slave-count - Use installation-dependent defaults for known-problem-directory. * distros.conf: - Update backports setup after integration into the regular archive. - Enable stable security updates when installing stable backports. - Set testing = jessie and add jessie-proposed-updates. * piupartslib/conf.py: - Raise MissingSection if the requested section is not in piuparts.conf. - Improve handling of virtual distros. * piupartslib/packagesdb.py: - get_package(): Add flag resolve_virtual, disabled by default. * Rename piuparts-master.py to piuparts-master-backend.py * piuparts-master: New. - Wrapper shell script to simplify calling the master from the slave. - Get the 'master-directory' from piuparts.conf and chdir there. - Get the (global) 'log-file' from piuparts.conf and handle the error logging. No longer uses the section specific logfile name for errors. * Allow the master-command to be set in ~piupartsm/.ssh/authorized_keys to restrict ssh keys to only run piuparts-master. * piuparts-master-backend.py: - Raise CommandSyntaxError on unknown commands. - Add "section" command: + Make section selection part of the master-slave protocol. + Allow switching sections without reconnecting. + No longer accept section argument on the command line. + Switch the logfile after switching sections. * piuparts-slave.py: - Fix parsing 'idle' return value from master. - Let the piuparts-master (wrapper script) handle chdir and error logging. - Skip sections that don't exist in piuparts.conf and continue to run. - Pass the section to piuparts-master via the new "section" command instead of using a command line argument. - Keep the connection to the master open while switching sections. This reduces the number of ssh connections attempted while checking idle sections. Close the connection before processing packages, before going to sleep, and after communication errors. - Support empty master-command if the command is set in master's .ssh/authorized_keys file. - Change default basetgz name to {DISTRO}_{ARCH}.tar.gz. * piuparts-report.py: - Skip sections that don't exist in piuparts.conf. - "Missing" binary packages don't influence the overall test result of a source package. These are either udebs or for other architectures. (Closes: #699160) - Report the correct version for binary packages. * archive_old_logs: - Compress the archived logs. * detect_well_known_errors: - Skip sections that don't exist in piuparts.conf. * piuparts-master.deb: - Add Depends: xz-utils. * piuparts-slave.deb: - Restrict the ssh key added to master's authorized_keys to only run piuparts-master. * slave_run: Support starting multiple concurrent slaves. (Closes: #678725) * Update and add new exceptions for buggy packages. * scripts-wheezy/post_setup_wheezy-fake-essential: New custom script to suppress some purge failures in wheezy. * post_{setup,distupgrade}_squeeze-backports: New custom scripts to improve support for squeeze-backports. * New known problem: Database setup failure (issue). [ David Steele ] * detect_well_known_errors: - Replace the bash script with an equivalent Python script. - Sort known error and issue packages by reverse dependency count, separating 'fail' from everything else. (Closes: #698526) - Add a PTS link to issue and error summary entries. - Replace the known_problem COMMAND with PATTERN and EXCLUDE_PATTERN, and replace grep shell calls with python re modules calls, for a 10x speedup. - Validate field names, and only use valid known problem conf files. - Minor HTML syntax fix. - Minor template integration. [ Holger Levsen ] * piuparts.py: - Add to ignored_files: + /var/log/fontconfig.log (Closes: #703810) - Thanks to Hideki Yamane. - switch default Ubuntu distribution to "saucy". * Bump Standards-Version to 3.9.4. * Use /srv/piuparts.debian.org/ on piatti.debian.org as per DSA policy. * Provide instance configuration for pejacevic.d.o and piu-slave-bm-a.d.o. * Move README_piatti.txt and README_pejacevic.txt and update accordingly. * Provide two new helper scripts for running from git: update-piuparts-master-setup and update-piuparts-slave-setup, drop update-piuparts-setup. * Drop instances/forward.* files. * slave-bin/slave_run: - Run slave_cleanup before starting the slave. * slave-bin/detect_slave_problems: remove check limiting this script to be run on piatti only. * master-bin/prepare_backup: also backup sources.txt. -- Holger Levsen Wed, 15 May 2013 11:31:31 +0200 piuparts (0.50) unstable; urgency=low [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files: + /var/lib/dpkg/arch + /usr/lib/python2.[67]/dist-packages/python-support.pth broken symlinks caused by python-support (see #635493 and #385775, thanks David Steele) - Call dpkg-deb with --nocheck to allow bad version numbers. - Pass unqualified package names to apt-cache show for lenny support. - Support plain URLs or local paths as --testdebs-repo argument and prepend/append "deb", "file://", and "./" as needed. - Improve the "dirname part contains a symlink" test and report overwritten files owned by other packages as failures. - Add --proxy=URL option. - piuparts.deb: Add Depends: piuparts-common for using distros.conf and piupartslib. - Use distros.conf to generate mirror configuration. This enables more complex scenarios involving partial distributions and dist-upgrades. - Use apt-get -t if set in distros.conf. * piuparts.conf: - New global settings: + proxy + slave-directory + known-problem-directory (for known_problem definitions) + backup-directory (for the master script prepare_backup) + PYTHONPATH (to find the python modules in non-default path setups) + slave-load-max - New per-section settings: + depends-sections + basetgz-directory - Obsolete settings: upgrade-test-chroot-tgz. - Setting tmpdir is now mandatory for master/slave operation. * distros.conf: - New configuration file to describe full distributions (e.g. sid, squeeze) and partial distributions (e.g. squeeze/security, squeeze-backports, experimental) along with their dependencies (e.g. base distribution) and non-default mirrors. (Closes: #699028) - Allows setting a target-release (for apt-get -t ...). (Closes: #687995) - Shipped in package piuparts-common. * piupartslib/conf.py: - Add methods: get_distros(), get_start_distro(). - Add class DistroConfig for reading distros.conf. * piupartslib/packagesdb.py: - Add Package.waiting_count() and friends, populated in calc_rrdep_count. - Optimize reserve() and skip unavailable candidates. - Add method: set_dependency_databases(). - Lookup missing packages, states, ... in the dependency_databases. * piuparts-master.py: - Sort packages to be tested by importance, i.e. descending waiting_count. - Use piuparts.conf global proxy setting as http_proxy. - Use distros.conf to compute URLs. - Load depends-sections for package lookup in partial distros. * piuparts-slave.py: - Tarball recreation can be disabled by setting max-tgz-age to 0. - Relax package existence checks to allow distupgrading to backports. - Fix SIGHUP handling while recycling. - Flush and unreserve all sections on interrupt. - Add support for running piuparts under /usr/bin/timeout. - Check system load frequently and enter sleep mode if threshold (piuparts.conf global setting slave-load-max) is exceeded. Operation will be resumed after load drops below 'slave-load-max - 1.0'. Disabled by default. - Use piuparts.conf global proxy setting as http_proxy and pass it to piuparts via --proxy. - Use distros.conf to compute URLs. - Use the upgrade-test-distros setting to select between install/purge test (if unset) and dist-upgrade test (if there are at least 2 distros). Drop support for running both tests for a section. * piuparts-analyze.py: - Add support for magic "$DISTRO/None" versions. * piuparts-report.py: - Call r.dev_off() after generating a plot. (Closes: #657799) - Use piuparts.conf global proxy setting as http_proxy. - Use distros.conf to compute URLs. - Generate/prepend/append a default description. - Load depends-sections for package lookup in partial distros. * Makefile: - Add DESTDIR support. - Overhaul. Add dependencies between build and install targets. - Add support for placeholder substitution in scripts and config. - Perform syntax check on *.py before installation. * debian/rules: Set 'prefix' and 'DESTDIR' for build and install. * lib/read_config.sh: Factored out the piuparts.conf reading function that is used by all master/slave shell scripts. Shipped in piuparts.common. - Add support for RFC822 line continuation (wrap + indent) and comments. * master-bin/slave-bin: Switch to sourcing read_config.sh. * conf/crontab*: Substitute @sharedir@ at build time. * known_problems: Install to @sharedir@/piuparts/known_problems/. * Replace all remaining references to $HOME/bin and ~/bin with @sharedir@. * prepare_backup: - Get paths from piuparts.conf. - Prepare a copy of submissions.txt, too. * master-bin/slave-bin: Use PYTHONPATH from piuparts.conf. * piuparts-slave.deb: - Use /var/cache/piuparts/tmp as tmpdir. - Use /var/cache/piuparts/basetgz as basetgz-directory. - Populate ~piupartss/.ssh/known_hosts with localhost's hostkey. - Restrict the ssh key added to master's authorized_keys to only run piuparts-master. * slave_{run,join}: Rename the screen session to "piuparts_slave_screen". * slave_cleanup: Use pgrep to find running piuparts-slave processes. * slave_{run,join}: Use screen -ls to find running screen sessions. * slave_join: Use 'script /dev/null' hack if needed to get a writable tty. This avoids running the piuparts-slave screen session as root. * slave_run: Get paths etc. from piuparts.conf. * slave_run: Ensure the tmpdir exists, this could be on tmpfs. * detect_leftover_processes: Move statefile to slave-directory. * detect_stale_mounts: Merge into detect_tmp_cruft. * detect_tmp_cruft: Move to slave-bin and run from slave crontab. * detect_slave_problems: - Move to slave-bin and run from slave crontab. - Use idle-sleep setting from piuparts.conf. - Move statefile to slave-directory. * master: Get rid of the monitor-slave directory. * pre_install_exceptions: Handle bootcd-ia64 installation in squeeze. * post_distupgrade_exceptions: Handle libdb4.8 removal in wheezy. * pre_remove_50_find_bad_permissions: Add some exceptions. * pre_remove_50_find_missing_copyright: Add some exceptions. * post_{setup,distupgrade}_experimental: New custom scripts to improve support for experimental. * Add several exceptions for lenny and lenny2squeeze tests. * New script to enable Debug::pkgPackageManager during upgrades: scripts-debug-packagemanager/pre_distupgrade_debug_packagemanager * New known problem: Packages that need to be rebuild or updated for newer tools, e.g. due to obsolete install-info or tex-common usage. * New bug template: unhandled_symlink_to_directory_conversion. * New bug template: installs_over_existing_symlink. [ Holger Levsen ] * Mention http://piuparts.debian.org in README.txt prominently. * Mention github clone in htdocs/index.tpl. [ David Steele ] * Replace the Tango icon symlinks with copies. * Make the Tango weather icons required for master. -- Holger Levsen Fri, 15 Mar 2013 15:36:12 -0700 piuparts (0.49) unstable; urgency=low [ Andreas Beckmann ] * Add support for dist-upgrade tests of "disappearing" packages: - master/report: Get candidate packages from distro in the "distro" setting in piuparts.conf and (target) versions to test from last distro entry in "upgrade-test-distros". In case a package no longer exists, use a pseudo-version "None". (Closes: #656438) - slave: Support pseudo-version "None". - piuparts: Do not try to install unavailable packages after dist-upgrade. * piuparts.py: - Run custom scripts only once if dependencies are installed via a metapackage. - Rename --no-debsums option to --warn-on-debsums-errors. - Add --extra-repo= option for adding extra package sources, e.g. backports, security or local repositories. The argument must be a valid line (including whitespace) that is added to sources.list verbatim. The option may be repeated to add more lines. - Qualify to the package names extracted from .debs with the version. - Add --testdebs-repo= option. Provide the package(s) to be tested (and additional dependencies that are not yet in the archive) in a repository to enable dependency resolution with apt-get. Use for complex install and upgrade scenarios. (Closes: #676694) - Add --distupgrade-to-testdebs option. During distupgrade tests override packages from the target distro with the packages from the testdebs repository. This allows doing distupgrade tests before uploading. - Enable upgrade tests if both --apt and --testdebs-repo are given. * piupartslib/conf.py: - Add get_final_distro() method. - Add optional distro argument to get_mirror() method. * piupartslib/packagesdb.py: - Speed up LogDB by fetching existing files with os.listdir(). * piuparts-slave.py: - Disable X forwarding while connecting to master. - Move checks for package existence to test_package(). * piuparts-analyze.py: - Sort bugs and try the newest bug first. * piuparts-report.py: - Fix URLs to piatti's config. - Hardlink the logfiles to the htdocs tree (with copy as fallback). * post_setup_squeeze-fake-essential: Restrict fake-essential install-info to the 4 packages that actually need this. * New custom script: pre_remove_40_find_obsolete_conffiles, report obsolete conffiles after package upgrades. Suggested by Michael Biebl. * pre_remove_50_find_missing_copyright: Skip check for packages that are not installed or have only config files remaining. * post_remove_cleanup: Cleanup dovecot certificates. * Add support for installing and upgrading to the multiarch ia32-libs* in wheezy/sid on amd64. (Closes: #679780) * Merge known problem analyses excessive_output and runtime_exceeded into new known problem resource_violation. * New known problem: Leaving obsolete conffiles after upgrade. * New known problem: Modifying conffiles or other shipped files (issue). * New bug template: partial-upgrade-file-overwrite. [ Holger Levsen ] * README.txt: - reformat with shorter line lengths (Closes: #603873). - fix some typos. -- Holger Levsen Sat, 12 Jan 2013 12:12:45 +0100 piuparts (0.48) unstable; urgency=low [ Andreas Beckmann ] * piuparts.py: - Ignore *.dpkg-* conffile backups found in --scriptsdir directories. - Report attributes (for reference and current file) in case of file modification. - Report paths of installed files that contain symlinks (for better debugging of the problems caused by such a setup). ( test $(dirname $x) != $(readlink -f $(dirname $x)) ) - Add debsums, processes, and broken-symlinks check after the first purge in --install-purge-install mode. - Add --install-recommends option, disabled by default. - Mark temporary directories with a .piuparts.tmpdir file. - Use rm -rf --one-file-system to delete chroots. * piuparts.conf: - New per section settings: expire-old-days, expire-fail-days (defaulting to 0 which means expiration is disabled). - Split and rename piuparts-cmd into piuparts-command ([global] setting that should include all common flags) and piuparts-flags (for additional section-specific flags). - Rename the tempdir setting to tmpdir, this will be used for the --tmpdir argument of piuparts, too. * piuparts-master.py: - Enable recycling before initializing the real package db. - Enable logging by default and set default log-file to $SECTION/master.log. Use "/dev/null" to disable logging. * piuparts-slave.py: - Retry sections with higher precedence more often. - Construct the piuparts command line from the new config options piuparts-command, piuparts-flags, and tmpdir. * piuparts-report.py: - Don't hardcode section names, always show all known_problem reports. * detect_piuparts_issues: Catch the chroot running out of space. * reschedule_oldest_logs: Implement logfile expiration. Delete logfiles that are marked for recycling and are older than expire-{old,fail}-days. These packages have not been retested because of some dependency problem making the package currently untestable. Deleting the log will cause the package to be moved to dependency-failed-testing or similar state. expire-*-days needs to be greater than reschedule-*-days to enable this. * slave_cleanup: Only remove directories containing a .piuparts.tmpdir file. * New bug template: copyright_file_missing_after_upgrade. * New known problem: Installing something over existing symlinks. -- Holger Levsen Fri, 30 Nov 2012 12:12:09 +0100 piuparts (0.47) unstable; urgency=low [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files/ignored_patterns: + /var/log/dbconfig-common/dbc.log - Print a final message before terminating in the panic() handler. - Add support for running debsums before package removal, enabled by default. May be disabled via --no-debsums. (Closes: #687538) - Add debsums as dependency. * piuparts-slave.py: - Detect incomplete piuparts output and ensure such tests fail. * piuparts-report.py: - Rewrite find_files_with_suffix(), handle disappearing files. * New custom script pre_remove_50_find_missing_copyright: check for missing /usr/share/doc/$package/copyright files. (Closes: #683685) * Rename custom script post_install_find_bad_permissions to pre_remove_50_find_bad_permissions to avoid early failures during upgrade tests that may be fixed in a later version. * Remove post_purge_dbconfig_cleanup custom script. * reschedule_oldest_logs: Refactored. - Fix duplicate variable name usage that sometimes prevented recycling. - Support a [global] auto-reschedule=no setting. * detect_well_known_errors: Unclassified logs may still be bugged. * detect_piuparts_issues: Detect incomplete logfiles. * New bug template: modifies_conffiles. * New known problem: Modifying conffiles or other shipped files. * New custom script post_distupgrade_hack_debsums to work around #687611 which makes all debsum checks fail due to modification of /usr/share/keyrings/debian-archive-removed-keys.gpg during upgrades. -- Holger Levsen Fri, 28 Sep 2012 13:24:47 +0200 piuparts (0.46) unstable; urgency=low [ Holger Levsen ] * piuparts-report: drop in-code index.tpl stub. * slave-join: fix bug preventing detection that the slave is not running. * piuparts: call schroot with session: argument, thanks to Stephan Sürken for the patch. (Closes: #686360) [ Andreas Beckmann ] * piuparts.py: - Add to ignored_files/ignored_patterns: + /var/lib/cdebconf/{questions,templates}.dat-old + /var/lib/onioncat/ + /var/lib/vmm/(./.*)? (Closes: #682184) - Fix some leftover files and directories getting incorrectly accounted to the package being tested instead of the dependencies in --warn-on-others mode. - Implement --install-purge-install option: test purging with all dependencies still installed; re-install after purge. - Install the dependencies and snapshot the chroot before the --install-purge-install test. Check that the chroot after purge matches the snapshot. (Closes: #668713) * piupartslib/dependencyparser.py: - Package names may now have a ":any" qualification in some cases. Extend the parser to recognize and discard this suffix. * piupartslib/packagesdb.py: - Implement logfile recycling mode that avoids getting a large backlog of untested packages due to uncoordinated log deletion. Delay log deletion and give preference to packages (or versions) that have not yet been tested. (Closes: #670150) + logfiles to be recycled should be hardlinked from pass/, fail/, ... to recycle/ + recycling can be enabled if recycle/ is not empty + treat packages with logfile in recycle/ as initially "unknown" and compute state ("waiting[-for-dependency]-to-be-tested") + delete old logfile while reserving such a package * piuparts-master.py: - Accept and discard duplicate log submissions, preventing the slave from retrying infinitely. - Delay loading the Packages file which is only needed for "status" and "reserve" commands, but not for submitting logs. - Add "recycle" command to enter logfile recycling mode (needs to be issued before "status" or "reserve"). - Remember idle state by creating stamp files. - Add "idle" command to check a remembered idle status before trying to "reserve" packages (avoid package state computation). Idle status (i.e. no packages available for "reserve") will be remembered for an hour, but any log modification (submission, deletion, marking for recycling) will clear the idle status. * piuparts-slave.py: - Fix stopping the currently running test (Ctrl-C Ctrl-C). - Handle master communication exceptions while sending logs or reserving packages: skip the section but keep the slave running. - Flush finished logs from sections that have processing postponed due to precedence settings. - Skip connection to master if we have reserved but no submittable logs. - Submit finished logs after completing all reserved logs in a section. - Send finished logs to master if interrupted with a single Ctrl-C. - Try to unreserve all reserved logs after interrupt. - Add SIGHUP handler that triggers flushing finished logs. Flushing (all sections) will be done after the current test has finished. - Enter logfile recycling mode if no section has packages left for regular processing. Recycle logfiles until a section becomes available for regular processing. If no logfiles could be recycled, retry after an hour. - Issue "idle" command before "reserve" and skip status computation and reserving if master is idle. * Simplify running piuparts from GIT. * Reorganize layout in the GIT repository to reduce path nesting and length. * Makefile: - Separate build and install targets. - Adjust for new layout, cleanup, simplify. * conf/piuparts.apache: Set DefaultType text/plain (for the logfiles). * reschedule_oldest_logs: Hardlink old logfiles to recycle/ instead of deleting them. Cleanup obsolete rescheduling requests. * New helper script: master-bin/reclassify_bugged. * New custom script post_setup_forbid_home: replace /home with a file to prevent creating home directories there. (Closes: #677332) * New custom script post_install_find_bad_permissions: look for world writable directories without sticky bit set. -- Holger Levsen Mon, 03 Sep 2012 15:28:04 +0200 piuparts (0.45) unstable; urgency=low [ Holger Levsen ] * Merge piatti.git into piuparts.git, so that piatti.git has become obsolete now. Update documentation accordingly. * Further split packaging into piuparts-master, piuparts-slave and piuparts-common binary packages (Closes: #585861) and cleanup dependencies. * Update piuparts.NEWS about master+slave packages. * Switch to debhelper8 packaging. * debian/copyright: use versioned URI of the machine-readable copyright file, thanks lintian. * Move /etc/sudoers.d/piuparts to piuparts-slave package. * piuparts-master: depend on tango-icon-theme and drop sunny.png and weather-severe-alert.png and use links instead. * Reviewed and merged/cherry-picked all of the following changes. [ David Steele ] * Remove debian/postinst as it was only dealing with lenny time area configuration files. * Added server configuration files for apache, sudo, and cron. * Use dh_install, in preparation for multiple package build. * Added packaging to create the piuparts-server package out of the old piatti directory tree (Closes: #652934). - Create and configure piupartss and piupartsm users. - Coordinate ssh keys between master and slave users. - Create working directory tree for each under /var/lib/piuparts * packagesdb.py: - Calculate recursive reverse dependency count, and store in PackagesDB. - Calculate the count of packages being blocked by a failed package. * piuparts-report.py: - Release Section object when section processing is complete. (Closes: #673919) - Display reverse dependency counts and block counts to failure summaries. - Sort the failed-testing and cannot-be-tested summaries by block count. (Closes: #674498) - Replace O(n^2) search in remove_old_logs() with a hash piuparts-report run time improved 20% on mature environment. - Another search fix in create_package_summaries (find_links_to_logs) yielding a 10x speedup for piuparts-report runs. (Closes: #674672) - Cleanup/wrap some long source lines. - Cleanup/wrap some long html lines. - Allow the html root to be changed from / (piuparts.conf: e.g. [global] doc-root = /piuparts/). [ Andreas Beckmann ] * piuparts.py: - Implement --install-remove-install option to test installation in config-files-remaining state. (Closes: #359154) - Report leftover symlinks with target. - Report leftover directories with a trailing slash. - Allow ignore entries to specifically match directories. - Ignore patterns need to match the full filename, not a substring. - Cleanup and regroup ignore lists. + Remove many obsolete entries not needed for squeeze or later. + Group ignore entries into piuparts state, system state, package management, logfiles, home directories, application data, and HACKS. - Add to ignored_files/ignored_patterns: + Default system logfiles as created by syslogd in /var/log/ + Common empty directories in /etc not removed due to dpkg bug #316521 + cdebconf status files + /boot/grub/ + /etc/X11/default-display-manager + /etc/aliases.db + /etc/apt/trusted.gpg.d/.*.gpg~ + /etc/network/interfaces + /etc/news/whoami + /usr/share/keyrings/debian-archive-removed-keys.gpg~ + /var/crash/ + /var/lib/citadel/(data/.*)? + /var/lib/gozerbot/ + /var/lib/nagios/ (Closes: #668756) + /var/lib/openvswitch/(pki/.*)? + /var/spool/powerdns/ (Closes: #531134, #531135) - Implement a timeout for commands being run in the chroot. - Set time limit to 30 minutes per command. (Closes: #654423) - Terminate all processes running inside the chroot before removing the chroot; also in case piuparts aborts due to an error. - Continue killing running processes as long as new processes get spawned in the chroot. - Perform chroot cleanup after receiving Ctrl-C/SIGINT, too. - Let Chroot register/de-register the cleanup panic handler. - Remove temp_tgz on error exit. - Remove metapackage build directory on error exit. (Closes: #663702) - Don't remove eatmydata when minimizing a chroot. (Closes: #658962) - Add support for version-qualified package arguments (--apt foo=1.2-3). - Switch fallback mirror to http://cdn.debian.net/debian. * piuparts.conf: - Defaults for all [section] settings can be set in the [global] section. - Change master-command to not include the section name so that it can be set in the [global] section. The section name will be given as an argument to this command. - Make doc-root default to "/". - New per section key: area (set to one of main, contrib, non-free). - New defaults: mirror=http://cdn.debian.net/debian, area=main. - Compute URLs from mirror, distro, area, arch. - Removed deprecated keys: known_circular_depends, packages-url, sources-url. * master/db/report: Add a new category /affected/ for failed logs where the bug is in a dependency, but only exposed by the package being tested. - In the BTS these are reported as (Closes: #657740) Package: buggy-package Version: buggy-version Affects: package-under-test Found: package-under-test/tested-version - Reschedule /affected/ logs like /fail/ logs. * piupartslib/conf.py: - Add support for getting default values from a special section. - Add methods to get mirror, distro, area, arch from config file or defaults. - Add methods to compute URLs to Packages and Sources files. * piupartslib/packagesdb.py: - Do not consider a package as "untestable" if untestable/ contains a logfile from an obsolete version of that package. - Simplify handling of package and dependency states. - Do not inherit 'dependency-does-not-exist' state (which may be fixed in the package) but propagate it to 'dependency-cannot-be-tested'. - Remove 'no-dependency-from-alternatives-exists' state and use 'dependency-does-not-exist' instead. - Remove 'unknown-preferred-alternative' state as it interferes with circular dependencies involving alternatives and virtual packages. - Fix state resolution of package dependencies containing alternatives, virtual packages and circular dependencies. - Remove 'circular-dependency' state. - Stop using static known_circular_depends. - Compute dependency cycles and test packages on such cycles if all non-circular dependencies are satisfied. - Consider any combination of two error states for the blocking count. - Always run create_subdirs() during initialization. * piuparts-master.py: - Remove known_circular_depends handling. - Reduce logfile verboseness: do not include received logs. - Always chdir to master_directory, do not rely on being run from there. - Record timestamps of submitted logs in submissions.txt. * piuparts-slave.py: - Randomize waiting time (between 60 and 180 seconds) if master is busy. - Sleep until the next section can be tried, but at least 1 minute. - Simplify and merge Section.setup() into Section.__init__(). - Generate master command: use global setting and append section. - Test the 'dpkg' package for creating/updating a base_tgz. - Don't update the tarball for disabled sections. - Rewrite starting piuparts. - Rewrite stopping piuparts for more reliable cleanup. - Introduce Ctrl-C/SIGINT handling: press Ctrl-C once to exit after the current piuparts test has finished, press Ctrl-C again to cleanly terminate the currently running test, press Ctrl-C again to abort any ongoing cleanup. - Pass a version qualified package name (foo=1.2-3) to piuparts to ensure the correct package version gets tested. - Increase verboseness for untestable packages. - Remove idle.stamp functionality. * piuparts-report.py: - Remove known_circular_depends handling. - Exclude obsolete states from generated report. - Establish packagesdb working directory in Section. - Handle logs in /affected/ directories. - detect_well_known_errors: Add bug links if bugs are known. - New known problem: "packages have been kept back". - Report rdeps and blocking counts in all error state reports. - Fix another race with logfiles disappearing while copying. * piuparts-analyze.py: - Don't report further bugs/versions if we found a match. - Classify logs with bugs filed into /bugged/ or /affected/. - Write .bug files with links to the BTS. * detect_archive_issues: Only consider logs older than one day for marking as "untestable" to allow piuparts-analyze to handle the log first. * Replace all references to lenny and squeeze with squeeze and wheezy. * Add/update several bug reporting templates. * Add custom scripts to aid debugging common problems in maintainer scripts. * Add custom script for squeeze to ensure adduser, ucf, and install-info are not purged. * Add custom scripts to enable/perform additional cleanup procedures to reduce the "FAIL: Package purging left files on system" errors if --warn-on-leftovers-after-purge is not used. (Closes: #668752) * Ship custom scripts in the piuparts package. * Comment sudoers and crontabs shipped in the master/slave packages, requiring explicit editing after installation to activate them. * Bump debhelper and dpkg dependencies for using dpkg-maintscript-helper. * Remove obsolete /etc/piuparts/piuparts.conf from old piuparts package. * Ship /etc/piuparts/piuparts.conf in the piuparts-slave package. * Create missing home directories if the piupartsm/piupartss users remain from a previously purged installation. [ Leo Iannacone ] * piuparts.py: add --existing-chroot option, specify a directory as source for the chroot, instead of building a new one with debootstrap or decompressing an archived one. (Closes: #658044) [ Mika Pflüger ] * piuparts.py: Remove obsolete functions shellquote, apt_get_knows, and append_to_file. [ Mehdi Abaakouk ] * piuparts.py: Add support for schroot. (Closes: #530733) -- Holger Levsen Thu, 21 Jun 2012 22:17:26 +0200 piuparts (0.44) unstable; urgency=low [ Andreas Beckmann ] * piuparts.py: - Add to self.ignored_patterns: /var/lib/ucf/.* - Increase output limit to 3 MB (for dist-upgrading daptup). - Do not miss Pre-Depends in --warn-on-others mode. - Remove packages and reinstall missing ones in one apt-get call. - Run 'apt-get dist-upgrade' (instead of upgrade) during chroot setup. * piupartslib/packagesdb.py: - Only consider the newest version of a package as test candidate. (Packages files may contain more than version of the same package.) * piuparts-master.py: - Implement simple section locking to allow concurrent master instances but allow only one instance per section. * piuparts-slave.py: - Do not leak the logfile file descriptor to piuparts. - Skip disabled sections (max-reserved = 0). - Do not abort if connection to master failed, just skip the section. - Do not abort if Packages file cannot be fetched, just skip the section. - Implement simple section locking to allow concurrent slave instances. - Sections wait before retry 15 minutes after error and one hour if idle. - Create idle.stamp files to remember idle sections for one hour and notify concurrent slaves about idle state. - Skip misconfigured sections that have neither 'distro' nor 'upgrade-test-distros' in their config. (Closes: #668863) * piuparts-report.py: - Add link to the list of untestable logs. - Add more known problem reports: forgotten alternatives, warnings in passed logs: leftover files and failed maintainer scripts. - Do not abort if logfiles disappear during copying. * Makefile: Install documentation with mode 0644 instead of 0755. [ Holger Levsen ] * piuparts-report: don't create output for disabled sections. (Bye bye Lenny!) * Bump Standards-Version to 3.9.3, no changes necessary. -- Holger Levsen Thu, 17 May 2012 14:36:53 +0200 piuparts (0.43) unstable; urgency=low [ Andreas Beckmann ] * piuparts.py: - Add to self.ignored_patterns: /run/.* - Unset DISPLAY variable before running commands in the chroot. - Read possibly remaining buffered output after a command terminated. - Run apt-cache policy on the packages to be installed. - Simplify keeping the reference chroot state: move tree info, selections and diversions into a dict, can be extended easily. - Fix --*end-meta documentation and actually enable these options. - Cleanup options and settings. * piupartslib/packagesdb.py: - Restrict circular-dependency state to packages on the circle. * piuparts-slave.py: - Fix chdir in/out of the slave dir to work with relative paths. * piuparts-report.py: - Reorder the list of known problem analyses and add a few new ones. - Correctly split + strip comma-separated lists. * piuparts-analyze.py: - Also consider bugs that 'affects' a package. - Treat bugs without 'found' version as older than anything. * Makefile: Support version number substitution in piuparts-*.py, too. * README.txt: Document piuparts.conf settings used by piatti.git scripts. [ Holger Levsen ] * piuparts.1.txt: correctly reverse the description of --save-end-meta and --end-meta. * piuparts-report.py: - add support for more static pages and add bug_howto.tpl, explaining how to file meaningful bugs piuparts-analyse understands. This page also links to the templates used for bug filing. * piuparts-slave.py: make status message more useful by including the section. -- Holger Levsen Sun, 22 Jan 2012 13:03:59 +0100 piuparts (0.42) unstable; urgency=low [ Holger Levsen ] * piuparts.py: - add to self.ignored_files: /etc/blkid.tab (Closes: #638831) - add to self.ignored_patterns: /var/lib/apt/lists/.* - apply patch by Gregor Herrmann to fix --minimize. (Closes: #648423) * Remove Debian.NEWS entry about source in git. (Closes: #640121) * piuparts.py, piuparts-report.py, ChangeLog: Expand tabs to spaces. * Remove whitespaces from whitespace-only lines. * piuparts-report: - create maintainer subdirs if they don't exist. - detect tests being terminated due to excessive output. * Add git to Build-Depends-Indep: as git describe is used to generate part of the version string for development builds. * Add debian/.gitignore [ Mika Pflüger ] * piuparts-analyze.py: - Rewrite to use python-debianbts to analyze if bugs are filed already. - The BTS only tracks source versions, so remove binNMU-part from package versions when comparing with versions from the BTS. - Reduce noise in the output by only printing one action/advise per package. - Fix extraction of package versions from bug reports. Thanks to Andreas Beckmann for catching and solving the error. * debian/control: Add python-apt and python-debianbts to piuparts depends. [ Scott Schaefer ] * debian/copyright: Make it compliant with DEP-5. * piuparts-slave.py: - Replace deprecated os.popen2 with subprocess.Popen. (Closes: #640646) - Add some more logging. - Kill children (hard-coded value, 45 minutes) to ensure test doesn't run "forever" (Closes: #640647, #387428) * piuparty.py: - Kill leftover processes (via SIGTERM, then if that fails, via SIGKILL). (Closes: #522918) - Test for incorrect diversion handling: (Closes: #588313) a) Existing diversions removed/modified, and/or b) Installed diversions not removed by purge. * piupartslib/packagesdb.py: Modify several functions in PackagesDB class to use python 'set' type, instead of a list. This permitted replacing inefficient combination of unique() function and random.shuffle() with python set.pop() method. Since python prohibits storing non-hashable object in a set, minor modifications to piuparts-report and to piuparts-master required. (Closes: #640648) [ Andreas Beckmann ] * *.py: Add vim modeline. * piuparts.py: - Add unittests for misbehaving symlinks. - Fix resolving absolute symlinks of intermediate directory components, i.e. /var/run -> /run while checking /etc/motd -> /var/run/motd. Solves about 30000 false positives of 'Broken symlinks: /etc/motd -> /var/run/motd'. (Closes: #648784) - When running commands in the chroot, redirect stdin from /dev/null. - Stop using Popen.communicate() as it may run out of memory. - Terminate commands producing more than 2 MB of output. (Closes: #649335) - Create /etc/dpkg/dpkg.cfg.d/ if missing inside the chroot (e.g. on lenny). (Closes: #647752) - Remove logrotate and its dependencies only once. - Only run 'apt-get update' after updating the sources.list. - Only run 'apt-get clean' before creating tarballs or saving meta data. - Do the same checks for running processes and broken symlinks in all tests. (Closes: #648708) - Create piupart's apt config in the chroot as /etc/apt.conf.d/piuparts instead of /etc/apt.conf in order to allow overriding the settings from custom scripts by just dropping new config bits in e.g. /etc/apt/apt.conf.d/piuparts-foobar. - Integrate diversion checks with Chroot.check_result(). - Use 'apt-get remove' to remove all packages at once with proper dependency ordering. (Closes: #648162) - Purge all packages at once instead of doing it one-by-one. - Make restore_selections() reinstall missing packages. (Closes: #648707) - Set more environment variables to let custom scripts know where and when they are being run: PIUPARTS_TEST, PIUPARTS_PHASE, PIUPARTS_DISTRIBUTION{,_PREV,_NEXT}. See the README for details. (Closes: #589498) - Add missing post_install_* custom scripts to install_packages_by_name(). (Closes: #628077) - Remove pre_upgrade_* custom scripts, they can be replaced with pre_install_* scripts that check for PIUPARTS_PHASE=upgrade. - Add pre_test_* custom scripts. These are run exactly once at the beginning of each test (after recording the initial chroot state). - Allow multiple --scriptsdir options, the contents will be merged. - Exclude /tmp/scripts when creating a tarball. - Use --one-file-system when creating a tarball to exclude bindmounts etc. - Restore base system from temp_tgz instead of running debootstrap again. (Closes: #648193) - Do not fail upgrade/distupgrade tests for a set of packages where not all packages exist in the start distribution. This happens e.g. when testing .changes files and packages were split/added. Only install the old packages that actually exist according to 'apt-cache show'. - Add --extra-old-packages option to intall additional old packages that are not in the set of new packages to be tested. Useful for testing upgrades with Conflicts/Replaces relationships, e.g. in package renames or merges. - Use consistent variable names for package lists. (Closes: #648177) - Compute the changes in restore_selections(). - Check for settings.scriptsdir inside run_scripts(). - Consistently use chroot.relative() to build filenames inside the chroot. * piupartslib/packagesdb.py: - Handle 'unknown-preferred-alternative' state correctly. - Add 'does-not-exist' state for dependency resolution to distinguish this from 'unknown' state so that the latter only indicates 'unresolvable' or 'not yet resolved'. - Handle virtual packages separately from real packages. - Use get_package_state() internally which 'resolves' (purely) virtual packages by default (can be disabled). * piuparts-master.py: - Add a 'status' command that reports package counts for all states. * piuparts-slave.py: - Fix triggering tarball recreation. - Check tarball age regularily. - Log tarball creation in *.tgz.log. - Request and print package counts from master. - Reload section config every time a section is being run. - Add precedence attribute to allow prioritizing different sections and to suspend processing of low priority ones while there are packages waiting in more important sections. * piuparts-report.py: - state-*.html: Sort package lists by name, display state of all alternative dependencies and packages providing a virtual dependency. - source/?/*.html: Sort binary packages by name. - maintainer/?/*.html: Sort source packages by name. - Update list of error states to be highlighted. - Archive logs of packages that were removed from the distribution. - Speedup generating maintainer summaries. * Makefile: Use 'git describe' to get an exact stamp for development versions. [ Dave Steele ] * piuparts-slave.py: make Section.run() report the number of packages processed and use this to decide whether a slave should sleep. (Closes: #649967) [ Stefano Rivera ] * piuparts.py: - Properly install and remove logrotate. (Closes: #638832) - Use eatmydata by default, add option --no-eatmydata. (This was discussed in #633033.) -- Holger Levsen Fri, 23 Dec 2011 10:51:28 +0100 piuparts (0.41) unstable; urgency=low * piupartslib/packagesdb.py: Apply patch by Scott Schaefer to also consider alternative depends: the algorithm used to select the "best" alternative is: 1) Prefer first alternative in state "essential-required" 2) If no "essential-required" alternatives, prefer first alternative in state "successfully-tested" 3) Otherwise, prefer first alternative in state "waiting-to-be-tested" IF NO REMAINING alternatives are in one of the "unknown/fail" states For this two new states have been introduced: a) "unknown-preferred-alternative": equivalent of "unknown", this defers calculation of this package's state, since one or more of its alternative depends are "unknown" (or "unknown-preferred-alternative"), and no alternative is either "essential-required" or "successfully-tested". The alternatives will be re-tested on subsequest pass. b) "no-dependency-from-alternatives-exists": if none of the alternatives can be found in the archive. (Closes: #526045) * piuparts-report.py: - correct a typo from the patch for #523950. - make_stats_graph: draw the two new states as well. - point to source code in git instead of svn. * piuparts.py: - use proxy settings either from apt configuration or http_proxy environment variable, the latter overwriting the former (if present) - Thanks to Scott Schaefer for the patch. (Closes: #632046) - new option "--no-install-purge-test" to only do upgrade tests - Thanks to Andreas Beckmann for the patch (Closes: #588482) - run dpkg with --force-unsafe-io by default and introduce new option "--dpkg-noforce-unsafe-io" to disable this feature. (Closes: #633033) Thanks to Scott once more! - new option: "--fail-on-broken-symlinks". Remove option "--warn-symlinks" as this is the default now. * piuparts-slave: - check if chroot-tgz is older than max-tgz-age (defaulting to 30 days) and recreate it, if it is. Keep backup and put in back in place when debootstrapping a new chroot-tgz fails. Retry after min-tgz-retry-delay seconds. Thanks to Scott Schaefer for the patch. (Closes: #632924) - document new configuration options max-tgz-age and min-tgz-retry-delay in README.txt. - fix bug in piuparts-slave which prevented running a single section by including section name as command-line argument - thanks again to Scott and see 632924 too. * debian/control: - require python >= 2.6.6-3~, get rid of python-support build-depends. - add X-Python-Version: pseudo-header * debian/rules: use dh_python2 instead of python-support to build the package * Makefile: build for python 2.6 and 2.7 instead of 2.5 and 2.6. * Source code is now stored in git. https://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git git clone git+ssh://git.debian.org/git/piuparts/piuparts.git Thanks to Thomas Koch for setting those up and keeping history! * The configuration and scripts used on piatti.debian.org have also been moved to git, the URLs are the same, just s#piuparts#piatti#. * Add entry about git to NEWS. * Update Vcs-*-Headers in control to reflect move to git. -- Holger Levsen Wed, 24 Aug 2011 13:47:07 +0200 piuparts (0.40) unstable; urgency=low * piuparts.py: - make "natty" the default Ubuntu distribution. - add to self.ignored_files: - /var/log/bootstrap.log * debian/control: depend on ${python:Depends} instead of just python. Thanks to Jakub Wilk. (Closes: #614617) * *.py: do not use Python built-in function names as variable name. Thanks to Carl Chenet for a first patch and Scott Schaefer for the applied one. (Closes: #523950) * Bump Standards-Version to 3.9.2, no changes necessary. * debian/rules: add empty build-arch and build-indep to appease lintian. * Makefile: drop workaround for #559305 in lenny. -- Holger Levsen Sat, 25 Jun 2011 21:20:31 +0000 piuparts (0.39) unstable; urgency=low [ Holger Levsen ] * The vintage of 2010 release! * piuparts-report.py: - report packages which failed - due to broken maintainer scripts, - due to output from cron after removal, - due to not enough force being, - due to a problem with pre-depends, - due to files having been modified after purge, - due to files having disappeared after purge and - due to problems configuring a database. - only report about (un)owned files and directories and symlink issues in sid - that's hardcoded as it's not practical to track them anywhere else anyway. And at least for broken symlinks this will also stay like this for a long time. - visual improvements in the output. - ignore udebs and binaries from other archs too. - graph is generated using all the available data points. * piupartslib/packagesdb.py: - automatically break circular dependencies when there are only circular dependencies left, using a configurable list of circular depdencies in piuparts.conf (Closes: #526046) The list of circular depends is taken from a script written by Robert Lemmen available at http://debian.semistable.com/debgraph.out.html - obviously it would be better to merge this into piuparts directly. Note that the dependency resovler is still buggy, the state waiting-for-dependency-to-be-tested still contains some packages with failed depends (due to them being dependent on packages with circular depends). This bug has no effect other than that (so the state waiting-to-be-tested is calculated correctly). This bug is also no regression. - get rid of the longtime unused states "fixed" and "fix-not-yet-tested" (also in piuparts-master and piuparts-report). - forget reservations of untestable packages. * piuparts-slave.py: - fix crash when going into sleep when idle. - add support for only doing upgrade tests (without the basic test in the upgrade target distro). - honor mirror configuration option also for upgrade-tests. - also do upgrade tests if the version of a package being tested (the one in the upgraded distro) is not available in the distro being upgraded from. (This is a very short test, but a successfully one.) - make configuration setting "distro" to not default to any distro. * piuparts.py: - new option: --log-level to specify the verbosity of piuparts output. Thanks to Fabrice Coutadeur for the patch! (Closes: #567190) - new option: --warn-on-leftovers-after-purge to not fail if a package leaves files behind after purge. (Closes: #566599) - add to self.ignored_files: - /var/lib/apt/lists/partial/.delete-me-later - /var/log/alternatives.log (Closes: #591966) - /var/log/apt/history.log (Closes: #594964) - /usr/share/fonts/X11/misc/fonts.alias (Closes: #576321) - add to self.ignored_patterns: - /etc/init.d/.depend.* - /var/mail(/.*)? - this used to be just /var/mail/.* - /var/lib//update-rc.d(/.*)? (Closes: #605127) - create temporary DEBIAN directory for the piuparts-depends-dummy package with 0755 perms regardless of the umask of the calling shell. (Closes: #573904) - fix --do-not-verify-signatures option (Closes: #574936) - thanks to Cleto Martin Angelina for the patch. - make Chroot.diff_meta_data() special case and ignore files matching (/etc/rc.\.d/)[SK][0-9]{2}(.*)$ so that piuparts doesn't complain about initscripts renamed by insserv. Thanks to by Andreas Beckmann for the patch! (Closes: #586793) - remove logrotate and depended packages after the test. (Closes: #602409) This is a fix for the incomplete patch for #566597. Again, this is a hardcoded list. :-( - new custom script: pre_remove_ (Closes: #539146) - set environment variable PIUPARTS_OBJECTS in custom scripts to a space separated list of packages / changes files being tested. - do not call apt-get with --no-remove when installing packages. (Closes: #603453) - apply patch by Andres Mejia to fix parsing of the --force-confdef option and also to configure apt to use that option. Thanks, Andres. (Closes: #605475) - don't panic if package is not known by apt-get, this probably just means a package not yet in the archive is being tested. (Closes: #566672) Thanks to Cleto Martin Angelina for the patch! - fix parsing of the options --end-meta and --save-end-meta. - supply help texts for --end-meta and --save-end-meta options, also add comments to install_and_upgrade_between_distros() to make the code easier to understand and hopefully improve in future. (Closes: #560050). - add logging to install_and_upgrade_between_distros() to point people to read the functions source code if they wonder why the log (for distro upgrade tests) looks like it looks. * piuparts.1.txt: - update the pointer to custom-scripts.txt to point to README.txt, as those two files have been merged. Thanks to Gregor Hermann for spotting this. (Closes: #574504) - indicate that it's possible to check several packages or .changes files at once. - update several options descritpions. * README.txt - better documentation of custom scripts. - reformat to achieve consistent width. * Add ${misc:Depends} to Depends in debian/control. * Remove versioned dependencies on debootstrap as the version available in Etch is sufficient. Drop build-depends on dpkg-dev as it's in build-essential. * Build-depend on debhelper>=7 and use dh_prep instead of dh_clean -k, bump compat level to 7. * Bump Standards-Version to 3.9.1, no changes necessary. * Remove Ian Jackson from uploaders - thanks for your work, Ian! * Makefile: support python 2.5 and 2.6 instead of 2.4 and 2.5, adjust debian/control accordingly. * debian/preinst: update code snipplet rm_conffile() from https://wiki.debian.org/DpkgConffileHandling [ Evgeni Golov ] * piuparts.py: - Check for remaining logrotate files which produce output after the package was removed. (Closes: #566597) [ John Wright ] * Use debian module when available, else debian_bundle module. (Closes: #586443) * Use built-in set type, available since python-2.4. -- Holger Levsen Tue, 04 Jan 2011 14:12:30 +0100 piuparts (0.38) unstable; urgency=low * piuparts.py: - Add support for using LVM snapshots. Thanks to Patrick Schoenfeld for the patch. (Closes: #559449) - Add support for warning on broken symlinks. Thanks to Carl Chenet for the patch. (Closes: #440151) - Make --help usuable without needing to be run as root. (Closes: #547379) - --skip-minimize is now the default. This is because debootstrap is called with --variant=minbase instead of --resolv-deps now and because if a base.tgz or an lvm snapshot is specified, piuparts should not touch it. (Closes: #539142, #559048) Introduce a new option, --minimize, to minimize a chroot with debfoster. (This is the old default behaviour. In future versions the --minimize option might be removed again.) - Add two new options: --keyring to specify the keyring to use with debootstrap (defaults to /usr/share/keyrings/debian-archive-keyring.gpg) and --do-not-verify-signatures to not use --keyring with debootstrap. (Closes: #545907) In the chroots, APT::Get::AllowUnauthenticated is set accordingly in /etc/apt/apt.conf. - Add new option, --pedantic-purge-test, to tell piuparts to be pedantic when checking if a purged package leaves files behind. If this option is not set, files left in /tmp are ignored. (Closes: #528266) - Add new option, --dpkg-force-confdef, to make dpkg use --force-confdev, which lets dpkg always choose the default action when a modified conffile is found. This options will make piuparts ignore errors it was designed to report and therefore should only be used to hide problems in dependent packages. (Closes: #466118) - Add another type of custom-scripts, post_setup_, which are executed after the chroot was setup. (Closes: #466043) - Create policy-rc.d in 2nd chroot when using -b. (Closes: #466049) Thanks again to Patrick Schoenfeld for his help. * piuparts-report: report broken symlinks. * piuparts-slave: - gracefully deal with upgrade-test-distros and upgrade-test-chroot-tgz not being set in the configuration. Thanks to Carl Chenet and Patrick Schoenfeld for the patch. (Closes: #519192) - new default value for idle-sleep, 300 instead of 10 seconds. Also the slave will now only sleep if there is nothing to do for any section. * Makefile: workaround 559305 by calling a2x twice. * Update debian/NEWS. * Replace all references to sarge and etch with ones to lenny and squeeze. -- Holger Levsen Thu, 17 Dec 2009 14:10:27 +0000 piuparts (0.37) unstable; urgency=low * piuparts-report.py: report packages with update-rc.d warnings and those failing due to insserv errors. (Closes: #546540) * Fix typo in NEWS.Debian, thanks to Justin Rye for spotting it. (Closes: #547439) * piuparts.py: - added to self.ignored_patterns: - /var/cache/man(/.*)? (Closes: #550953) - /var/lib/insserv/run.*.log - /var/lib/mercurial-server(/.*)? - removed from self.ignored_files: - /var/cache/man/index.db * fix typo in -i option in manpage (Closes: #555202), thanks to James Vega for spotting it. * Make "lucid" the default Ubuntu distribution (Closes: #559047), thanks to Krzysztof Klimonda. * fix FTBFS by adding "-r /etc/asciidoc" to the a2x call in Makefile, thanks to Michael Vogt. (Closes: #559299) -- Holger Levsen Thu, 03 Dec 2009 13:25:43 +0000 piuparts (0.36) unstable; urgency=low [ Holger Levsen ] * The "For me. For you. For everybody." release. * Break backwards compatibility of the configuration files for master-slave- mode. Merge those three into one: /etc/piuparts/piuparts/piuparts.conf. Introduce a new global section in piuparts.conf which is shared among the other sections there. * piuparts.py: - add check whether scriptsdir exits, to fail gracefully if not. - copy scriptsdir to chroot also when doing upgrade tests. - added to self.ignored_files: - /etc/shadow and /etc/shadow- (just as /etc/passwd* and /etc/group*) - /var/games (see #524461) - /etc/apt/trusted.gpg and /etc/apt/trusted.gpg~ (just as /etc/apt/trustdb.gpg and other backup files) - added to self.ignored_patterns: - /var/lib/dpkg/triggers/* - /var/lib/ldap(/.*)? (see #527898) - /var/backups/.* - modified self.ignored_patters: - ignore everything in /tmp, not just in /tmp/scripts (see #528266) - configure apt in chroots to not install recommends and suggests. - add support for scanning for packages in changes files, thanks to Andres Mejia for the patch. (Closes: #352940) - change some methods from using 'args' to 'package_list'. This more accurately represents what is being passed into these methods now. - add an optional parameter to panic() method to specify what exit status to use. Also thanks to Andres. - add advice how to read the logfile to the top of the logfiles produced. - add "FAIL: " to logging output for seven more failure types, so that it becomes easier to group failure causes. - allow piuparts to be stopped with control-C. Thanks to Carl Chenet for the patch. (Closes: #523958) - fail gracefully if piuparts is run as non-root user. Thanks to Ignace Mouzannar for the patch. (Closes: #538273) * piupartslib/packagesdb.py: - change the test whether a package is testable to check whether the package is of priority "required", and not whether it's "Essential". - rename status "essential-required-important" to "essential-required" as important packages can be tested like all the others. - handle binNMUs where the Source header in the Packages file includes the Version and the Source package name in one line. - fix get_packages_in_state() to only return unique packages and not also the provided ones as exact copies. * Install scripts for master-slave mode without .py extension. * piuparts-master.py: remove code to write statistics, that is done only in piuparts-report.py now. * piuparts-slave.py: - support looping trough multiple sections in piuparts.conf via new config value "sections". Thanks to Frank Ploss for writing this patch with me and helping me understand the code much better! - allow the user to stop the slave with control-C. - call piuparts with --mirror when mirror is set in piuparts.conf. * Re-add piuparts-reports.py which got removed/lost between 0.14 and 0.20 without changelog entry. * piuparts-report.py: - support sections in configuration file. - support looping trough multiple sections in piuparts.conf via new config value "sections". - generate counts.txt (per section) with raw numbers of packages per state, include these stats in the debug output (so it gets included in the mails send by cron.) - introduce new setting "sources-url" for piuparts.conf. - generate sources.txt (per section) with a summary status per source package (for the PTS to display a source packages piuparts state) - generate html status pages for all source packages (to link from the PTS) with links to state explaination and available logfiles, handle udebs. - provide links to logfiles in statistics page. - provide links to source packages pages from state pages and back, as well as links to the dependencies state. - draw graphs of package states over time (if pypthon-rpy and gs are installed) - thanks to Thomas Viehmann and Dirk Eddelbüttel for inspiration and support. - create maintainer centric pages for each email address found in Maintainers or Uploaders. - improve layout, generate navigation for all pages. - use icons to provide a quick overview of a source packages status. (Thanks to tango.freedesktop.org for the icons!) - include index.tpl (if it exists) from output-directory into the generated index page, so one can add news to the index page without editing piuparts-report.py. - generate pages for known issues, which are detected by bash helper script, run by cron on piuparts.debian.org and available from svn/piuparts/piatti/home/piupartsm/bin/detect_well_known_errors * Always use alphabetic time zone abbreviation in timestamps. * Makefile: - add "~$date" to versionstring if building an unreleased version, thus adding dpkg-dev to build-depends. - split install target into install-doc, install-conf and install, to aid development and deployment from trunk. * debian/control: - depend on python (>>2.4), make dependency to python-debian unversioned. - add build-dependencies on debhelper, asciidoc and xmlto. - dependency gs has been renamed to ghostcript - remove build-dependencies on docbook2x and docbook-xml. - suggest python-rpy and gs to improve piuparts-report output. - set Homepage: to http://piuparts.debian.org - remove Lucas Nussbaum, Ana Guerrero, John Wright and Bill Allombert from uploaders - thanks for your work! - Replace Ians email address with one that doesn't give an errorcode when sending mail to it - bump Standards-Version to 3.8.3, no changes necessary. * Rewrite debian/rules from scratch using debhelper. * Merge README, how-to-use-piuparts.txt and custom-scripts.txt into README.txt, convert it to asciidoc and build pdf and html versions of it. * Restructure and update README.txt to reflect the configuration changes in master-slave mode. * Add debian/NEWS file. * Rewrite piuparts manpage in asciidoc. * Build and install html version of the piuparts manpage. * Update debian/copyright to reflect that piuparts is licenced under GPL2+. * Update FSF address in all files referencing the GPL. * Remove unused file piuparts.css. [ Lars Wirzenius ] * Removed Lars Wirzenius as uploader. -- Holger Levsen Tue, 08 Sep 2009 14:35:42 +0200 piuparts (0.35) unstable; urgency=medium * Fix recursive failure which occured when selinux-utils was installed but not enabled. Thanks to Peter De Wachter for the patch. (Closes: #519017) * Output path to temp directory if -k is used. (Closes: #466112) -- Holger Levsen Tue, 10 Mar 2009 00:43:22 +0100 piuparts (0.34) unstable; urgency=low [ Holger Levsen ] * Mount /selinux on systems where selinux is enabled. Thanks to Filippo Giunchedi for the patch! (CLoses: 507171) * Remove wrong advice in debian/copyright which irritated lintian. [ Filippo Giunchedi ] * Check for any output when running cron files in addition to exit code -- Holger Levsen Fri, 27 Feb 2009 12:34:31 +0100 piuparts (0.33) unstable; urgency=low * Added --bindmount option, thanks to Aníbal Monsalve Salaza for the patch. (Closes: #496186) -- Holger Levsen Sat, 08 Nov 2008 17:07:22 +0000 piuparts (0.32) unstable; urgency=low [ John Wright ] * Fix a typo in how-to-use-piuparts.txt. [ Holger Levsen ] * Replace all references to sarge and etch with etch and lenny. (Closes: #466111) * Update README to reflect that piuparts runs fine in etch. * Rename the post_upgrade custom script to post_distupgrade and introduce pre_distupgrade. * Bumped standards version, no changes needed. -- Holger Levsen Fri, 17 Oct 2008 10:28:14 +0000 piuparts (0.31) unstable; urgency=low [ Lars Wirzenius ] * piuparts.docbook: Added a few words of description of the tarball that -b expects. (Closes: 466048) [ Lucas Nussbaum ] * Added a --debfoster-options option, to allow the user to override debfoster's default options and test with more packages installed in the chroot. (Closes: #458922) * Mention piuparts -s in in how-to-use-piuparts.txt. * Finally fixes the check for broken symlinks. Thanks go to Tobias Grimm for the patch. (Closes: #468157, 478587) * Ignore /var/cache/ldconfig/aux-cache. * Added myself to uploaders. * Keep /proc mounted. Switch to calling tar directly in pack_into_tgz(). Minor refactorings. Thanks go to Tobias Grimm for the patch. (Closes: #478577) * Move piuparts to priority: extra, since it depends on debootstrap which is extra. (Closes: #477634) [ Luk Claes ] * Added -w to lsof call to suppress warnings. (Closes: #466102). [ Holger Levsen ] * Add a copyright statement to debian/copyright. * Add support for post_upgrade custom script. * Minor fixes in how-to-use-piuparts.txt. * piuparts.docbook: Correct the path to dtd, it's /usr/share/xml/docbook/schema/dtd/4.1.2/docbookx.dtd and add a build-dependency for docbook-xml -- Holger Levsen Sun, 11 May 2008 22:17:52 +0200 piuparts (0.30) unstable; urgency=low * piuparts.py: - Implement Chroot.create_temp_tgz_file() (since it's used in the VirtServ subclass) - Fix a typo -- chroot.create_temp_tgz() was being called instead of chroot.create_temp_tgz_file() (Closes: #465416) -- John Wright Thu, 14 Feb 2008 20:32:35 -0700 piuparts (0.29) unstable; urgency=low [ Ana Beatriz Guerrero Lopez ] * Updated uploaders, remove Amaya and Alastair. [ Lars Wirzenius ] * piuparts.py: Call "apt-get update" before calling Chroot.minimize, so that we can find the debfoster package on Ubuntu. * debian/control: Fixed "Uploaders:" to have my preferred e-mail address (liw@iki.fi) instead of my Debian one. * piuparts.py: Added -D option to set Debian flavor, plus two sets of default settings depending on the flavor, one for Debian itself, and another for Ubuntu. The settings choose default mirror and distribution set (sid vs gutsy). This will allow an Ubuntu version of the package to set defaults at package building time. * piuparts.py: Report target of broken symlink correctly, instead of saying "True". * piuparts.py: Use lsb-release to guess the Debian flavor. * debian/control: Added dependency on lsb-release. * piuparts-master.py: Make the master write summary of total pass/fail packages, plus status per binary package. This is for having the Debian PTS and the Ubuntu developer weather report show summaries of piuparts results. * integraged changes from Ubuntu by Ian Jackson to support autopkgtest. * wrapped long lines in debian/control. [ Holger Levsen ] * Added myself to uploaders. * Added Homepage: header to debian/control and changed the homepage to be https://wiki.debian.org/piuparts. * Use Vcs-* headers in debian/control instead of XS-Vcs-*. * Bumped policy version to 3.7.3. * Updated download location in debian/copyright. [ John Wright ] * Change the --warn-on-others implementation to create a dummy metapackage with the needed dependencies and install it in the chroot, rather than using "apt-get install package package-". The previous implementation had issues when the package was already installed (and especially if the package had essential packages in its reverse dependencies). This has the pleasant side-effect of making --warn-on-others work both with and without --apt. (Closes: #458929) * Fix a bug in check_for_broken_symlinks(): the targets for the broken symlinks (used for logging) were being read from files on the real root filesystem, rather than in the chroot. -- Lars Wirzenius Fri, 01 Feb 2008 16:38:38 +0200 piuparts (0.28) unstable; urgency=low * Fix an UnboundLocalError introduced in the --warn-on-others fix that would occur if run without the --apt option. (Closes: #443641) -- John Wright Sat, 22 Sep 2007 23:32:17 -0600 piuparts (0.27) unstable; urgency=low * Add support for testing multiple distributions and architectures to piuparts-master.py and piuparts-slave.py. Please see the README file, piuparts-master.conf.sample and piuparts-slave.conf.sample for more details. (Closes: #349365) * Fix the --warn-on-others option. Now, dependencies are installed before the packages we wish to test, and an inventory is taken then, so that we can know which errors were caused by the packages explicitly specified on the command-line. Currently, this requires --apt, and doesn't work quite as advertised if there are circular dependencies with the packages you wish to test (see the man page for more details). (Closes: #440310) * debian/control: - Update my email address in the Uploaders field -- John Wright Sun, 16 Sep 2007 22:28:14 -0600 piuparts (0.26) unstable; urgency=low * Update list of ignored files. (Closes: #439592) * In the cron test, not executable files are not run. (Closes: #440141) -- Ana Beatriz Guerrero Lopez Fri, 31 Aug 2007 15:44:36 +0200 piuparts (0.25) unstable; urgency=low * Add the option to run custom scripts inside the piuparts chroot. Scripts are stored in a directory and give it as argument with the option --scriptsdir=/dir/with/the/scripts This feature is still experimental :) * Add custom-scripts.txt with information about the custom scripts. -- Ana Beatriz Guerrero Lopez Tue, 28 Aug 2007 14:39:32 +0200 piuparts (0.24) unstable; urgency=low * Add /var/lib/apt/extended_states to ignored_files. Thanks Anibal! (Closes: #434980) * Add quick howto about how to use piuparts under docs. * Add test that checks the output from the cron files left in the system after removing a package. This includes the option --skip-cronfiles-test, to allow skipping this test. -- Ana Beatriz Guerrero Lopez Mon, 20 Aug 2007 10:27:29 +0200 piuparts (0.23) unstable; urgency=low * Cosmetic change, now file owner and file are shown in the same line. * Add option --list-installed-files, that list files added/removed/modified to the chroot after the package's installation. (Closes: #431821) * Add option --no-upgrade-test, that allows skip testing upgrade from an existing version in the archive. (Closes: #349933) -- Ana Beatriz Guerrero Lopez Tue, 10 Jul 2007 19:47:21 +0200 piuparts (0.22) unstable; urgency=low [ John Wright ] * debian/control: - Add XS-Vcs-Svn and XS-Vcs-Browser fields [ Ana Beatriz Guerrero Lopez ] * Rename piuparts.py.in back to piuparts.py. * Add option --skip-minimize that allows skip the minimize chroot step with debfoster. * Remove m4 substitution, and use sed instead. Drop Build-Depends on m4. (Closes: #431248) * Drop support for python 2.3 and add support for python 2.5. * debian/rules: - Remove execution of checkversion in package build, not longer needed. * debian/control: - Add myself to uploaders. - Add (future) piuparts website. -- Ana Beatriz Guerrero Lopez Wed, 04 Jul 2007 21:03:44 +0200 piuparts (0.21) unstable; urgency=low * Convert to Debian-native packaging style. From now on, changes to piuparts will be tracked here. (Closes: #389610) * Makefile: - Give docbook2x-man a --encoding=utf-8 argument. It was failing to build. - Generate piuparts.py from piuparts.py.in, filling in the VERSION variable with the version from debian/changelog * debian/control: - Fix the name of the maintainer (we're the "piuparts developers team", not Lustre) - Correct my email address in the uploaders field - Add Build-Depends on m4 * piuparts.py.in: - Don't call shellquote() on arguments, since we're not using the shell. (Closes: #386839) - Add a --warn-on-others option. See the man page for details. * piuparts.docbook: - Document the --warn-on-others option. -- John Wright Thu, 21 Jun 2007 00:27:22 +0100 piuparts (0.20-3) unstable; urgency=low * New Maintainer(s): piuparts team. Closes: #390754. -- Alastair McKinstry Mon, 23 Oct 2006 16:02:19 +0100 piuparts (0.20-2) unstable; urgency=low * Orphaning. -- Lars Wirzenius Mon, 2 Oct 2006 23:43:00 +0300 piuparts (0.20-1) unstable; urgency=low * New upstream version. Fixed Debian bugs: - Symlink correctness checking was broken when symlinks pointed at symlinks. (Closes: #387796) - fails if a deb filename contains a "strange" char. (Closes: #386839) -- Lars Wirzenius Fri, 22 Sep 2006 12:58:24 +0300 piuparts (0.19-1) unstable; urgency=low * New upstream version. No Debian bugs fixed. -- Lars Wirzenius Fri, 8 Sep 2006 20:28:31 +0300 piuparts (0.18-1) unstable; urgency=low * New upstream version. No Debian bugs fixed. * debian/control: Updated for new Python policy. * debian/prerm, debian/postrm: Written. They compile/remove byte code files. -- Lars Wirzenius Thu, 7 Sep 2006 20:42:03 +0300 piuparts (0.17-1) unstable; urgency=low * debian/control: Added dependency on lsof, since that is now used to check that there are no processes running inside the chroot. * New upstream version, fixes the following open bugs in Debian: - missing words in piuparts(1) (Closes: 362551) - default ignore pattern for papercut files added (Closes: #355401) -- Lars Wirzenius Sun, 25 Jun 2006 19:35:19 +0300 piuparts (0.16-1) unstable; urgency=low * New upstream version. Fixes the following bugs reported against the Debian package: - TODO: keep track of bugs to show expected failures and unexpected successes (Closes: #353215) - It would be nice if the chroot were updated before the snapshot (Closes: #356678) * debian/control: Made the dependency on debootstrap be versioned. Closes: #355875. -- Lars Wirzenius Fri, 22 Mar 2006 22:40:00 +0200 piuparts (0.15-1) unstable; urgency=low * New upstream version. Includes fixes for Debian bugs: - "Specify packages to use for testing" (Closes: #354811) - "Misleading documentation of -v" (Closes: #352941) -- Lars Wirzenius Wed, 01 Mar 2006 20:53:00 +0200 piuparts (0.14-1) unstable; urgency=low * New upstream version. Includes fix for: - "piuparts should flag as an error a failed installation of a dependency" (Closes: #343324) -- Lars Wirzenius Sun, 18 Dec 2005 23:22:00 +0200 piuparts (0.13-1) unstable; urgency=low * New upstream version. * debian/changelog: Minor tweaking to the description. -- Lars Wirzenius Sat, 12 Nov 2005 00:53:04 +0200 piuparts (0.12-1) unstable; urgency=low * New upstream version. Fixes bug in Debian: - dist-upgrade to experimental does not work (now documented as a feature). Closes: #330749. -- Lars Wirzenius Mon, 17 Oct 2005 21:03:12 +0300 piuparts (0.11-1) unstable; urgency=low * New upstream version. Fixes bugs in Debian: - Checks whether there are packages on the command line before creating a chroot. Closes: #322441. - apt-get configured to allow unauthenticated repositories. Closes: #326705. -- Lars Wirzenius Fri, 14 Oct 2005 01:16:25 +0300 piuparts (0.10-1) unstable; urgency=low * New upstream version. -- Lars Wirzenius Thu, 15 Sep 2005 23:31:30 +0300 piuparts (0.9-1) unstable; urgency=low * New upstream version, fixes bugs reported against Debian: - Upstream version number in Debian package and in upstream sources now match. Closes: #326058. - Files reported by piuparts are now associated with their packages, when possible. Closes: #324248. * debian/rules: Added "checkversion" target to make sure we no longer do stupid things like #326058, at least in the Debian package. -- Lars Wirzenius Fri, 09 Sep 2005 00:34:36 +0300 piuparts (0.8-1) unstable; urgency=low * New upstream version, fixes bugs: - Example in manual page now works (Closes: #319990) - Temporary directories are now removed on failure (Closes: #323653) - Parsing /etc/apt/sources.list now better (Closes: #319989) - Temporary directorys are now put in $TMPDIR or /etc (Closes: #322440) * debian/control: Added build-depends on docbook2x. Closes: #318693 * debian/changelog: Added note about upstream source repository. -- Lars Wirzenius Mon, 29 Aug 2005 23:03:32 +0300 piuparts (0.7-1) unstable; urgency=low * New upstream version. * This is the first package that will be uploaded to the Debian archive. -- Lars Wirzenius Fri, 15 Jul 2005 13:09:00 +0300 piuparts (0.6-1) unstable; urgency=low * First release of the Debian package. -- Lars Wirzenius Tue, 5 Jul 2005 20:08:00 +0300 piuparts/debian/compat0000644000000000000000000000000212567171321012240 0ustar 9 piuparts/debian/control0000644000000000000000000000707013061276553012454 0ustar Source: piuparts Maintainer: piuparts developers team Uploaders: Holger Levsen Section: devel Priority: extra Standards-Version: 3.9.8 Build-Depends: debhelper (>= 9.20120909~), dh-python, python (>= 2.7), python-debian, python-apt, python-distro-info, python-nose, python-debianbts, python-yaml, python-mox3, python-lzma Build-Depends-Indep: asciidoc, git, xmlto Homepage: https://piuparts.debian.org Vcs-Git: https://anonscm.debian.org/git/piuparts/piuparts.git Vcs-Browser: https://anonscm.debian.org/git/piuparts/piuparts.git X-Python-Version: >= 2.7 Package: piuparts Architecture: all Pre-Depends: dpkg (>= 1.15.7.2~), Depends: piuparts-common (= ${binary:Version}), debootstrap, debsums, lsb-release, lsof, python-debian, ${misc:Depends}, ${python:Depends} Recommends: adequate Suggests: schroot Description: .deb package installation, upgrading, and removal testing tool piuparts tests that .deb packages (as used by Debian) handle installation, upgrading, and removal correctly. It does this by creating a minimal Debian installation in a chroot, and installing, upgrading, and removing packages in that environment, and comparing the state of the directory tree before and after. piuparts reports any files that have been added, removed, or modified during this process. . piuparts is meant as a quality assurance tool for people who create .deb packages to test them before they upload them to the Debian package archive. Package: piuparts-master Architecture: all Depends: piuparts-common (= ${binary:Version}), adduser, openssh-server, python-debianbts, tango-icon-theme, xz-utils, python-yaml, ${misc:Depends}, python (>= 2.7), ${python:Depends} Recommends: apache2 | httpd, ghostscript, python-rpy2, r-base-dev, r-recommended, devscripts, libsoap-lite-perl, Breaks: piuparts (<= 0.44) Replaces: piuparts (<= 0.44) Description: piuparts master components piuparts is meant as a quality assurance tool for people who create .deb packages to test them before they upload them to the Debian package archive. A master/slave piuparts installation is used for testing complete archives as it is done on https://piuparts.debian.org/. . This package provides the piuparts-master, which needs one or more slaves to operate. It also contains piuparts-report to generate web pages with the testing results as well as documentation. Package: piuparts-slave Architecture: all Depends: piuparts (= ${binary:Version}), piuparts-common (= ${binary:Version}), adduser, openssh-client, screen, sudo, ${misc:Depends}, python (>= 2.7), ${python:Depends} Suggests: schroot Breaks: piuparts (<= 0.44) Replaces: piuparts (<= 0.44) Description: piuparts slave components piuparts is meant as a quality assurance tool for people who create .deb packages to test them before they upload them to the Debian package archive. A master/slave piuparts installation is used for testing complete archives as it is done on https://piuparts.debian.org/. . This package provides the piuparts slave components, which need a piuparts master to operate. See the documentation in the piuparts-master package. Package: piuparts-common Architecture: all Depends: python-apt, python-distro-info, python-lzma, ${misc:Depends}, ${python:Depends} Breaks: piuparts (<= 0.44) Replaces: piuparts (<= 0.44) Description: common piuparts components piupartslib, common library used by piuparts-master, -report, -analyze and -slave. . See https://piuparts.debian.org or the piuparts package to learn more about piuparts. piuparts/debian/copyright0000644000000000000000000000430213036653026012773 0ustar Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: piuparts Upstream-Contact: Holger Levsen Source: https://anonscm.debian.org/cgit/piuparts/piuparts.git Copyright: 2005-2008 Lars Wirzenius 2008-2017 Holger Levsen © 2011-2017 Andreas Beckmann Comment: Original author is also Lars Wirzenius Upstream is bunch of lunatics who don't make release tarballs publicly available. The Debian .orig.tar.gz is what would be released, though. Files: * Copyright: 2005-2008 Lars Wirzenius 2008-2017 Holger Levsen © 2011-2017 Andreas Beckmann License: GPL-2+ This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. . This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. . You should have received a copy of the GNU General Public License along with this package; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA . On Debian systems, the full text of the GNU General Public License version 2 can be found in the file '/usr/share/common-licenses/GPL-2'. Files: */debian.png */openlogo-ng-50.png Copyright: 1999 Software in the Public Interest License: Free This logo or a modified version may be used by anyone to refer to the Debian project, but does not indicate endorsement by the project. . Note: we would appreciate that you make the image a link to https://www.debian.org/ if you use it on a web page. Files: */valid-html401.png */w3c-valid-css.png Copyright: 2009 W3C License: FreeUse In most cases, W3C logos and icons may be used without requesting permission from W3C. When the logo usage policy requires that you obtain express written permission, please send your request to the W3C Communications Team (w3t-pr@w3.org) piuparts/debian/piuparts-common.install0000644000000000000000000000007512567171321015571 0ustar etc/piuparts/distros.conf usr/lib/* usr/share/piuparts/lib/* piuparts/debian/piuparts-common.pyinstall0000644000000000000000000000022612567171321016140 0ustar piupartslib/__init__.py piupartslib piupartslib/conf.py piupartslib piupartslib/dependencyparser.py piupartslib piupartslib/packagesdb.py piupartslib piuparts/debian/piuparts-master.dirs0000644000000000000000000000002112567171321015056 0ustar var/lib/piuparts piuparts/debian/piuparts-master.docs0000644000000000000000000000004712567171321015055 0ustar README_server.txt README_pejacevic.txt piuparts/debian/piuparts-master.examples0000644000000000000000000000015512567171321015743 0ustar instances/piuparts.conf.anbe instances/piuparts.conf.pejacevic conf/piuparts.conf.sample conf/crontab-master piuparts/debian/piuparts-master.install0000644000000000000000000000047712567171321015602 0ustar etc/apache2/conf-available/piuparts-master.conf #etc/piuparts/known_problems/ usr/share/piuparts/htdocs/* usr/share/piuparts/master/* usr/share/piuparts/known_problems/* usr/share/piuparts/piuparts-analyze usr/share/piuparts/piuparts-master usr/share/piuparts/piuparts-master-backend usr/share/piuparts/piuparts-report piuparts/debian/piuparts-master.maintscript0000644000000000000000000000014712567171321016463 0ustar mv_conffile /etc/apache2/conf.d/piuparts.apache /etc/apache2/conf-available/piuparts-master.conf 0.58~ piuparts/debian/piuparts-master.postinst0000644000000000000000000000325312567171321016012 0ustar #!/bin/sh set -e pgroup=piuparts muser=piupartsm scriptroot=/usr/share/piuparts userhome=/var/lib/piuparts if [ "$1" = "configure" ] ; then addgroup --system --quiet $pgroup for user in $muser do adduser --system --quiet --home $userhome/$user --ingroup $pgroup \ --shell /bin/sh $user if ! [ -d $userhome/$user ]; then mkdir -m 0755 $userhome/$user chown $user:$pgroup $userhome/$user fi touch $userhome/$user/.profile chown $user:$pgroup $userhome/$user/.profile if [ $user = "piupartsm" ] ; then role="master" else role="slave" fi if ! grep -q $scriptroot/$role $userhome/$user/.profile ; then echo PATH=\$PATH:$scriptroot:$scriptroot/$role \ >>$userhome/$user/.profile fi if [ ! -d $userhome/$user/.ssh ] ; then install -d -o $user -g $pgroup -m 0755 $userhome/$user/.ssh fi done install -d -o $muser -g $pgroup -m 0755 $userhome/master install -d -o $muser -g $pgroup -m 0755 $userhome/backup install -d -o $muser -g $pgroup -m 0755 $userhome/htdocs cp -rf $scriptroot/htdocs/* $userhome/htdocs/ for icon in weather-severe-alert.png sunny.png do if [ -h $userhome/htdocs/images/$icon ] ; then rm -f $userhome/htdocs/images/$icon fi if [ -e /usr/share/icons/Tango/24x24/status/$icon ] ; then cp -f /usr/share/icons/Tango/24x24/status/$icon $userhome/htdocs/images/$icon fi done if dpkg --compare-versions "$2" lt-nl "0.50" ; then rm -rf $userhome/master/monitor-slave fi fi #DEBHELPER# piuparts/debian/piuparts-master.postrm0000644000000000000000000000075512567171321015457 0ustar #!/bin/sh set -e user=piupartsm if [ "$1" = "purge" ] ; then rm -rf /var/lib/piuparts/$user if [ -d /var/lib/piuparts/htdocs ] ; then rm -rf /var/lib/piuparts/htdocs fi if [ -d /var/lib/piuparts/backup ] ; then rm -rf /var/lib/piuparts/backup fi if [ -d /var/lib/piuparts/master ] ; then rm -rf /var/lib/piuparts/master fi rmdir /var/lib/piuparts 2>/dev/null || true rmdir /etc/piuparts 2>/dev/null || true fi #DEBHELPER# piuparts/debian/piuparts-slave.dirs0000644000000000000000000000010312567171321014676 0ustar var/lib/piuparts var/cache/piuparts/basetgz var/cache/piuparts/tmp piuparts/debian/piuparts-slave.examples0000644000000000000000000000005712567171321015563 0ustar conf/crontab-slave conf/piuparts-slave.sudoers piuparts/debian/piuparts-slave.install0000644000000000000000000000013012567171321015403 0ustar etc/piuparts/piuparts.conf usr/share/piuparts/piuparts-slave usr/share/piuparts/slave/* piuparts/debian/piuparts-slave.links0000644000000000000000000000030413061276553015063 0ustar /usr/share/piuparts/slave/slave_run /usr/bin/piuparts_slave_run /usr/share/piuparts/slave/slave_join /usr/bin/piuparts_slave_join /usr/share/piuparts/slave/slave_stop /usr/bin/piuparts_slave_stop piuparts/debian/piuparts-slave.manpages0000644000000000000000000000024413061276553015541 0ustar debian/tmp/usr/share/man/man8/piuparts_slave_run.8.gz debian/tmp/usr/share/man/man8/piuparts_slave_join.8.gz debian/tmp/usr/share/man/man8/piuparts_slave_stop.8.gz piuparts/debian/piuparts-slave.postinst0000644000000000000000000000312512567171321015627 0ustar #!/bin/sh set -e pgroup=piuparts suser=piupartss scriptroot=/usr/share/piuparts userhome=/var/lib/piuparts if [ "$1" = "configure" ] ; then addgroup --system --quiet $pgroup for user in $suser do adduser --system --quiet --home $userhome/$user --ingroup $pgroup \ --shell /bin/sh $user if ! [ -d $userhome/$user ]; then mkdir -m 0755 $userhome/$user chown $user:$pgroup $userhome/$user fi touch $userhome/$user/.profile chown $user:$pgroup $userhome/$user/.profile if [ $user = "piupartsm" ] ; then role="master" else role="slave" fi if ! grep -q $scriptroot/$role $userhome/$user/.profile ; then echo PATH=\$PATH:$scriptroot:$scriptroot/$role \ >>$userhome/$user/.profile fi if [ ! -d $userhome/$user/.ssh ] ; then install -d -o $user -g $pgroup -m 0755 $userhome/$user/.ssh fi done if [ ! -e $userhome/$suser/.ssh/id_rsa ] ; then ssh-keygen -q -f $userhome/$suser/.ssh/id_rsa -N "" chown $suser:$pgroup $userhome/$suser/.ssh/id_rsa \ $userhome/$suser/.ssh/id_rsa.pub fi if [ ! -e $userhome/$suser/.ssh/known_hosts ] ; then ssh-keyscan localhost > $userhome/$suser/.ssh/known_hosts 2>/dev/null chown $suser:$pgroup $userhome/$suser/.ssh/known_hosts fi install -d -o $suser -g $pgroup -m 0755 $userhome/slave install -d -o $suser -g $pgroup -m 0755 $userhome/slave/0 chown $suser:$pgroup /var/cache/piuparts/basetgz fi #DEBHELPER# piuparts/debian/piuparts-slave.postrm0000644000000000000000000000112012567171321015261 0ustar #!/bin/sh set -e user=piupartss if [ "$1" = "purge" ] ; then rm -rf /var/lib/piuparts/$user if [ -d /var/lib/piuparts/slave ] ; then rm -rf /var/lib/piuparts/slave fi rmdir /var/lib/piuparts 2>/dev/null || true rmdir /etc/piuparts 2>/dev/null || true rm -rf /var/cache/piuparts/basetgz/*.log rm -rf /var/cache/piuparts/basetgz/*.tar.gz # rm -rf is safe if it's not mounted anymore mount | grep -q /var/cache/piuparts/tmp || rm -rf --one-file-system /var/cache/piuparts/tmp rmdir /var/cache/piuparts 2>/dev/null || true fi #DEBHELPER# piuparts/debian/piuparts.NEWS0000644000000000000000000000402312567171321013406 0ustar piuparts (0.45) unstable; urgency=low For those interested in running their own local instance of https://piuparts.debian.org there are now piuparts-master and piuparts-slave packages for ease of installation. This is useful for both developing piuparts as well as doing QA on the Debian archive or other .deb archives. See README_server.txt and README_piatti.txt in the piuparts-master package for more information. -- Holger Levsen Mon, 18 Jun 2012 09:26:27 +0200 piuparts (0.38) unstable; urgency=low New default behaviours in piuparts: * --skip-minimize is now the default. This is because debootstrap is called with --variant=minbase instead of --resolv-deps now and because if a base.tgz or an lvm snapshot is specified, piuparts should not touch it. (Closes: #539142) Introduce a new option, --minimize, to minimize a chroot with debfoster. (This is the old default behaviour. In future versions the --minimize option might be removed again.) Add two new options: --keyring to specify the keyring to use with debootstrap (defaults to /usr/share/keyrings/debian-archive-keyring.gpg) and --do-not-verify-signatures to not use --keyring with debootstrap. (Closes: #545907) In the chroots, APT::Get::AllowUnauthenticated is set accordingly in /etc/apt/apt.conf. Add another type of custom-scripts, post_setup_, which are executed after the chroot was setup. (Closes: #466043) -- Holger Levsen Thu, 17 Dec 2009 14:10:27 +0000 piuparts (0.36) unstable; urgency=low piuparts 0.36 introduces many changes to master-slave mode, please read debian/changelog carefully. Backwards compatibility of the configuration files for master-slave-mode has been broken, as the three config files got merged into one: /etc/piuparts/piuparts.conf piuparts-reports has been re-added and improved. The documentation files have been merged and converted to asciidoc. -- Holger Levsen Tue, 10 Mar 2009 15:23:59 +0100 piuparts/debian/piuparts.docs0000644000000000000000000000006712567171321013566 0ustar README.txt README.html piuparts.1.html docbook-xsl.css piuparts/debian/piuparts.install0000644000000000000000000000005212567171321014276 0ustar /usr/sbin/piuparts /etc/piuparts/scripts* piuparts/debian/piuparts.maintscript0000644000000000000000000000112113061276553015166 0ustar rm_conffile /etc/piuparts/piuparts.conf 0.45~ rm_conffile /etc/piuparts/scripts-leftovers/post_purge_dbconfig_cleanup 0.47~ mv_conffile /etc/piuparts/scripts/post_install_find_bad_permissions /etc/piuparts/scripts/pre_remove_50_find_bad_permissions 0.47~ rm_conffile /etc/piuparts/scripts/post_remove_extras 0.61~ rm_conffile /etc/piuparts/scripts/post_distupgrade_experimental 0.74~ rm_conffile /etc/piuparts/scripts-debug-prerm/pre_remove_prerm_set-x 0.74~ rm_conffile /etc/piuparts/scripts/post_distupgrade_hack_debsums 0.75~ rm_conffile /etc/piuparts/scripts/post_setup_experimental 0.76~ piuparts/debian/piuparts.manpages0000644000000000000000000000005512567171321014426 0ustar debian/tmp/usr/share/man/man1/piuparts.1.gz piuparts/debian/piuparts.postrm0000644000000000000000000000023512567171321014157 0ustar #!/bin/sh set -e if [ "$1" = "purge" ] ; then # may be left over from a previous version rmdir /etc/piuparts 2>/dev/null || true fi #DEBHELPER# piuparts/debian/rules0000755000000000000000000000066212567171321012126 0ustar #!/usr/bin/make -f # Uncomment this to turn on verbose mode. #export DH_VERBOSE=1 %: dh $@ --with python2 override_dh_auto_build: $(MAKE) prefix=/usr build build-doc override_dh_auto_install: $(MAKE) DESTDIR=$(CURDIR)/debian/tmp prefix=/usr etcdir=/etc install install-doc install-conf override_dh_python2: dh_python2 -p piuparts-master -p piuparts-slave /usr/share/piuparts dh_python2 -N piuparts-master -N piuparts-slave piuparts/debian/source/0000755000000000000000000000000012567171321012342 5ustar piuparts/debian/source/format0000644000000000000000000000000412567171321013547 0ustar 1.0 piuparts/debian/source/options0000644000000000000000000000002212567171321013752 0ustar tar-ignore = .git piuparts/htdocs/0000755000000000000000000000000013061276553011107 5ustar piuparts/htdocs/bug_howto.tpl0000644000000000000000000001060112567171321013620 0ustar
How to file bugs based on tests run on piuparts.debian.org
This page shall grow into a well written explaination how to file useful bugs fast. It assumes you are familar with reporting bugs in Debian.
First, of all, read the piuparts logfile and identify why piuparts testing failed.
Then, check the BTS for that package, to see if this issue was already filed as a bug. Often it's also useful to check the source packages bug page. Sometimes a bug already exists, describing the problem piuparts has found. More often, new bugs have to be filed.
Usertagging existing bugs to make them known to piuparts.debian.org
If there already is a bug describing the same problem you're seeing in the piuparts logfile, you can usertag it, so that the next piuparts-analyse run will be able to link the bug report with the logfile on piuparts.debian.org. (piuparts-analyse runs twice a day.) 
 User: debian-qa@lists.debian.org
 Usertags 987654 + piuparts
Filing new bugs
More often, there is no existing bug and you need to file one. To make this easy as well to have consistent quality bug reports, we collect templates for filing these bugs. Please use these templates! The following is an example bug report for illustration: 
 To: submit@bugs.debian.org
 Subject: $package: fails to upgrade from 'testing' - trying to overwrite ...

 Package: $package
 Version: $version
 Severity: serious
 User: debian-qa@lists.debian.org
 Usertags: piuparts

 Hi,

 during a test with piuparts I noticed your package fails to upgrade from
 'testing'. It installed fine in 'testing', then the upgrade to 'sid'
 fails because it tries to overwrite other packages files without
 declaring a replaces relation.

 See policy 7.6 at
 https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces

 From the attached log (scroll to the bottom...):

 $useful_except_from_logfile

 cheers,
        $your_name

 attachment: $failed_logfile
Please take care when filing bugs to file meaningful bugs and to not annoy maintainers. Don't nitpick or insist on severities, the important thing is to get the bug fixed, not the right severity. Optionally you can also send copies to the piuparts-devel mailinglist by adding X-debbugs-cc: piuparts-devel@lists.alioth.debian.org pseudo-headers.
Also, you should be aware that what you are doing can probably be seen as mass bug filing (even if you just file a few now, they are part of a series of bugs of one kind) and as such needs to be discussed on debian-devel@lists.d.o first! For many types of bugs this has already been done. This is or should be indicated in the summary web pages as well as the mail templates.
Marking bugs as affecting other packages
Sometimes there is a bug in another package which affects a package being tested. The following explains how to tell this to the BTS in a way piuparts-analyse will pick up: 
 # assume 987654 is our bug report in buggy-package,
 # but the problem only shows up when testing (upgrades of)
 # failing-package with piuparts:
 bts affects 987654 failing-package

 # and if failing-package is from a different source with a different
 # version number:
 bts found 987654 failing-package/$FAILED_VERSION
piuparts/htdocs/favicon.ico0000644000000000000000000001114612567171321013230 0ustar  F h (>( @ʦ @ ` @@ @@@`@@@@`` `@`````` @` @` @` @`@@ @@@`@@@@@ @ @ @@ `@ @ @ @ @@@@ @@@@@`@@@@@@@@@`@` @`@@``@`@`@`@`@@ @@@`@@@@@@ @@@`@@@@@@ @@@`@@@@@@ @@@`@@@@ @` @ ` @@ @@@`@@@@`` `@`````` @` @` @` @` @` @ ` @@ @@@`@@@@`` `@`````` @` @` @`NFFNFFFNFFFFFFFNFFFFFFFWNNFFFFFFNNNFFFFNFFNFWFFFWNFFFFFNFFNFFFFFFFFFFFNFNWFNFFNFFFNFFFFWFFNFFFFNFFFFFFFFFNFFFFFFFFFFFFFNFFFFNFFFFFFFFFFFFFFFFFFFFFFFFFFNNFFNW??8?{?( @ ??|???( @ʦ @ ` @@ @@@`@@@@`` `@`````` @` @` @` @`@@ @@@`@@@@@ @ @ @@ `@ @ @ @ @@@@ @@@@@`@@@@@@@@@`@` @`@@``@`@`@`@`@@ @@@`@@@@@@ @@@`@@@@@@ @@@`@@@@@@ @@@`@@@@ @` @ ` @@ @@@`@@@@`` `@`````` @` @` @` @` @` @ ` @@ @@@`@@@@`` `@`````` @` @` @`NNNNFVFFNFFFNNFVNVNFFFNNFNFFNFVVNFVFFV?:+3J1XO(  < 3'ـ95piuparts/htdocs/images/0000755000000000000000000000000012654073103012345 5ustar piuparts/htdocs/images/debian.png0000644000000000000000000000746712567171321014317 0ustar PNG  IHDR=i pHYsHHFk>IDATx tMǟJ%!5 b-U AKi"]J4֩RGH֒-bH,HOwry^4y;sg3ej^Z*$X@_)zo+V`%ہﴒa<{̂Pb"~x")))00k׮J߷o}qq(pFFNNÇϟߪU+k6l٪ɓ͛ WZUE2^)~|ʪY1t~~mMI7{߾};22r޼yժUަM]N2eGh(dӽV={.jQ\Msc!<g%ر"d.G}dj2 p&d6m%*.E'MTT>,,F^Ph_c&޽;o:d͵!>j(u"‘l|QfB2 b)2 yv>,!cʔ)$Cf#xeD.%ԩSf9Hk;친##// '*V2?[{J +2Y| KQ\\gqJ2$C233y~7@?T`_t{Çuk >uV=z$.EVL]:u*mUVyyy'RA6lIFaa! [еozt;wGmoooggfӧ,Y2tP2S͚5}||-[v9}AAO-XI& cU.\ rrr"!Ųx@O888h6<9s|(...[l1gϺ3򷵵Ezt!<|т#Fٳgztuuu0n8tMקO'N0Zɓx`J&E2L2RV{*[l1Zd@կ__ 1&䝧Hʢ͓Jۛ ֭$}QQ}*I hFAP #,1gq˗ߵkTXx,{=h]b%lX}^dpb!Ȑ!CqFQv48Ќ 2(zUz[nHJJT*UJ 򠨁 -fbb"'l#"r=M$A&O,7VRQuvv6Ȉ48he[\h~;>Q[J& uC hx/^<|UN2 fbn7L66h```Æ %T V1}t(ƣk׮Iܹs)aw$얃O Ehd$''KA 8q=F cFF@@mX F ># %xVĤ͚5C'1? FtŴa݈:\Au'o޼wezQhJ]3f ?0b8::~(oߞ?ʍ7O!\bMGaKE ޽9ɘ3gq@<}A=6d˫`_%,] _asQϜBBx9/Cc!F=a __~Q;DM|ڵ;~80UwUF,*7(~.]شi1=Ñ=ݙ1MWd d "ؙG8]&j7|C{H2 'EơCrt #((K { CȞW8!1aE2`AD"gkԨ!f ~P}ˢ5%ܹ^Aܓ“?їHF3hKV@ƶmT!Ɔ=$ȀPreb͹HN:O= y*zf#f%bY~~~ d.bڵksGb_|d8;;+P1sd)66,$$pL,?dt &n&o|}IALϜ9f͚J 3fBzD2hǼ?-Z28w9" ECԪU+h&-Cʕ+)Əoj2E2ϵT8FV̒A۪YJh׮1EC4*U_2KZEݻ7bjU̒1vXFdgf[l_mt1tPSΆ RKF^^^Չ6mQiP1j9ӑM/2NJL^ u2ԩ#bŊ5~2 =999Ru}EmAZ.?"IF׮]i6l2}-Ed?iv,ю"nԨqd5%RjՂ3C\!$D(h{E(>O$7``o 2  9j+]s/jJNUeXOOOkӇv}E||bI2[(,?Ns…Zj1="O>G.F=Clbaa1dВA|3QFFqdߟ 44ìYߟǽ7/[={hK{I,NcD4e !QA " _ ֜9stoYbh+s)&fPڨuuu%EhΝd!7Q"~iҤ Q:vȹ.j%wy)|4eF%rww7l"Gpd@<z@#CC+++Øa%~˘[-Q 42.yN".Ϫnp%i߾͛7_2a)/U ̙30._y8F>'O?uVƵRi&))I\ HЕnݺ6˱ܹs.0]KcϏf0ho-Ji80J< '+رcĈ'h{AݻoˆE5 Ɔ,"DZ`/w5G ~Fubbb@Ұ.簰0ٯ+W|-ׯ?F݋ogD" ٤IIxo̘1V3$ @!oܸmD#<ըQ#"KIIQ%AWa ю֭=ٳV{u۷o'LLLDǎ\bRSS=xI׉k׮_v =(|FHgO/2 m[rAIENDB`piuparts/htdocs/images/openlogo-nd-50.png0000644000000000000000000000136712654073103015525 0ustar PNG  IHDR2=طV3PLTE6Cꟳ@h` O0\p揧PubKGDH pHYs  ~tIMEGJIDATxVI DŽ $6:֩)CdX|\DxК[NSi`ͳ@М;W$@!IGt#jDЮY2Îa_ΰzεc󋛭"]O~.G2SEw bcF2LdxfaP[s>0ÁI`P?ȕ *r vy)*E{SklW=ͳpY9#Tm=!4+ŋ- {娔~Lm{-0x^_GY7XQ'J`WSX)*NZm]'F86h_X T3BR XFcL?߽kT^Y?I cWe-[,:moOoeȟ#]5b@~sMIb ?7_VOp>!Od;Ӝx7j7mU:@BK[Vw 7?sj~wɻOxRE^ZħpP<l5t59XpLI ֊{ߙJTȍ4]NJr JFz|$*+lG)VzE[IW}lU! o~qn'Y%I->x4~g$gKMOUr!,ZPUפ@T+j[U1A@kEwsO?m9|[YV PIK&z*&.Rj T!lHQ il)I r@0ֲPIYWYޒGZo !kIENDB`piuparts/htdocs/index.tpl0000644000000000000000000010303613061276553012742 0ustar
About piuparts.debian.org
piuparts is a tool for testing that .deb packages can be installed, upgraded, and removed without problems. piuparts is short for "package installation, upgrading and removal testing suite" and is a variant of something suggested by Tollef Fog Heen.
It does this by creating a minimal Debian installation in a chroot, and installing, upgrading, and removing packages in that environment, and comparing the state of the directory tree before and after. piuparts reports any files that have been added, removed, or modified during this process. piuparts is meant as a quality assurance tool for people who create .deb packages to test them before they upload them to the Debian package archive.
A quick introduction is available in the piuparts README, and all the options are listed on the piuparts manpage.
To make sure piuparts is run on all packages in Debian, piuparts.debian.org has been set up to run piuparts in master/slave mode. This setup currently consists of three hosts: pejacevic.debian.org and piu-slave-bm-a.debian.org and piu-slave-ubc-01.debian.org:
  • pejacevic acts as the piuparts-master, which is responsible for scheduling test jobs to the slaves. The other main task is to generate the reports which are served via https://piuparts.debian.org.
  • piu-slave-bm-a runs four piuparts-slave instances, which then run piuparts itself.
  • piu-slave-ubc-01 also runs four piuparts-slave instances.
The first two of these hosts run as virtualized hardware on a nice cluster hosted at Bytemark and the last one runs virtualised at the University of British Columbia.
To learn more about this setup, follow the "Documentation" links in the navigation menu on the left. Read those READMEs. The piuparts configuration for all the different suite(-combination)s that are currently being tested is also linked there.
Besides all the information provided here, there is some more information available on wiki.debian.org:
You can talk to us on #debian-qa on irc.debian.org (OFTC) or send an email on the piuparts development mailinglist. The best ways to contribute are to provide patches via GIT pull requests and/or to file bugs based on piuparts runs.
These pages are updated twice a day.
Bugs submitted which were found using piuparts
News
2017-03-08 Change cron to update the webpages two times a day instead of four. In the past updating took longer than six hours, so effectivly only two updates per day were done anyway.
2017-03-07 Another suite added: wheezy222testing, to test package installation in wheezy, then upgrade to jessie, then upgrade to testing, then removal and purge..
2017-02-27 Thanks again to DSA, pejacevic.debian.org is now a quad-core system instead of the single core system it was until now.
2017-02-25 To cope with the ever increasing Debian archive, a second slave node was added to the piuparts.debian.org setup, so that now there is pejacevic, the master host, plus piu-slave-bm-a, the old node, and piu-slave-ubc-01, the new node. All these nodes still only run the amd64 architecture, help to extend the code (and the web UI) to support testing several architectures would be greatly appriciated. Many thanks to DSA for maintaining the machines piuparts.debian.org is being run on!
2016-12-20 Another suite added: sid-strict, to test packages a bit stricter than usual: first instead of a simple install, an install, followed by a remove and then install is done and then file leftover after purge are also considered an error.
2016-12-04 piuparts results have been integrated with britney, the tool the release team uses for testing migrations.
2016-06-09 Two new suites were added: jessie2bpo, to test packages upgrades from jessie to jessie-backports and jessie2bpo2stretch, where these packages are also tested for upgrading to stretch.
2015-09-29 Another new suite was added: wheezy-pu, to only test packages in wheezy-proposed-updates.
2015-04-25 With the release of Jessie two new suites are being tested: jessie2stretch and stretch, which will become the next Debian release.
2015-02-04 Link to the new Debian Package Tracker (tracker.debian.org) instead to the old Package Tracker System (PTS).
2015-01-24 Another suite was added: jessie-rcmd, to test installations in jessie with --install-recommends.
2014-12-19 Two more new suites were added: jessie-pu, to only test packages in jessie-proposed-updates and wheezy2jessie-rcmd, to test package upgrades from wheezy to jessie with --install-recommends.
2014-12-05 In preparation of the jessie release, another new suite was added: jessie2proposed, testing installation in jessie, then upgrade to jessie-proposed-upgrades, ending in purge as usual. Web pages are now updated four times a day.
2014-05-30 Results from debsums on wheezy2jessie and wheezy2bpo2jessie are not being ignored anymore as #744398 has been fixed.
2014-05-22 Add squeeze-lts to the distros being testing (by testing squeeze2squeeze-lts upgrades).
2014-05-19 Add a graph to the startpage showing the number of RC and non-RC bugs filed due to running piuparts.
2014-05-11 Temporarily ignore debsums results for wheezy2jessie and wheezy2bpo2jessie due to #744398.
2014-02-26 A new JSON summary file is being published, showing package testing state, status URL, and the number of packages being blocked by failures, for each distribution.
2013-07-16 To better track bugs in piuparts.debian.org and piuparts itself, a new pseudo-package was created in the BTS: piuparts.debian.org, which will be used for tracking all issues with the piuparts.debian.org service.
2013-06-05 In preparation of the first wheezy point release, another new suite was added: squeeze2wheezy-proposed, testing installation in squeeze, then upgrade to wheezy-proposed-upgrades, ending in purge as usual.
2013-05-30 Another new suite added: wheezy2proposed, testing installation in wheezy, then upgrade to wheezy-proposed-upgrades, ending in purge as usual.
2013-05-29 Another new suite added: squeeze2bpo-sloppy, testing the upgrade from squeeze to squeeze-backports-sloppy, ending in purge as usual.
2013-05-22 The webpages served by https://piuparts.debian.org are updated twice a day now. Further changes which were applied last week: debsums failures have been reenabled, adequate is now run by piuparts (see #703902) and two new suites were added: experimental and sid-nodoc, which tests sid without files in /usr/share/doc/<package>.
2013-05-14 Thanks to the new "hardware", piu-slave-bm-a is running four slaves now. Plus, these slaves are also considerably faster than piatti. And there are two new suites being tested: wheezy2jessie and wheezy2bpo2jessie - whoohoo!
2013-05-13 piuparts.debian.org has been moved to a new hardware and hosting location, now running virtualized on this nice cluster at Bytemark. Thanks to the Debian System Administrators for their assistence in setting up the host and maintaining the Debian infrastructure! Also many thanks and kittos to the Department of Computer Science at the University of Helsinki, Finland, for hosting piatti.debian.org since 2006 (at least)!
For maintaining this setup we used the *bikeshed* git branch.
2013-03-15 Among many other new features the 0.50 release offers much greater flexibility for configuring and selecting (partial) suites and different mirrors. Therefore it is possible to test nearly arbitrary upgrade pathes. On piuparts.debian.org this is now used for testing squeeze2bpo2wheezy and sid2experimental. Thanks to Andreas Beckmann for this great new feature!
2013-03-02 While the piuparts.git repo on Alioth will continue to be the main repo, there is also a piuparts clone on github, for those who prefer to send pull requests that way.
2012-06-21 piuparts 0.45 has been released, featuring piuparts-master and piuparts-slave packages to ease installation of such a setup. If you run piuparts in master/slave mode, please let us know.
2012-06-04 Wheezy freeze is approaching and lots of uploads happening. Old piatti hardware has problems keeping up with the pace of uploads, number of packages and distros being tested! :-) Piatti is about six years old...
2012-03-31 Disable lenny2squeeze tests, as lenny has been archived.
2012-03-05: temporarily disabled this again until we've sorted out problems with it.
2012-02-20: piuparts-analyse now sends commands the BTS: if a bug has not been explicitly marked fixed in the new version, it can rather very savely be assumed it's still present.
2012-01-30: Add new suite to be tested, testing2sid, to catch upgrade problems before they reach testing.
2012-01-22: Since some weeks, piuparts-analyse is captable of moving logfiles from fail to bugged, if there is a bug report usertagged 'piuparts' against that package+version combination. Thus, since today there is a webpage, explaining how to file bugs based on tests run on piuparts.debian.org. So now the question how to help can easily be answered: read that page and start filing bugs!
2012-01-20: As squeeze2wheezy has been fully tested by today, re-enable rescheduling of old logs for sid, wheezy and squeezewheezy: 200 successful logs older than 90 days are rescheduled each day, plus 25 failed logs older than 30 days.
2011-12-20: Currently, while the machine is busy testing all of squeeze2wheeezy, all old log rescheduling has been disabled. Normally, these reschedulings happen for sid, wheezy and squeezewheezy: 200 successful logs old than 180 days are rescheduled each day, plus 25 failed logs older than 30 days.
2011-12-10: Finally, upgrades from squeeze to wheezy are also being tested. Yay!
2011-11-21: All mails created by the piuparts master-slave setup on piatti.d.o are now sent to the piuparts-reports mailinglist on alioth. Subcribe and learn more about the details of this setup!
2011-10-31: Re-create base.tgz's every week now, as they will only be replaced if the recreation was successful.
2011-10-23: piuparts.debian.org is now maintained in git, using the piatti branch.
2011-07-10: Since today dpkg is run with --force-unsafe-io by for all suites except lenny2squeeze, as dpkg from lenny doesn't support this option.
2011-07-10: systemd-sysv is the eighth package getting special treatment by piuparts as it needs removal of sysvinit before installation and installation of that package before removal...
2011-04-02: New daily cronjob to reschedule the oldest 200 logfiles of each sid and wheezy, if they are older then 180 days. IOW: make sure no logfile for sid and wheezy is older than half a year.
2011-02-22: piatti.debian.org has been upgraded to squeeze.
2011-02-07: Add wheezy! Whoohoo!
For now, the Wheezy distribution has just been added with the same testing options as Squeeze. In future, squeeze and lenny2squeeze will not be tested anymore, and squeeze2wheezy will also be added...
2011-01-25: Reschedule 27655 successfully tested packages in Squeeze, since they were tested before the deep freeze. Yesterday all 70 failed and bugged packages were rescheduled too, which surprisingly led to 6 successful tests, followed by a few more dependent packages also being tested.
2011-01-15: Reschedule 10123 successful and failed logs in lenny2squeeze for re-testing. Those are logs which have been tested before Squeeze was deep frozen or while there was still a bug in piuparts-slave, see last news entry for details.
2011-01-03: Reschedule 12306 successful and 8 bugged logs in lenny2squeeze for re-testing. Those are logs older than 148 days, which refers to when Squeeze was initially frozen (2010-08-06). Deep freeze was announced on 2010-12-13 and there are 3800 logs older then that too, but for future deletions it's better to use 2010-01-03 (=commit r857), which fixes a bug in piuparts-slave resulting in using the sid packages file for lenny2squeeze tests.
2010-11-28: debconf-english is the seventh package getting special treatment by piuparts: before removal, debconf-i18n is installed (see #539146 has the details and the news entry for 2010-11-25 lists the other six packages.)
2010-11-26: Schedule all 159 failed packages in lenny2squeeze for re-testing.
2010-11-25: Treat six packages specially: sudo (sensibly refuses removal if no root password is set), apt-listbugs (is called by apt and exists if there are RC buggy packages being upgraded), fai-nfsroot, ltsp-client-core (these two packages modify the installed system heavily and thus will only install if conditions are met), file-rc and upstart (these two replace essential packages and therefore apt needs to be told to do this).
2010-11-24: Disable the logrotate test until #582630 is fixed and reschedule all 51 packages in sid failed due to it.
2010-11-14: Schedule all 402 failed packages in sid for re-testing.
2010-11-12: Schedule all 108 failed packages in squeeze for re-testing. (Followup on 2010-09-04.)
2010-11-06: The lists of known circular depends is now taken from http://debian.semistable.com/debgraph.out.html and maintained separately (and manually) for each tested distribution in piuparts.conf - this is not optimal (which would be piuparts detecting them automatically) but much better than the hardcoded list which we had in the piuparts library since December 2009.
2010-09-04: Schedule all 27438 passed packages in squeeze for re-testing now that squeeze is frozen.
2009-07-24: #531349 has been fixed, piuparts results are now displayed in the PTS.
2010-05-18: From today on, broken logrotate scripts after purge are only reported in sid.
2010-05-16: Finally enabled testing of sid again. (Actually, sid was enabled on 2010-03-05, but piuparts.d.o was broken until today.)
2010-02-28: Due to #571925 testing of sid had to be disabled temporarily. On an unrelated note, testing of lenny2squeeze still has some issues atm...
2010-02-25: Since yesterday, squeeze and lenny2squeeze are being tested with "--warn-on-leftovers-after-purge" making piuparts only warn about leftover files after purge. This has two effects: an decrease in the number of failed logs to process, to better focus on more important problems and second, more packages will be tested, as less packages are (seen as) buggy. Today all failed packages in squeeze and lenny2squeeze have been rescheduled for testing.
2010-02-23: Since today, piuparts is able to detect broken logrotate scripts after purge, which will need retesting of all successfully tested packages eventually. The failed packages in squeeze also needs retesting, due to split into squeeze and lenny2squeeze last week.
2010-02-16: The squeeze test has been split into squeeze and lenny2squeeze, where squeeze means package installation in squeeze, removal and purge test, while lenny2squeeze means package installation in lenny, then upgrade to squeeze, then removal and purge test. This allows more issues to be found in squeeze since (potential) brokeness in lenny is not blurring the results in squeeze.
2010-01-05: Reschedule testing for 319 failed packages in sid and 544 in squeeze, since --warn-on-others is now used.
2009-12-24: Enable work-in-progress code to enable testing of packages with circular depends. This will allow testing of 5-6000 more packages in sid and squeeze, see #526046 and the 0.39 changelog for details. The list of packages with circular depends is currently hard-coded and will probably become a configuration option but not auto detected. But that's code yet to be written :-)
2009-12-21: So testing of 13398 in squeeze has taken 12 days, which is no big surprise as the squeeze tests are more complex. Today 499 failed packages from sid and 235 from squeeze have been rescheduled for testing, to catch broken symlinks in those too.
2009-12-12: After testing 14416 packages in sid in three days, reschedule 15944 packages in squeeze... see previous entry for an explanation why.
2009-12-09: Reschedule testing for 14287 successfully tested packages in sid, those in squeeze will be rescheduled once all testable package in sid have been tested again. This is because piuparts now creates and maintains chroots securily (using gpg signed Release files for both debootstrap and apt-get) and because it warns if broken symlinks are found in a package.
2009-12-05: Reschedule testing for ~400 failed packages in sid and ~600 in squeeze, to be followed by a rescheduling of all successful packages. This is because piuparts now warns if broken symlinks are found in a package.
2009-10-08: Reschedule testing for ~2000 failed packages in sid, which failed because of a problem when minimizing the chroot at the beginning of the piuparts tests. As of today, piuparts running on piuparts.debian.org does not minimize the chroots anymore.
2009-09-18: Reschedule testing for 17170 (successfully tested) packages in sid, to make sure they still install fine with dependency based booting enabled now in sid. Throwing away 42806 (successful) logfiles from those packages :-)
2009-09-16: Reschedule testing for 233 failing packages in sid which were affected by #545949. No packages in squeeze were affected.
2009-06-20: Failed logs are not grouped into (at the moment) seven types of known errors and one type of issues is detected in successful logs.
2009-06-06: Reschedule testing for 163 successful and 27 failing packages in sid which were affected by #530501. Once openssh 1:5.1p1-6 has reached squeeze, this will be done again with 194 packages there.
2009-05-27: Throw away all failed logs as there was a bug in piuparts leading to use a more uptodate mirror for getting the list of available packages and another for doing the tests. This lead to at least one fixed package which was incorrectly tested as failing, as an old version of the package was tested. To rule out some false positives about 1000 packages will be retested, but on this machine this will only take about a day :-)
2009-05-11: Filed #528266 and made piuparts ignore files in /tmp after purge. This got rid of 20 failures in sid and 14 in squeeze.
2009-05-06: Only believe statistics you faked yourself! Up until today piuparts used to include virtual packages (those only exist true the Provides: header) into the calculations of statistics of package states and the total number of packages. Suddenly, sid has 2444 packages less!
2009-05-01: All packages in squeeze and sid which can be tested have been tested. So it takes about one month to do a full piuparts run against one suite of the archive on this machine, that's almost 1000 packages tested per day.
2009-04-20: Deleted 86 more failed logfiles (out of 692 failures in total atm) which were due to broken packages, which most likely are temporarily uninstallable issues - a good indicator for this is that all of those failures happened in sid and none in squeeze. For the future there is a cronjob now, to notify the admins daily of such problems. In more distant future those issues should be detected and avoided.
2009-04-18: Deleted all 14 failed logfiles which complained about /var/games being present after purge, as this ain't an issue, see #524461.
2009-04-04: Deleted all failed logfiles so far for two reasons: until now, only three out of ten failure types where logged with a pattern to search for in the logfiles, now this is done for all ten types of failures. And second, the way of breaking circular dependencies was not bulletproof, thus there were false positives in the failures. Now it should be fine, though maybe this will lead to lots of untestable packages... we'll see.
2009-03-19: lenny2squeeze is not needed, so all logs for squeeze (as well as lenny2squeeze) were deleted. (As squeeze now includes two kinds of tests: installation and removal in squeeze, and installation in lenny, upgrade to squeeze, removal in squeeze.)
2009-02-28: Start maintaining piatti.debian.org via the piuparts svn repository on alioth.
2007-02-24: Holger puts piuparts source in svn.
2006-10-02: #390754 O: piuparts -- package installation, upgrading and removal testing tool"
2006-09-29: Lars seeks help maintaining piuparts.
2005-07-05: #317033 ITP: piuparts -- .deb package installation, upgrading, and removal testing tool
2005-06-19: Lars writes the first blog post about piuparts (version 0.4).
piuparts/htdocs/robots.txt0000644000000000000000000000003712567171321013155 0ustar User-agent: * Disallow: /fail/ piuparts/htdocs/style.css0000644000000000000000000001156012567171321012761 0ustar #main { border: none; padding-top: 10px; padding-bottom: 10px; padding-left: 10px; padding-right: 10px; vertical-align: top; background-color: #ddd; } hr { width: 100%; color:#d70751; background-color:#d70751; height:2px; margin: 0px; padding: 0px; } em { font-weight: bold; } p.note { font-family: sans-serif; color: #900; text-align: center; padding: 5px; font-size: 11px; font-weight: normal; } div.c1 {text-align: center} p.text { font-family: sans-serif; padding: 5px; font-weight: normal; } body { padding: 0px; margin: 0px; font-family: sans-serif; font-size: 90%; color: #000000; background-color: white; } #obeytoyourfriend { color: #000; background-color: white; border: 0px solid #000; border-collapse: collapse; margin-top: 0px; margin-bottom: 5px; padding-top: 0px; padding-left: 15px; font-family: serif; font-variant:small-caps; font-size: 140%; } p.validate { text-align: center; } table { color: #000; background-color: #000; border: 0px solid #000; border-collapse: separate; border-spacing: 1px; } h1 { font-size: 140%; text-align: center; color: #000; } h2 { font-size: 120%; text-align: center; color: #000; } h3 { font-size: 110%; text-align: center; color: #000; } p { font-size: 100%; text-align: justify; } tr { background-color: #FFF; } tr.odd { background-color: #FFFFFF; } tr.even { background-color: #e8e8e8; } td.sid { color: #000; text-align: left; } tr.experimental { color: #cc0000; } tr.unstable { color: #345677; } tr.sid_odd { color: #000; } td.exp { color: #cc0000; text-align: left; } tr.exp_odd { color: #900; } th { font-size: 120%; text-align: center; font-weight: bold; background-color: #BDF; border: 0px solid #000; padding-top: 10px; padding-bottom: 10px; padding-left: 6px; padding-right: 6px; } th.reject { font-size: 120%; text-align: center; font-weight: bold; background-color: #BDF; border: 0px solid #000; padding-top: 10px; padding-bottom: 10px; padding-left: 6px; padding-right: 6px; } td { font-size: 100%; border: 0px solid #000; padding: 4px; padding-left: 6px; padding-right: 6px; } a.needs-bugging { background-color:#e0c0d0; } span.needs-bugging { background-color:#e0c0d0; } #needs-bugging { background-color:#e0c0d0; } a.bugged { background-color:#c0c0f0; } a:link { color: #0000FF; text-decoration: none; } a:visited { color: #800080; text-decoration: none; } a:active { color: #FF0000; text-decoration: none; } a:hover { color: #0000FF; text-decoration: underline; } #header { padding: 3px; } h1.header { font-size: 32px; text-align: left; margin-bottom: 0px; margin-top: 0px; padding-top: 0px; padding-bottom: 0px; padding-left: 15px; } #footer { padding: 15px; font-size: 90%; text-align: center; } /* the following layout is taken from the PTS */ /* Tables */ table.containertable { clear: both; background-color: #ddd; border: none; } table.lefttable { width: 100%; border-collapse: collapse; border: 2px solid black; background-color: white; color: black; } table.righttable { width: 100%; border-collapse: collapse; border: 2px solid black; background-color: white; color: black; } td.containercell { background-color: #ddd; } td.titlecell { color: white; background-color: #d70751; font-weight: bold; text-align: center; padding: 0.2em 0.2em 0.1em 0.2em; border-top: 3px solid #999; border-bottom: 1px solid #999; } td.alerttitlecell { color: white; background-color: #0755d7; font-weight: bold; text-align: center; padding: 0.2em 0.2em 0.1em 0.2em; border-top: 3px solid #999; border-bottom: 1px solid #999; } td.labelcell { vertical-align: top; text-align: left; padding: 0.2em 0.3em 0.2em 0.3em; border-bottom: 1px dotted #999; font-weight: bold; } td.alertlabelcell { color: white; background-color: #0755d7; vertical-align: top; text-align: left; padding: 0.2em 0.3em 0.2em 0.3em; border-bottom: 1px dotted #999; font-weight: bold; } td.lightalertlabelcell { color: white; background-color: #90c0ff; vertical-align: top; text-align: left; padding: 0.2em 0.3em 0.2em 0.3em; border-bottom: 1px dotted #999; font-weight: bold; } td.labelcell2 { padding: 0.2em 0.2em 0.1em 0.2em; border-top: 1px solid black; border-right: 1px dotted black; font-weight: bold; } td.contentcell { text-align: center; padding: 0.2em 0.3em 0.2em 0.3em; border-bottom: 1px dotted #999; } td.lastcell { font-size: 80%; text-align: center; padding: 0.2em 0.3em 0.2em 0.3em; border-bottom: 1px dotted #999; } td.contentcell2 { text-align: left; vertical-align: top; padding: 0.2em 0.3em 0.2em 0.3em; border-bottom: 1px dotted #999; border-right: 1px dotted #999; } piuparts/instances/0000755000000000000000000000000013061276553011612 5ustar piuparts/instances/Makefile0000644000000000000000000000126013061276553013251 0ustar ARCH = amd64 MIRROR = http://mirror.bm.debian.org/debian/ ARCH_piu-slave-1und1-01 = i386 MIRROR_piu-slave-1und1-01 = http://mirror.1und1.de/debian/ MIRROR_piu-slave-ubc-01 = http://mirror-ubc.debian.org/debian/ GENCONF = piuparts.conf.pejacevic GENCONF += piuparts.conf.piu-slave-bm-a GENCONF += piuparts.conf.piu-slave-1und1-01 GENCONF += piuparts.conf.piu-slave-ubc-01 all: $(GENCONF) clean: $(RM) $(GENCONF) piuparts.conf.%: piuparts.conf-template.pejacevic Makefile piuparts.conf-header.% cat piuparts.conf-header.$* > $@.tmp echo "" >> $@.tmp sed 's,@MIRROR@,$(or $(MIRROR_$*),$(MIRROR)),; s,@ARCH@,$(or $(ARCH_$*),$(ARCH)),;' < $< >> $@.tmp mv $@.tmp $@ piuparts/instances/README0000644000000000000000000000061012567171321012464 0ustar This directory contains live config files for real instances, usually with a .$HOSTNAME suffix. These are not intended for being used in the piuparts Debian packages (or only in /usr/share/doc/.../examples/). The update-piuparts-(master|slave)-setup scripts (run by piuparts(s|m)) use them when running and updating an instance directly from GIT. General config files are placed in ../conf/ piuparts/instances/piuparts.conf-header.pejacevic0000644000000000000000000000044013061276553017504 0ustar # # This is the configuration file for piuparts running in master-slave mode on # - pejacevic.debian.org (as master) and # - piu-slave-bm-a.debian.org (as slave). # # For more information on this setup see https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/README_pejacevic.txt # piuparts/instances/piuparts.conf-header.piu-slave-1und1-010000644000000000000000000000077213061276553020634 0ustar # # This is the configuration file for piuparts running in master-slave mode on # - piu-slave-1und1-01.debian.org (as slave). # Except for the architecture being tested and the mirror configuration # this file is identical to the one for pejacevic and piu-slave-bm-a. # # NOTE: However this slave does exist yet, as piuparts-master does not yet # support multiple architectures… # # For more information on this setup see https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/README_pejacevic.txt # piuparts/instances/piuparts.conf-header.piu-slave-bm-a0000777000000000000000000000000013061276553026165 2piuparts.conf-header.pejacevicustar piuparts/instances/piuparts.conf-header.piu-slave-ubc-010000644000000000000000000000054513061276553020453 0ustar # # This is the configuration file for piuparts running in master-slave mode on # - piu-slave-ubc-01.debian.org (as slave). # Except for the mirror configuration this file is identical to the one # for pejacevic and piu-slave-bm-a. # # For more information on this setup see https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/README_pejacevic.txt # piuparts/instances/piuparts.conf-template.pejacevic0000644000000000000000000003105013061276553020070 0ustar [DEFAULT] # these are needed always flags-base = # see https://bugs.debian.org/604807 --skip-logrotatefiles-test # restrict to problems in the package being tested --warn-on-others # default exceptions --scriptsdir /etc/piuparts/scripts # pejacevic's slaves are doing everything relevant on a ramdisk anyway --no-eatmydata # allow starting database servers --allow-database # default flags, only warning on leftover files flags-default = %(flags-base)s --warn-on-leftovers-after-purge # like default flags, but failing on leftover files flags-leftovers = %(flags-base)s # perform some additional cleanup --scriptsdir /etc/piuparts/scripts-leftovers # common flags for tests starting in stretch flags-start-stretch = # no flags needed # common flags for tests ending in stretch flags-end-stretch = # no flags needed # common flags for tests starting in jessie flags-start-jessie = # no flags needed # common flags for tests ending in jessie flags-end-jessie = # extra fake-essential packages for successfully purging in jessie --scriptsdir /etc/piuparts/scripts-jessie # debsums failures won't be fixed in jessie, mostly related to # obsolete/renamed conffiles that moved to different packages --warn-on-debsums-errors # won't be fixed in jessie --warn-on-install-over-symlink # common flags for tests starting in wheezy flags-start-wheezy = # no flags needed # common flags for tests ending in wheezy flags-end-wheezy = # extra fake-essential packages for successfully purging in wheezy --scriptsdir /etc/piuparts/scripts-wheezy # debsums failures won't be fixed in wheezy --warn-on-debsums-errors # won't be fixed in wheezy --warn-on-install-over-symlink # common flags for tests starting in squeeze flags-start-squeeze = # up to squeeze a non-empty /etc/shells was shipped, actually installing # and removing a shell would remove its entry from /etc/shells -i /etc/shells # squeeze has been archived --no-check-valid-until # common flags for tests ending in squeeze flags-end-squeeze = # extra fake-essential packages for successfully purging in squeeze --scriptsdir /etc/piuparts/scripts-squeeze # squeeze has been archived --warn-on-debsums-errors --warn-on-install-over-symlink # common flags for tests starting in lenny flags-start-lenny = # dpkg --force-unsafe-io was added in squeeze --dpkg-noforce-unsafe-io # same flags needed as in squeeze %(flags-start-squeeze)s [global] basetgz-sections = tarball/sid tarball/stretch tarball/jessie tarball/wheezy tarball/squeeze tarball/lenny sections = experimental sid2experimental sid sid-strict sid-nodoc testing2sid stretch stretch-rcmd stretch-security stretch-pu # stretch-proposed # stretch2proposed jessie2stretch jessie2stretch-rcmd jessie2bpo2stretch jessie2bpo jessie jessie-rcmd jessie-security jessie-pu # jessie-proposed jessie2proposed wheezy222testing wheezy2jessie wheezy2jessie-rcmd wheezy2bpo2jessie # wheezy-proposed wheezy2proposed wheezy wheezy-security wheezy-pu squeeze2wheezy-proposed squeeze2wheezy squeeze2bpo-sloppy squeeze2bpo2wheezy squeeze2squeeze-lts squeeze lenny2squeeze mirror = @MIRROR@ master-host = pejacevic.debian.org master-user = piupartsm bts-from = piuparts-devel@lists.alioth.debian.org master-command = /srv/piuparts.debian.org/share/piuparts/piuparts-master piuparts-command = sudo env PYTHONPATH=%(PYTHONPATH)s timeout -s INT -k 5m 80m /srv/piuparts.debian.org/sbin/piuparts PYTHONPATH = /srv/piuparts.debian.org/lib/python2.7/dist-packages master-directory = /srv/piuparts.debian.org/master slave-directory = /srv/piuparts.debian.org/slave basetgz-directory = /srv/piuparts.debian.org/slave/basetgz output-directory = /srv/piuparts.debian.org/htdocs backup-directory = /srv/piuparts.debian.org/backup tmpdir = /srv/piuparts.debian.org/tmp doc-root = / components = main arch = @ARCH@ area = main # the slave-count setting is for the slave(s) slave-count = 4 # 30*60 idle-sleep = 1800 max-tgz-age = 0 max-reserved = 100 # rescheduling settings expire-old-days = 120 reschedule-old-days = 90 reschedule-old-count = 250 expire-fail-days = 15 reschedule-fail-days = 10 reschedule-fail-count = 50 reschedule-untestable-days = 1 [tarball/sid] piuparts-flags = %(flags-default)s distro = None upgrade-test-distros = sid # 3 days (60*60*24*3) max-tgz-age = 259200 [tarball/stretch] piuparts-flags = %(flags-default)s %(flags-start-stretch)s distro = None upgrade-test-distros = stretch # 1 week (60*60*24*7) max-tgz-age = 604800 [tarball/jessie] piuparts-flags = %(flags-default)s %(flags-start-jessie)s distro = None upgrade-test-distros = jessie # 1 month (60*60*24*30) max-tgz-age = 2592000 [tarball/wheezy] piuparts-flags = %(flags-default)s %(flags-start-wheezy)s distro = None upgrade-test-distros = wheezy # 1 month (60*60*24*30) max-tgz-age = 2592000 [tarball/squeeze] piuparts-flags = %(flags-default)s %(flags-start-squeeze)s distro = None upgrade-test-distros = squeeze max-tgz-age = 0 [tarball/lenny] piuparts-flags = %(flags-default)s %(flags-start-lenny)s distro = None upgrade-test-distros = lenny max-tgz-age = 0 [experimental] precedence = 5 piuparts-flags = %(flags-default)s distro = experimental depends-sections = sid [sid] precedence = 1 piuparts-flags = --scriptsdir /etc/piuparts/scripts-log-alternatives %(flags-default)s distro = sid [sid-strict] precedence = 4 json-sections = none description = + Fails if there are leftover files after purge. piuparts-flags = --install-remove-install %(flags-leftovers)s # Once there are no packages left which leave files on purge behind, # --pedantic-purge-test should be added distro = sid [sid-nodoc] precedence = 100 description = + Testing without files in /usr/share/doc. piuparts-flags = %(flags-default)s --scriptsdir /etc/piuparts/scripts-no-usr-share-doc distro = sid [testing2sid] precedence = 2 piuparts-flags = %(flags-default)s distro = testing upgrade-test-distros = testing sid reschedule-old-count = 0 [sid2experimental] precedence = 4 piuparts-flags = %(flags-default)s depends-sections = sid distro = experimental upgrade-test-distros = sid experimental [stretch] precedence = 3 piuparts-flags = --scriptsdir /etc/piuparts/scripts-log-alternatives %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s distro = stretch [stretch-rcmd] precedence = 10 json-sections = none description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s distro = stretch [stretch-security] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch distro = stretch/updates [stretch-pu] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch distro = stretch-proposed-updates [stretch-proposed] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s distro = jessie-stretch [stretch2proposed] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s distro = stretch upgrade-test-distros = stretch stretch-proposed [jessie2stretch] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch-rcmd] precedence = 10 json-sections = none description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s distro = jessie upgrade-test-distros = jessie stretch [jessie2bpo2stretch] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s depends-sections = jessie distro = jessie-backports upgrade-test-distros = jessie jessie-backports stretch [jessie2bpo] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie distro = jessie-backports upgrade-test-distros = jessie jessie-backports [jessie] precedence = 3 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s distro = jessie [jessie-rcmd] precedence = 10 json-sections = none description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s distro = jessie [jessie-security] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie distro = jessie/updates [jessie-pu] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie distro = jessie-proposed-updates [jessie-proposed] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s distro = jessie-proposed [jessie2proposed] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s distro = jessie upgrade-test-distros = jessie jessie-proposed [wheezy222testing] precedence = 30 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-stretch)s distro = wheezy upgrade-test-distros = wheezy jessie stretch [wheezy2jessie] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-rcmd] precedence = 10 json-sections = none description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s -i /usr/lib/dbus-1.0/dbus-daemon-launch-helper distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2bpo2jessie] precedence = 5 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s depends-sections = wheezy distro = wheezy-backports upgrade-test-distros = wheezy wheezy-backports jessie [wheezy] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s distro = wheezy [wheezy-proposed] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s distro = wheezy-proposed [wheezy-security] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy distro = wheezy/updates [wheezy-pu] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy distro = wheezy-proposed-updates [wheezy2proposed] precedence = 3 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s distro = wheezy upgrade-test-distros = wheezy wheezy-proposed [squeeze2wheezy-proposed] precedence = 4 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s distro = squeeze upgrade-test-distros = squeeze wheezy-proposed [squeeze2wheezy] precedence = 4 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s distro = squeeze upgrade-test-distros = squeeze wheezy [squeeze2bpo2wheezy] precedence = 6 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze distro = squeeze-backports upgrade-test-distros = squeeze squeeze-backports wheezy [squeeze2bpo-sloppy] precedence = 99 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze squeeze2bpo2wheezy upgrade-test-distros = squeeze squeeze-backports-sloppy reschedule-old-count = 0 [squeeze] precedence = 99 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s distro = squeeze reschedule-old-count = 0 reschedule-fail-count = 0 [squeeze2squeeze-lts] precedence = 99 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze distro = squeeze-lts upgrade-test-distros = squeeze squeeze-lts reschedule-old-count = 0 reschedule-fail-count = 0 [lenny2squeeze] precedence = 99 json-sections = none piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-squeeze)s distro = lenny upgrade-test-distros = lenny squeeze reschedule-old-count = 0 reschedule-fail-count = 0 piuparts/instances/piuparts.conf.anbe0000644000000000000000000017571013061276553015247 0ustar # # This is the configuration file for piuparts in master-slave mode as run by AnBe. # [DEFAULT] flags-base = --skip-logrotatefiles-test --warn-on-others --no-eatmydata --scriptsdir /etc/piuparts/scripts --allow-database flags-default = %(flags-base)s --warn-on-leftovers-after-purge flags-leftovers = %(flags-base)s --scriptsdir /etc/piuparts/scripts-leftovers flags-start-testing = %(flags-start-stretch)s flags-end-testing = %(flags-end-stretch)s flags-start-stable = %(flags-start-jessie)s flags-start-stretch = flags-end-stretch = flags-start-jessie = flags-end-jessie = --warn-on-debsums-errors --warn-on-install-over-symlink --scriptsdir /etc/piuparts/scripts-jessie flags-start-wheezy = flags-end-wheezy = --warn-on-debsums-errors --warn-on-install-over-symlink --scriptsdir /etc/piuparts/scripts-wheezy flags-start-squeeze = -i /etc/shells --no-check-valid-until flags-end-squeeze = --warn-on-debsums-errors --warn-on-install-over-symlink --scriptsdir /etc/piuparts/scripts-squeeze flags-start-lenny = --dpkg-noforce-unsafe-io %(flags-start-squeeze)s flags-end-lenny = --skip-cronfiles-test %(flags-end-squeeze)s -i /etc/udev/ -i /etc/udev/rules.d/ [global] # sections may be wrapped with continuation lines being indented # and may contain non-indented comments between continuation lines basetgz-sections = tarball/sid/amd64 tarball/sid/i386 tarball/stretch/amd64 tarball/stretch/i386 tarball/jessie/amd64 tarball/jessie/i386 tarball/wheezy/amd64 tarball/wheezy/i386 tarball/squeeze/amd64 tarball/squeeze/i386 tarball/lenny/amd64 tarball/lenny/i386 sections = experimental/main experimental_i386/main sid/main sid-plus/main sid-rcmd/main sid-bl/main sid-ipi/main sid-debug/main testing-rcmd/main testing-nodocs/main stretch/main stretch-pu/main stretch-debug/main jessie/main jessie-rcmd/main jessie-proposed/main jessie-security/main jessie-updates/main jessie-pu/main jessie-backports/main wheezy/main wheezy-rcmd/main wheezy-proposed/main wheezy-security/main wheezy-updates/main wheezy-pu/main wheezy-backports/main sid2experimental/main testing2sid/main stable2sid/main stable2testing2sid/main jessie2proposed/main jessie2stretch/main jessie2stretch-rcmd_i386/main jessie2stretch-iri/main jessie2bpo2stretch/main wheezy2proposed/main wheezy2jessie/main wheezy2jessie-rcmd/main wheezy2jessie-rcmd_i386/main wheezy2jessie-sysv/main wheezy2jessie-apt1st/main wheezy2jessie-iri/main wheezy2bpo2jessie/main wheezy222testing/main squeeze222testing/main lenny222testing/main # experimental/contrib experimental_i386/contrib sid/contrib sid-plus/contrib sid-rcmd/contrib sid-bl/contrib sid-ipi/contrib sid-debug/contrib testing-rcmd/contrib testing-nodocs/contrib stretch/contrib stretch-pu/contrib stretch-debug/contrib jessie/contrib jessie-rcmd/contrib jessie-proposed/contrib jessie-security/contrib jessie-updates/contrib jessie-pu/contrib jessie-backports/contrib wheezy/contrib wheezy-rcmd/contrib wheezy-proposed/contrib # wheezy-security/contrib # wheezy-updates/contrib wheezy-pu/contrib wheezy-backports/contrib sid2experimental/contrib testing2sid/contrib stable2sid/contrib stable2testing2sid/contrib jessie2proposed/contrib jessie2stretch/contrib jessie2stretch-rcmd_i386/contrib jessie2stretch-iri/contrib jessie2bpo2stretch/contrib wheezy2proposed/contrib wheezy2jessie/contrib wheezy2jessie-rcmd/contrib wheezy2jessie-rcmd_i386/contrib wheezy2jessie-sysv/contrib wheezy2jessie-apt1st/contrib wheezy2jessie-iri/contrib wheezy2bpo2jessie/contrib wheezy222testing/contrib squeeze222testing/contrib lenny222testing/contrib # experimental/non-free experimental_i386/non-free sid/non-free sid-plus/non-free sid-rcmd/non-free sid-bl/non-free sid-ipi/non-free sid-debug/non-free testing-rcmd/non-free testing-nodocs/non-free stretch/non-free stretch-pu/non-free stretch-debug/non-free jessie/non-free jessie-rcmd/non-free jessie-proposed/non-free jessie-security/non-free jessie-updates/non-free jessie-pu/non-free jessie-backports/non-free wheezy/non-free wheezy-rcmd/non-free wheezy-proposed/non-free # wheezy-security/non-free # wheezy-updates/non-free wheezy-pu/non-free wheezy-backports/non-free sid2experimental/non-free testing2sid/non-free stable2sid/non-free stable2testing2sid/non-free jessie2proposed/non-free jessie2stretch/non-free jessie2stretch-rcmd_i386/non-free jessie2stretch-iri/non-free jessie2bpo2stretch/non-free wheezy2proposed/non-free wheezy2jessie/non-free wheezy2jessie-rcmd/non-free wheezy2jessie-rcmd_i386/non-free wheezy2jessie-sysv/non-free wheezy2jessie-apt1st/non-free wheezy2jessie-iri/non-free wheezy2bpo2jessie/non-free wheezy222testing/non-free squeeze222testing/non-free lenny222testing/non-free # squeeze/main squeeze-lts/main squeeze-backports/main squeeze-backports-sloppy/main squeeze2lts2wheezy/main squeeze2wheezy/main squeeze2wheezy_i386/main squeeze2wheezy-rcmd/main squeeze2wheezy-rcmd_i386/main squeeze2wheezy-pu/main squeeze2wheezy-proposed/main squeeze2bpo2wheezy/main lenny/main lenny2squeeze/main # squeeze/contrib squeeze-lts/contrib squeeze-backports/contrib squeeze-backports-sloppy/contrib squeeze2lts2wheezy/contrib squeeze2wheezy/contrib squeeze2wheezy_i386/contrib squeeze2wheezy-rcmd/contrib squeeze2wheezy-rcmd_i386/contrib squeeze2wheezy-pu/contrib squeeze2wheezy-proposed/contrib squeeze2bpo2wheezy/contrib lenny/contrib lenny2squeeze/contrib # squeeze/non-free squeeze-lts/non-free squeeze-backports/non-free squeeze-backports-sloppy/non-free squeeze2lts2wheezy/non-free squeeze2wheezy/non-free squeeze2wheezy_i386/non-free squeeze2wheezy-rcmd/non-free squeeze2wheezy-rcmd_i386/non-free squeeze2wheezy-pu/non-free squeeze2wheezy-proposed/non-free squeeze2bpo2wheezy/non-free lenny/non-free lenny2squeeze/non-free # json-sections = none # proxy = http://localhost:3128 # mirror = http://ftp.de.debian.org/debian # master-host = localhost # master-user = piupartsm master-command = /srv/piuparts/share/piuparts/piuparts-master piuparts-command = sudo nice env PYTHONPATH=%(PYTHONPATH)s timeout -s INT -k 5m 110m /srv/piuparts/sbin/piuparts PYTHONPATH = /srv/piuparts/lib/python2.7/dist-packages master-directory = /srv/piuparts/master slave-directory = /srv/piuparts/slave basetgz-directory = /srv/piuparts/slave/basetgz output-directory = /srv/piuparts/htdocs backup-directory = /srv/piuparts/backup tmpdir = /tmp/piupartss slave-load-max = 20.5 idle-sleep = 1800 max-tgz-age = -1 max-reserved = 50 reschedule-untestable-days = 3 expire-old-days = 135 reschedule-old-days = 120 reschedule-old-count = 250 expire-fail-days = 8 reschedule-fail-days = 5 reschedule-fail-count = 100 ############################################################################ ### Tarball creation. ### ############################################################################ [tarball/sid/amd64] piuparts-flags = %(flags-default)s distro = None upgrade-test-distros = sid arch = amd64 area = main components = main # 60*60*24*2 max-tgz-age = 172800 [tarball/sid/i386] piuparts-flags = %(flags-default)s distro = None upgrade-test-distros = sid arch = i386 area = main components = main # 60*60*24*2 max-tgz-age = 172800 [tarball/stretch/amd64] piuparts-flags = %(flags-default)s %(flags-start-stretch)s distro = None upgrade-test-distros = stretch arch = amd64 area = main components = main # 60*60*24*7 max-tgz-age = 604800 [tarball/stretch/i386] piuparts-flags = %(flags-default)s %(flags-start-stretch)s distro = None upgrade-test-distros = stretch arch = i386 area = main components = main # 60*60*24*7 max-tgz-age = 604800 [tarball/jessie/amd64] piuparts-flags = %(flags-default)s %(flags-start-jessie)s distro = None upgrade-test-distros = jessie arch = amd64 area = main components = main # 60*60*24*30 max-tgz-age = 2592000 [tarball/jessie/i386] piuparts-flags = %(flags-default)s %(flags-start-jessie)s distro = None upgrade-test-distros = jessie arch = i386 area = main components = main # 60*60*24*30 max-tgz-age = 2592000 [tarball/wheezy/amd64] piuparts-flags = %(flags-default)s %(flags-start-wheezy)s distro = None upgrade-test-distros = wheezy arch = amd64 area = main components = main # 60*60*24*30 max-tgz-age = 2592000 [tarball/wheezy/i386] piuparts-flags = %(flags-default)s %(flags-start-wheezy)s distro = None upgrade-test-distros = wheezy arch = i386 area = main components = main # 60*60*24*30 max-tgz-age = 2592000 [tarball/squeeze/amd64] piuparts-flags = %(flags-default)s %(flags-start-squeeze)s distro = None upgrade-test-distros = squeeze arch = amd64 area = main components = main max-tgz-age = 0 [tarball/squeeze/i386] piuparts-flags = %(flags-default)s %(flags-start-squeeze)s distro = None upgrade-test-distros = squeeze arch = i386 area = main components = main max-tgz-age = 0 [tarball/lenny/amd64] piuparts-flags = %(flags-default)s %(flags-start-lenny)s distro = None upgrade-test-distros = lenny arch = amd64 area = main components = main max-tgz-age = 0 [tarball/lenny/i386] piuparts-flags = %(flags-default)s %(flags-start-lenny)s distro = None upgrade-test-distros = lenny arch = i386 area = main components = main max-tgz-age = 0 ############################################################################ ### Install, remove, and purge tests. ### ############################################################################ [experimental/main] precedence = 40 piuparts-flags = %(flags-default)s depends-sections = sid/main distro = experimental arch = amd64 area = main components = main reschedule-old-days = 30 expire-old-days = 45 [experimental/contrib] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = experimental/main sid/main sid/contrib sid/non-free distro = experimental arch = amd64 area = contrib reschedule-old-days = 30 expire-old-days = 45 [experimental/non-free] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = experimental/main sid/main sid/contrib sid/non-free distro = experimental arch = amd64 area = non-free reschedule-old-days = 30 expire-old-days = 45 [experimental_i386/main] precedence = 40 piuparts-flags = %(flags-default)s depends-sections = sid-rcmd/main distro = experimental arch = i386 area = main components = main reschedule-old-days = 30 expire-old-days = 45 [experimental_i386/contrib] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = experimental_i386/main sid-rcmd/main sid-rcmd/contrib sid-rcmd/non-free distro = experimental arch = i386 area = contrib reschedule-old-days = 30 expire-old-days = 45 [experimental_i386/non-free] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = experimental_i386/main sid-rcmd/main sid-rcmd/contrib sid-rcmd/non-free distro = experimental arch = i386 area = non-free reschedule-old-days = 30 expire-old-days = 45 [sid/main] precedence = 10 piuparts-flags = %(flags-default)s distro = sid arch = amd64 area = main components = main reschedule-old-days = 30 expire-old-days = 45 [sid/contrib] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = sid/main sid/non-free distro = sid arch = amd64 area = contrib reschedule-old-days = 30 expire-old-days = 45 [sid/non-free] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = sid/main sid/contrib distro = sid arch = amd64 area = non-free reschedule-old-days = 30 expire-old-days = 45 [sid-plus/main] precedence = 30 description = Debian %(distro)s / %(area)s: package installation, removal, installation, removal, and purge test. Failing on leftovers after purge. piuparts-flags = --install-remove-install %(flags-leftovers)s distro = sid arch = amd64 area = main components = main reschedule-old-days = 60 expire-old-days = 75 [sid-plus/contrib] precedence = 45 description = Debian %(distro)s / %(area)s: package installation, removal, installation, removal, and purge test. Failing on leftovers after purge. piuparts-flags = --install-remove-install %(flags-leftovers)s depends-sections = sid-plus/main sid-plus/non-free distro = sid arch = amd64 area = contrib reschedule-old-days = 60 expire-old-days = 75 [sid-plus/non-free] precedence = 45 description = Debian %(distro)s / %(area)s: package installation, removal, installation, removal, and purge test. Failing on leftovers after purge. piuparts-flags = --install-remove-install %(flags-leftovers)s depends-sections = sid-plus/main sid-plus/contrib distro = sid arch = amd64 area = non-free reschedule-old-days = 60 expire-old-days = 75 [sid-rcmd/main] precedence = 30 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s distro = sid arch = i386 area = main components = main reschedule-old-days = 60 expire-old-days = 75 [sid-rcmd/contrib] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s depends-sections = sid-rcmd/main sid-rcmd/non-free distro = sid arch = i386 area = contrib reschedule-old-days = 60 expire-old-days = 75 [sid-rcmd/non-free] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s depends-sections = sid-rcmd/main sid-rcmd/contrib distro = sid arch = i386 area = non-free reschedule-old-days = 60 expire-old-days = 75 [sid-bl/main] precedence = 130 description = + Failing on broken symlinks. piuparts-flags = %(flags-default)s --fail-on-broken-symlinks --install-recommends --fake-essential-packages libjs-sphinxdoc # ignore broken symlinks -i :/etc/systemd/system/display-manager.service -i :/usr/share/gocode/src/github.com/prometheus/procfs/fixtures/26231/exe -i :/usr/share/qt4/plugins -i :/usr/share/quilt/compat/sendmail # ignore broken symlinks (patterns) -I /usr/include/python2\.7_d/.*\.h -I /usr/include/python3\..dm/.*\.h -I :/usr/lib/jvm/java-.*-openjdk-.*/src\.zip -I :/usr/share/doc/kde/HTML/.*/common -I :/usr/share/man/man1/(.*-)?python.*-config\.1\.gz distro = sid arch = amd64 area = main components = main [sid-bl/contrib] precedence = 130 description = + Failing on broken symlinks. piuparts-flags = %(flags-default)s --fail-on-broken-symlinks --install-recommends # ignore broken symlinks (patterns) -I /usr/include/python2\.7_d/.*\.h -I /usr/include/python3\..dm/.*\.h -I :/usr/share/man/man1/(.*-)?python.*-config\.1\.gz depends-sections = sid-bl/main sid-bl/non-free distro = sid arch = amd64 area = contrib [sid-bl/non-free] precedence = 130 description = + Failing on broken symlinks. piuparts-flags = %(flags-default)s --fail-on-broken-symlinks --install-recommends # ignore broken symlinks (patterns) -I /usr/include/python2\.7_d/.*\.h -I /usr/include/python3\..dm/.*\.h -I :/usr/share/man/man1/(.*-)?python.*-config\.1\.gz depends-sections = sid-bl/main sid-bl/contrib distro = sid arch = amd64 area = non-free [sid-debug/main] precedence = 40 piuparts-flags = %(flags-default)s depends-sections = sid/main distro = sid-debug arch = amd64 area = main components = main reschedule-old-count = 0 [sid-debug/contrib] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = sid/main sid/contrib sid/non-free distro = sid-debug arch = amd64 area = contrib reschedule-old-count = 0 [sid-debug/non-free] precedence = 45 piuparts-flags = %(flags-default)s depends-sections = sid/main sid/contrib sid/non-free distro = sid-debug arch = amd64 area = non-free reschedule-old-count = 0 [testing-rcmd/main] precedence = 30 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-testing)s %(flags-end-testing)s distro = testing arch = i386 area = main components = main [testing-rcmd/contrib] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-testing)s %(flags-end-testing)s depends-sections = testing-rcmd/main testing-rcmd/non-free distro = testing arch = i386 area = contrib [testing-rcmd/non-free] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-testing)s %(flags-end-testing)s depends-sections = testing-rcmd/main testing-rcmd/contrib distro = testing arch = i386 area = non-free [testing-nodocs/main] precedence = 30 description = + Testing without files in /usr/share/doc. piuparts-flags = %(flags-default)s %(flags-start-testing)s %(flags-end-testing)s --scriptsdir /etc/piuparts/scripts-no-usr-share-doc distro = testing arch = i386 area = main components = main [testing-nodocs/contrib] precedence = 45 description = + Testing without files in /usr/share/doc. piuparts-flags = %(flags-default)s %(flags-start-testing)s %(flags-end-testing)s --scriptsdir /etc/piuparts/scripts-no-usr-share-doc depends-sections = testing-nodocs/main testing-nodocs/non-free distro = testing arch = i386 area = contrib [testing-nodocs/non-free] precedence = 45 description = + Testing without files in /usr/share/doc. piuparts-flags = %(flags-default)s %(flags-start-testing)s %(flags-end-testing)s --scriptsdir /etc/piuparts/scripts-no-usr-share-doc depends-sections = testing-nodocs/main testing-nodocs/contrib distro = testing arch = i386 area = non-free [stretch/main] precedence = 10 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s distro = stretch arch = amd64 area = main components = main reschedule-old-days = 60 expire-old-days = 75 [stretch/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main stretch/non-free distro = stretch arch = amd64 area = contrib reschedule-old-days = 60 expire-old-days = 75 [stretch/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main stretch/contrib distro = stretch arch = amd64 area = non-free reschedule-old-days = 60 expire-old-days = 75 [stretch-pu/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main distro = stretch-proposed-updates arch = amd64 area = main components = main [stretch-pu/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main stretch/non-free distro = stretch-proposed-updates arch = amd64 area = contrib [stretch-pu/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main stretch/contrib distro = stretch-proposed-updates arch = amd64 area = non-free [stretch-debug/main] precedence = 40 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main distro = stretch-debug arch = amd64 area = main components = main reschedule-old-count = 0 [stretch-debug/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main stretch/contrib stretch/non-free distro = stretch-debug arch = amd64 area = contrib reschedule-old-count = 0 [stretch-debug/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-stretch)s %(flags-end-stretch)s depends-sections = stretch/main stretch/contrib stretch/non-free distro = stretch-debug arch = amd64 area = non-free reschedule-old-count = 0 [jessie/main] precedence = 10 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s distro = jessie arch = amd64 area = main components = main reschedule-old-count = 100 [jessie/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main jessie/non-free distro = jessie arch = amd64 area = contrib [jessie/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main jessie/contrib distro = jessie arch = amd64 area = non-free [jessie-rcmd/main] precedence = 40 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s distro = jessie arch = i386 area = main components = main reschedule-old-count = 100 [jessie-rcmd/contrib] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie-rcmd/main jessie-rcmd/non-free distro = jessie arch = i386 area = contrib [jessie-rcmd/non-free] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie-rcmd/main jessie-rcmd/contrib distro = jessie arch = i386 area = non-free [jessie-security/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie/updates arch = amd64 area = main components = main [jessie-security/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie/updates arch = amd64 area = contrib [jessie-security/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie/updates arch = amd64 area = non-free [jessie-updates/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie-updates arch = amd64 area = main components = main [jessie-updates/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie-updates arch = amd64 area = contrib [jessie-updates/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie-updates arch = amd64 area = non-free [jessie-pu/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie-proposed-updates arch = amd64 area = main components = main [jessie-pu/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main jessie/non-free distro = jessie-proposed-updates arch = amd64 area = contrib [jessie-pu/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main jessie/contrib distro = jessie-proposed-updates arch = amd64 area = non-free [jessie-proposed/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s distro = jessie-proposed arch = amd64 area = main components = main reschedule-old-count = 500 reschedule-old-days = 25 [jessie-proposed/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie-proposed/main jessie-proposed/non-free distro = jessie-proposed arch = amd64 area = contrib reschedule-old-days = 25 [jessie-proposed/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie-proposed/main jessie-proposed/contrib distro = jessie-proposed arch = amd64 area = non-free reschedule-old-days = 25 [jessie-backports/main] precedence = 40 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main distro = jessie-backports arch = amd64 area = main components = main [jessie-backports/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main jessie/contrib jessie/non-free jessie-backports/main jessie-backports/non-free distro = jessie-backports arch = amd64 area = contrib [jessie-backports/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie/main jessie/contrib jessie/non-free jessie-backports/main jessie-backports/contrib distro = jessie-backports arch = amd64 area = non-free [wheezy/main] precedence = 10 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s distro = wheezy arch = amd64 area = main components = main reschedule-old-count = 100 [wheezy/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main wheezy/non-free distro = wheezy arch = amd64 area = contrib [wheezy/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main wheezy/contrib distro = wheezy arch = amd64 area = non-free [wheezy-rcmd/main] precedence = 40 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s distro = wheezy arch = i386 area = main components = main reschedule-old-count = 100 [wheezy-rcmd/contrib] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy-rcmd/main wheezy-rcmd/non-free distro = wheezy arch = i386 area = contrib [wheezy-rcmd/non-free] precedence = 45 description = + With recommended packages. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy-rcmd/main wheezy-rcmd/contrib distro = wheezy arch = i386 area = non-free [wheezy-security/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy/updates arch = amd64 area = main components = main [wheezy-security/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy/updates arch = amd64 area = contrib [wheezy-security/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy/updates arch = amd64 area = non-free [wheezy-updates/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy-updates arch = amd64 area = main components = main [wheezy-updates/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy-updates arch = amd64 area = contrib [wheezy-updates/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy-updates arch = amd64 area = non-free [wheezy-pu/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy-proposed-updates arch = amd64 area = main components = main [wheezy-pu/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main wheezy/non-free distro = wheezy-proposed-updates arch = amd64 area = contrib [wheezy-pu/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main wheezy/contrib distro = wheezy-proposed-updates arch = amd64 area = non-free [wheezy-proposed/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s distro = wheezy-proposed arch = amd64 area = main components = main reschedule-old-count = 333 reschedule-old-days = 25 [wheezy-proposed/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy-proposed/main wheezy-proposed/non-free distro = wheezy-proposed arch = amd64 area = contrib reschedule-old-days = 25 [wheezy-proposed/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy-proposed/main wheezy-proposed/contrib distro = wheezy-proposed arch = amd64 area = non-free reschedule-old-days = 25 [wheezy-backports/main] precedence = 40 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main distro = wheezy-backports arch = amd64 area = main components = main [wheezy-backports/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main wheezy/non-free wheezy-backports/main wheezy-backports/non-free distro = wheezy-backports arch = amd64 area = contrib [wheezy-backports/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy/main wheezy-backports/main wheezy-backports/contrib distro = wheezy-backports arch = amd64 area = non-free [squeeze/main] precedence = 10 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s distro = squeeze arch = amd64 area = main components = main reschedule-old-count = 0 [squeeze/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze/non-free distro = squeeze arch = amd64 area = contrib reschedule-old-count = 0 [squeeze/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze/contrib distro = squeeze arch = amd64 area = non-free reschedule-old-count = 0 [squeeze-lts/main] precedence = 20 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main distro = squeeze-lts arch = amd64 area = main components = main reschedule-old-count = 0 [squeeze-lts/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze/non-free distro = squeeze-lts arch = amd64 area = contrib reschedule-old-count = 0 [squeeze-lts/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze/contrib distro = squeeze-lts arch = amd64 area = non-free reschedule-old-count = 0 [squeeze-backports/main] precedence = 40 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main distro = squeeze-backports arch = amd64 area = main components = main reschedule-old-count = 0 [squeeze-backports/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze-backports/main squeeze-backports/non-free distro = squeeze-backports arch = amd64 area = contrib reschedule-old-count = 0 [squeeze-backports/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze-backports/main squeeze-backports/contrib distro = squeeze-backports arch = amd64 area = non-free reschedule-old-count = 0 [squeeze-backports-sloppy/main] precedence = 40 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze-backports/main distro = squeeze-backports-sloppy arch = amd64 area = main components = main reschedule-old-count = 0 [squeeze-backports-sloppy/contrib] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze-backports/main squeeze-backports/non-free distro = squeeze-backports-sloppy arch = amd64 area = contrib reschedule-old-count = 0 [squeeze-backports-sloppy/non-free] precedence = 45 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-squeeze)s depends-sections = squeeze/main squeeze-backports/main squeeze-backports/contrib distro = squeeze-backports-sloppy arch = amd64 area = non-free reschedule-old-count = 0 [lenny/main] precedence = 110 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-lenny)s distro = lenny arch = amd64 area = main components = main reschedule-old-count = 0 [lenny/contrib] precedence = 110 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-lenny)s depends-sections = lenny/main lenny/non-free distro = lenny arch = amd64 area = contrib reschedule-old-count = 0 [lenny/non-free] precedence = 110 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-lenny)s depends-sections = lenny/main lenny/contrib distro = lenny arch = amd64 area = non-free reschedule-old-count = 0 ############################################################################ ### Install, remove, purge, install, remove, and purge tests. ### ############################################################################ [sid-ipi/main] precedence = 175 description = Debian %(distro)s / %(area)s: package installation, purge, installation, removal, and purge test. piuparts-flags = --install-purge-install --warn-on-leftovers-after-purge -i /var/lib/ghc/package.conf.d/package.cache %(flags-leftovers)s distro = sid arch = amd64 area = main components = main reschedule-old-count = 100 [sid-ipi/contrib] precedence = 175 description = Debian %(distro)s / %(area)s: package installation, purge, installation, removal, and purge test. piuparts-flags = --install-purge-install --warn-on-leftovers-after-purge -i /var/lib/ghc/package.conf.d/package.cache %(flags-leftovers)s depends-sections = sid-ipi/main sid-ipi/non-free distro = sid arch = amd64 area = contrib [sid-ipi/non-free] precedence = 175 description = Debian %(distro)s / %(area)s: package installation, purge, installation, removal, and purge test. piuparts-flags = --install-purge-install --warn-on-leftovers-after-purge -i /var/lib/ghc/package.conf.d/package.cache %(flags-leftovers)s depends-sections = sid-ipi/main sid-ipi/contrib distro = sid arch = amd64 area = non-free ############################################################################ ### Install, distupgrade, remove, and purge tests. ### ############################################################################ [lenny2squeeze/main] precedence = 110 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-squeeze)s -i /usr/local/share/octave/ -i /usr/local/share/octave/site-m/ -i /usr/share/octave/ls-R -i /usr/share/octave/packages/ arch = amd64 area = main components = main distro = lenny upgrade-test-distros = lenny squeeze reschedule-old-count = 0 [lenny2squeeze/contrib] precedence = 110 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-squeeze)s depends-sections = lenny2squeeze/main lenny2squeeze/non-free arch = amd64 area = contrib distro = lenny upgrade-test-distros = lenny squeeze reschedule-old-count = 0 [lenny2squeeze/non-free] precedence = 110 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-squeeze)s depends-sections = lenny2squeeze/main lenny2squeeze/contrib arch = amd64 area = non-free distro = lenny upgrade-test-distros = lenny squeeze reschedule-old-count = 0 [lenny222testing/main] precedence = 120 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-testing)s arch = amd64 area = main components = main distro = lenny upgrade-test-distros = lenny squeeze wheezy jessie testing reschedule-old-count = 100 expire-fail-days = 20 reschedule-fail-days = 12 [lenny222testing/contrib] precedence = 120 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-testing)s depends-sections = lenny222testing/main lenny222testing/non-free arch = amd64 area = contrib distro = lenny upgrade-test-distros = lenny squeeze wheezy jessie testing reschedule-old-count = 100 [lenny222testing/non-free] precedence = 120 piuparts-flags = %(flags-default)s %(flags-start-lenny)s %(flags-end-testing)s depends-sections = lenny222testing/main lenny222testing/contrib arch = amd64 area = non-free distro = lenny upgrade-test-distros = lenny squeeze wheezy jessie testing reschedule-old-count = 100 [squeeze2lts2wheezy/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = squeeze/main distro = squeeze-lts arch = i386 area = main components = main upgrade-test-distros = squeeze squeeze-lts wheezy reschedule-old-count = 0 [squeeze2lts2wheezy/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = squeeze/main squeeze2lts2wheezy/main squeeze2lts2wheezy/non-free distro = squeeze-lts arch = i386 area = contrib upgrade-test-distros = squeeze squeeze-lts wheezy reschedule-old-count = 0 [squeeze2lts2wheezy/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = squeeze/main squeeze2lts2wheezy/main squeeze2lts2wheezy/contrib distro = squeeze-lts arch = i386 area = non-free upgrade-test-distros = squeeze squeeze-lts wheezy reschedule-old-count = 0 [squeeze2wheezy/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = amd64 area = main components = main distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 100 [squeeze2wheezy/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy/main squeeze2wheezy/non-free arch = amd64 area = contrib distro = squeeze upgrade-test-distros = squeeze wheezy [squeeze2wheezy/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy/main squeeze2wheezy/contrib arch = amd64 area = non-free distro = squeeze upgrade-test-distros = squeeze wheezy [squeeze2wheezy-proposed/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = amd64 area = main components = main distro = squeeze upgrade-test-distros = squeeze wheezy-proposed reschedule-old-count = 100 [squeeze2wheezy-proposed/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy-proposed/main squeeze2wheezy-proposed/non-free arch = amd64 area = contrib distro = squeeze upgrade-test-distros = squeeze wheezy-proposed [squeeze2wheezy-proposed/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy-proposed/main squeeze2wheezy-proposed/contrib arch = amd64 area = non-free distro = squeeze upgrade-test-distros = squeeze wheezy-proposed [squeeze2wheezy_i386/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = i386 area = main components = main distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy_i386/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy_i386/main squeeze2wheezy_i386/non-free arch = i386 area = contrib distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy_i386/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy_i386/main squeeze2wheezy_i386/contrib arch = i386 area = non-free distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy-rcmd/main] precedence = 70 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = amd64 area = main components = main distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy-rcmd/contrib] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy-rcmd/main squeeze2wheezy-rcmd/non-free arch = amd64 area = contrib distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy-rcmd/non-free] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy-rcmd/main squeeze2wheezy-rcmd/contrib arch = amd64 area = non-free distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy-rcmd_i386/main] precedence = 70 piuparts-flags = --install-recommends %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = i386 area = main components = main distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy-rcmd_i386/contrib] precedence = 95 piuparts-flags = --install-recommends %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy-rcmd_i386/main squeeze2wheezy-rcmd_i386/non-free arch = i386 area = contrib distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy-rcmd_i386/non-free] precedence = 95 piuparts-flags = --install-recommends %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy-rcmd_i386/main squeeze2wheezy-rcmd_i386/contrib arch = i386 area = non-free distro = squeeze upgrade-test-distros = squeeze wheezy reschedule-old-count = 0 [squeeze2wheezy-pu/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy/main wheezy/main arch = amd64 area = main components = main upgrade-test-distros = squeeze wheezy-proposed-updates reschedule-old-days = 30 expire-old-days = 45 [squeeze2wheezy-pu/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy/main squeeze2wheezy/non-free wheezy/main wheezy/contrib wheezy/non-free arch = amd64 area = contrib upgrade-test-distros = squeeze wheezy-proposed-updates reschedule-old-days = 30 expire-old-days = 45 [squeeze2wheezy-pu/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s depends-sections = squeeze2wheezy/main squeeze2wheezy/contrib wheezy/main wheezy/contrib wheezy/non-free arch = amd64 area = non-free upgrade-test-distros = squeeze wheezy-proposed-updates reschedule-old-days = 30 expire-old-days = 45 [squeeze2bpo2wheezy/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = squeeze/main distro = squeeze-backports arch = amd64 area = main components = main upgrade-test-distros = squeeze squeeze-backports wheezy [squeeze2bpo2wheezy/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = squeeze/main squeeze2bpo2wheezy/main squeeze2bpo2wheezy/non-free distro = squeeze-backports arch = amd64 area = contrib upgrade-test-distros = squeeze squeeze-backports wheezy [squeeze2bpo2wheezy/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = squeeze/main squeeze2bpo2wheezy/main squeeze2bpo2wheezy/contrib distro = squeeze-backports arch = amd64 area = non-free upgrade-test-distros = squeeze squeeze-backports wheezy [squeeze222testing/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-testing)s arch = amd64 area = main components = main distro = squeeze upgrade-test-distros = squeeze wheezy jessie testing reschedule-old-count = 100 [squeeze222testing/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-testing)s depends-sections = squeeze222testing/main squeeze222testing/non-free arch = amd64 area = contrib distro = squeeze upgrade-test-distros = squeeze wheezy jessie testing reschedule-old-count = 100 [squeeze222testing/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-squeeze)s %(flags-end-testing)s depends-sections = squeeze222testing/main squeeze222testing/contrib arch = amd64 area = non-free distro = squeeze upgrade-test-distros = squeeze wheezy jessie testing reschedule-old-count = 100 [wheezy2proposed/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = amd64 area = main components = main distro = wheezy upgrade-test-distros = wheezy wheezy-proposed [wheezy2proposed/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy2proposed/main wheezy2proposed/non-free arch = amd64 area = contrib distro = wheezy upgrade-test-distros = wheezy wheezy-proposed [wheezy2proposed/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-wheezy)s depends-sections = wheezy2proposed/main wheezy2proposed/contrib arch = amd64 area = non-free distro = wheezy upgrade-test-distros = wheezy wheezy-proposed [wheezy2jessie/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = amd64 area = main components = main distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = wheezy2jessie/main wheezy2jessie/non-free arch = amd64 area = contrib distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = wheezy2jessie/main wheezy2jessie/contrib arch = amd64 area = non-free distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-rcmd/main] precedence = 80 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver -i /usr/lib/dbus-1.0/dbus-daemon-launch-helper arch = amd64 area = main components = main distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-rcmd/contrib] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver -i /usr/lib/dbus-1.0/dbus-daemon-launch-helper depends-sections = wheezy2jessie-rcmd/main wheezy2jessie-rcmd/non-free arch = amd64 area = contrib distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-rcmd/non-free] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver -i /usr/lib/dbus-1.0/dbus-daemon-launch-helper depends-sections = wheezy2jessie-rcmd/main wheezy2jessie-rcmd/contrib arch = amd64 area = non-free distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-rcmd_i386/main] precedence = 80 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver -i /usr/lib/dbus-1.0/dbus-daemon-launch-helper arch = i386 area = main components = main distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-rcmd_i386/contrib] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver -i /usr/lib/dbus-1.0/dbus-daemon-launch-helper depends-sections = wheezy2jessie-rcmd_i386/main wheezy2jessie-rcmd_i386/non-free arch = i386 area = contrib distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-rcmd_i386/non-free] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver -i /usr/lib/dbus-1.0/dbus-daemon-launch-helper depends-sections = wheezy2jessie-rcmd_i386/main wheezy2jessie-rcmd_i386/contrib arch = i386 area = non-free distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-sysv/main] precedence = 80 description = + Keeping sysvinit instead of switching to systemd (if possible). piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver --scriptsdir /etc/piuparts/scripts-sysvinit arch = amd64 area = main components = main distro = wheezy upgrade-test-distros = wheezy jessie reschedule-old-count = 0 [wheezy2jessie-sysv/contrib] precedence = 95 description = + Keeping sysvinit instead of switching to systemd (if possible). piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver --scriptsdir /etc/piuparts/scripts-sysvinit depends-sections = wheezy2jessie-sysv/main wheezy2jessie-sysv/non-free arch = amd64 area = contrib distro = wheezy upgrade-test-distros = wheezy jessie reschedule-old-count = 0 [wheezy2jessie-sysv/non-free] precedence = 95 description = + Keeping sysvinit instead of switching to systemd (if possible). piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver --scriptsdir /etc/piuparts/scripts-sysvinit depends-sections = wheezy2jessie-sysv/main wheezy2jessie-sysv/contrib arch = amd64 area = non-free distro = wheezy upgrade-test-distros = wheezy jessie reschedule-old-count = 0 [wheezy2jessie-apt1st/main] precedence = 80 description = + Upgrading apt before the rest of the system. piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver --scriptsdir /etc/piuparts/scripts-apt-first arch = amd64 area = main components = main distro = wheezy upgrade-test-distros = wheezy jessie reschedule-old-count = 0 [wheezy2jessie-apt1st/contrib] precedence = 95 description = + Upgrading apt before the rest of the system. piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver --scriptsdir /etc/piuparts/scripts-apt-first depends-sections = wheezy2jessie-apt1st/main wheezy2jessie-apt1st/non-free arch = amd64 area = contrib distro = wheezy upgrade-test-distros = wheezy jessie reschedule-old-count = 0 [wheezy2jessie-apt1st/non-free] precedence = 95 description = + Upgrading apt before the rest of the system. piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver --scriptsdir /etc/piuparts/scripts-apt-first depends-sections = wheezy2jessie-apt1st/main wheezy2jessie-apt1st/contrib arch = amd64 area = non-free distro = wheezy upgrade-test-distros = wheezy jessie reschedule-old-count = 0 [wheezy2jessie-iri/main] precedence = 90 description = Debian %(area)s: package installation and removal in wheezy, then upgrade to jessie, installation, removal, and purge test. piuparts-flags = --install-remove-install %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s arch = amd64 area = main components = main distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-iri/contrib] precedence = 95 description = Debian %(area)s: package installation and removal in wheezy, then upgrade to jessie, installation, removal, and purge test. piuparts-flags = --install-remove-install %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s depends-sections = wheezy2jessie-iri/main wheezy2jessie-iri/non-free arch = amd64 area = contrib distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2jessie-iri/non-free] precedence = 95 description = Debian %(area)s: package installation and removal in wheezy, then upgrade to jessie, installation, removal, and purge test. piuparts-flags = --install-remove-install %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s depends-sections = wheezy2jessie-iri/main wheezy2jessie-iri/contrib arch = amd64 area = non-free distro = wheezy upgrade-test-distros = wheezy jessie [wheezy2bpo2jessie/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s depends-sections = wheezy/main distro = wheezy-backports arch = amd64 area = main components = main upgrade-test-distros = wheezy wheezy-backports jessie [wheezy2bpo2jessie/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s depends-sections = wheezy/main wheezy/non-free wheezy2bpo2jessie/main wheezy2bpo2jessie/non-free distro = wheezy-backports arch = amd64 area = contrib upgrade-test-distros = wheezy wheezy-backports jessie [wheezy2bpo2jessie/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-jessie)s depends-sections = wheezy/main wheezy2bpo2jessie/main wheezy2bpo2jessie/contrib distro = wheezy-backports arch = amd64 area = non-free upgrade-test-distros = wheezy wheezy-backports jessie [wheezy222testing/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-testing)s arch = amd64 area = main components = main distro = wheezy upgrade-test-distros = wheezy jessie testing reschedule-old-count = 100 [wheezy222testing/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-testing)s depends-sections = wheezy222testing/main wheezy222testing/non-free arch = amd64 area = contrib distro = wheezy upgrade-test-distros = wheezy jessie testing reschedule-old-count = 100 [wheezy222testing/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-wheezy)s %(flags-end-testing)s depends-sections = wheezy222testing/main wheezy222testing/contrib arch = amd64 area = non-free distro = wheezy upgrade-test-distros = wheezy jessie testing reschedule-old-count = 100 [jessie2proposed/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = amd64 area = main components = main distro = jessie upgrade-test-distros = jessie jessie-proposed [jessie2proposed/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie2proposed/main jessie2proposed/non-free arch = amd64 area = contrib distro = jessie upgrade-test-distros = jessie jessie-proposed [jessie2proposed/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-jessie)s depends-sections = jessie2proposed/main jessie2proposed/contrib arch = amd64 area = non-free distro = jessie upgrade-test-distros = jessie jessie-proposed [jessie2stretch/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = amd64 area = main components = main distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = jessie2stretch/main jessie2stretch/non-free arch = amd64 area = contrib distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = jessie2stretch/main jessie2stretch/contrib arch = amd64 area = non-free distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch-rcmd_i386/main] precedence = 80 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver arch = i386 area = main components = main distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch-rcmd_i386/contrib] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = jessie2stretch-rcmd_i386/main jessie2stretch-rcmd_i386/non-free arch = i386 area = contrib distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch-rcmd_i386/non-free] precedence = 95 description = + Testing with --install-recommends. piuparts-flags = --install-recommends %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s --scriptsdir /etc/piuparts/scripts-debug-problemresolver depends-sections = jessie2stretch-rcmd_i386/main jessie2stretch-rcmd_i386/contrib arch = i386 area = non-free distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch-iri/main] precedence = 90 description = Debian %(area)s: package installation and removal in jessie, then upgrade to stretch, installation, removal, and purge test. piuparts-flags = --install-remove-install %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s arch = amd64 area = main components = main distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch-iri/contrib] precedence = 95 description = Debian %(area)s: package installation and removal in jessie, then upgrade to stretch, installation, removal, and purge test. piuparts-flags = --install-remove-install %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s depends-sections = jessie2stretch-iri/main jessie2stretch-iri/non-free arch = amd64 area = contrib distro = jessie upgrade-test-distros = jessie stretch [jessie2stretch-iri/non-free] precedence = 95 description = Debian %(area)s: package installation and removal in jessie, then upgrade to stretch, installation, removal, and purge test. piuparts-flags = --install-remove-install %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s depends-sections = jessie2stretch-iri/main jessie2stretch-iri/contrib arch = amd64 area = non-free distro = jessie upgrade-test-distros = jessie stretch [jessie2bpo2stretch/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s depends-sections = jessie/main distro = jessie-backports arch = amd64 area = main components = main upgrade-test-distros = jessie jessie-backports stretch [jessie2bpo2stretch/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s depends-sections = jessie/main jessie/contrib jessie/non-free jessie2bpo2stretch/main jessie2bpo2stretch/non-free distro = jessie-backports arch = amd64 area = contrib upgrade-test-distros = jessie jessie-backports stretch [jessie2bpo2stretch/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-jessie)s %(flags-end-stretch)s depends-sections = jessie/main jessie/contrib jessie/non-free jessie2bpo2stretch/main jessie2bpo2stretch/contrib distro = jessie-backports arch = amd64 area = non-free upgrade-test-distros = jessie jessie-backports stretch [stable2sid/main] precedence = 70 piuparts-flags = %(flags-default)s %(flags-start-stable)s arch = amd64 area = main components = main distro = stable upgrade-test-distros = stable sid [stable2sid/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-stable)s depends-sections = stable2sid/main stable2sid/non-free arch = amd64 area = contrib distro = stable upgrade-test-distros = stable sid [stable2sid/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-stable)s depends-sections = stable2sid/main stable2sid/contrib arch = amd64 area = non-free distro = stable upgrade-test-distros = stable sid [stable2testing2sid/main] precedence = 80 piuparts-flags = %(flags-default)s %(flags-start-stable)s arch = amd64 area = main components = main distro = stable upgrade-test-distros = stable testing sid reschedule-old-count = 0 [stable2testing2sid/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-stable)s depends-sections = stable2testing2sid/main stable2testing2sid/non-free arch = amd64 area = contrib distro = stable upgrade-test-distros = stable testing sid reschedule-old-count = 0 [stable2testing2sid/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-stable)s depends-sections = stable2testing2sid/main stable2testing2sid/contrib arch = amd64 area = non-free distro = stable upgrade-test-distros = stable testing sid reschedule-old-count = 0 [testing2sid/main] precedence = 60 piuparts-flags = %(flags-default)s %(flags-start-testing)s arch = amd64 area = main components = main distro = testing upgrade-test-distros = testing sid reschedule-old-count = 0 [testing2sid/contrib] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-testing)s depends-sections = testing2sid/main testing2sid/non-free arch = amd64 area = contrib distro = testing upgrade-test-distros = testing sid reschedule-old-count = 0 [testing2sid/non-free] precedence = 95 piuparts-flags = %(flags-default)s %(flags-start-testing)s depends-sections = testing2sid/main testing2sid/contrib arch = amd64 area = non-free distro = testing upgrade-test-distros = testing sid reschedule-old-count = 0 [sid2experimental/main] precedence = 70 piuparts-flags = %(flags-default)s depends-sections = sid/main arch = amd64 area = main components = main upgrade-test-distros = sid experimental [sid2experimental/contrib] precedence = 95 piuparts-flags = %(flags-default)s depends-sections = sid2experimental/main sid/main sid/contrib sid/non-free arch = amd64 area = contrib upgrade-test-distros = sid experimental [sid2experimental/non-free] precedence = 95 piuparts-flags = %(flags-default)s depends-sections = sid2experimental/main sid/main sid/contrib sid/non-free arch = amd64 area = non-free upgrade-test-distros = sid experimental piuparts/known_problems/0000755000000000000000000000000013061276553012662 5ustar piuparts/known_problems/alternatives_after_purge_error.conf0000644000000000000000000000125412567171321022025 0ustar # # detect packages with unowned files in /etc/alternatives after purge (policy 6.8) # PATTERN='/etc/alternatives/.*not owned' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with leftover alternatives after purge' HELPTEXT='

Packages with unowned files in /etc/alternatives after purge (violating policy 6.8) see https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails

Alternatives are usually registered with update-alternatives in postinst configure and need to be unregistered again in prerm remove.

' piuparts/known_problems/alternatives_after_purge_issue.conf0000644000000000000000000000123412567171321022022 0ustar # # detect packages with unowned files in /etc/alternatives after purge (policy 6.8) # PATTERN='/etc/alternatives/.*not owned' WHERE='pass' ISSUE=1 HEADER='Packages with leftover alternatives after purge' HELPTEXT='

Packages with unowned files in /etc/alternatives after purge (violating policy 6.8) see https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails

Alternatives are usually registered with update-alternatives in postinst configure and need to be unregistered again in prerm remove.

' piuparts/known_problems/bin_or_sbin_binary_requires_usr_lib_library_inadequate_issue.conf0000644000000000000000000000076012567171321030152 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* bin-or-sbin-binary-requires-usr-lib-library' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'bin-or-sbin-binary-requires-usr-lib-library' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'bin-or-sbin-binary-requires-usr-lib-library' which indicates a bug.

" piuparts/known_problems/boring_broken_symlink_inadequate_issue.conf0000644000000000000000000000075412567171321023532 0ustar # # detect packages which have the string "Running adequate resulted in less interesting tags found" in their logs # PATTERN='(FAIL|WARN): Running adequate resulted in less interesting tags found: .* broken-symlink' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'broken-symlink' by adequate" HELPTEXT="

Some issues detected by adequate are also detected by piuparts, 'broken-symlink' is one of them. " piuparts/known_problems/boring_obsolete_conffile_inadequate_issue.conf0000644000000000000000000000076512567171321024167 0ustar # # detect packages which have the string "Running adequate resulted in less interesting tags found" in their logs # PATTERN='(FAIL|WARN): Running adequate resulted in less interesting tags found: .* obsolete-conffile' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'obsolete-conffile' by adequate" HELPTEXT="

Some issues detected by adequate are also detected by piuparts, 'obsolete-conffile' is one of them. " piuparts/known_problems/broken_binfmt_detector_inadequate_issue.conf0000644000000000000000000000076012567171321023651 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* broken-binfmt-detector' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'broken-binfmt-detector' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'broken-binfmt-detector' which indicates a bug: The detector registered with update-binfmts(8) does not exist.

" piuparts/known_problems/broken_binfmt_interpreter_inadequate_issue.conf0000644000000000000000000000077412567171321024410 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* broken-binfmt-interpreter' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'broken-binfmt-interpreter' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'broken-binfmt-interpreter' which indicates a bug: The interpreter registered with update-binfmts(8) does not exist.

" piuparts/known_problems/broken_symlinks_error.conf0000644000000000000000000000103612567171321020150 0ustar # # detect packages which have the string "Broken symlinks" in their logs # PATTERN='(WARN|FAIL): Broken symlink' WHERE='fail bugged affected' ISSUE=1 HEADER='Packages which have logs with the string "Broken symlinks"' HELPTEXT='

This is clearly an error, but as there are too many of this kind, piuparts can be configured to not fail if it detects broken symlinks. Another option is not to test for broken symlinks. See the piuparts manpage for details.

' piuparts/known_problems/broken_symlinks_issue.conf0000644000000000000000000000101612567171321020145 0ustar # # detect packages which have the string "Broken symlinks" in their logs # PATTERN='(WARN|FAIL): Broken symlink' WHERE='pass' ISSUE=1 HEADER='Packages which have logs with the string "Broken symlinks"' HELPTEXT='

This is clearly an error, but as there are too many of this kind, piuparts can be configured to not fail if it detects broken symlinks. Another option is not to test for broken symlinks. See the piuparts manpage for details.

' piuparts/known_problems/command_not_found_error.conf0000644000000000000000000000172212567171321020432 0ustar # # detect packages with miss a depends or use non-essential in purge # PATTERN='command not found|: not found' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs with the string "command not found"' HELPTEXT='

From the third paragraph about the meaning of the depends field in https://www.debian.org/doc/debian-policy/ch-relationships.html#s-binarydeps 

The Depends field should also be used if the postinst, prerm or postrm scripts
require the package to be present in order to run. __Note, however, that the
postrm cannot rely on any non-essential packages to be present during the
purge phase__.

NOTE: it has not been verified that this error really caused the package to fail the piuparts test, but it did fail.
There are also successful logs with "command not found" (though not listed below).

' piuparts/known_problems/command_not_found_issue.conf0000644000000000000000000000145312567171321020432 0ustar # # detect packages which passed the piuparts test but have the string "command not found" in their logs # PATTERN='command not found|: not found' WHERE='pass' ISSUE=1 HEADER='Packages which passed the piuparts test but have logs with the string "command not found"' HELPTEXT='

From the third paragraph about the meaning of the depends field in https://www.debian.org/doc/debian-policy/ch-relationships.html#s-binarydeps: 

The Depends field should also be used if the postinst, prerm or postrm scripts
require the package to be present in order to run. __Note, however, that the
postrm cannot rely on any non-essential packages to be present during the
purge phase__.
' piuparts/known_problems/conffile_prompt_error.conf0000644000000000000000000000233712762537303020135 0ustar # # detect packages with failed because they prompt due to a modified conffile # PATTERN='(EOF|end of file) on stdin at conffile prompt' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs with the string "EOF on stdin at conffile prompt"' HELPTEXT='

The piuparts upgrade test failed because dpkg detected a conffile as being modified and then prompted the user for an action. As there is no user input, this fails. But this is not the real problem, the real problem is that this prompt shows up in the first place, as there was nobody modifying this conffile at all, the package has just been installed and upgraded...

This is explained in detail in policy 10.7.3 at https://www.debian.org/doc/debian-policy/ch-files.html#s10.7.3 which says "[These scripts handling conffiles] must not ask unnecessary questions (particularly during upgrades), and must otherwise be good citizens."

According to the thread started at 200908191215.05079.holger@layer-acht.org these bugs are to be filed with severity serious.

' piuparts/known_problems/cron_error_after_removal_error.conf0000644000000000000000000000146112567171321022021 0ustar # # detect packages with cron errors after the package has been removed # PATTERN='(FAIL: Cron file .* has output with package removed|ERROR: Command failed.*./etc/cron\..*/.*.])' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because a cron file has output and/or exits with error after the package has been removed' HELPTEXT='

From https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.5 

The scripts or crontab entries in these directories should check if all
necessary programs are installed before they try to execute them. Otherwise,
problems will arise when a package was removed but not purged since
configuration files are kept on the system in this situation.
' piuparts/known_problems/db_setup_error.conf0000644000000000000000000000161613042113740016536 0ustar # # detect packages with failed to install due to problems configuring the db - see #595652 # PATTERN='(warning: database package not installed|dbconfig-common: .* configure: (aborted|noninteractive fail|ignoring errors from here forwards)|updating database schema for .*...command failed with code 0|(postgres|psql): could not connect to server: No such file or directory|DBI connect.* failed: could not connect to server|pg_pconnect\(\): Unable to connect to PostgreSQL server|Non-default port in /etc/postgresql/.*/main/postgresql.conf|Unable to connect to MySQL server|unable to connect to mysql server)' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because installation failed because no database could be connected.' HELPTEXT='

Just how buggy these packages really are is discussed in #595652 at the moment.

' piuparts/known_problems/db_setup_issue.conf0000644000000000000000000000157613042113740016542 0ustar # # detect packages with failed to install due to problems configuring the db - see #595652 # PATTERN='(warning: database package not installed|dbconfig-common: .* configure: (aborted|noninteractive fail|ignoring errors from here forwards)|updating database schema for .*...command failed with code 0|(postgres|psql): could not connect to server: No such file or directory|DBI connect.* failed: could not connect to server|pg_pconnect\(\): Unable to connect to PostgreSQL server|Non-default port in /etc/postgresql/.*/main/postgresql.conf|Unable to connect to MySQL server|unable to connect to mysql server)' WHERE='pass' ISSUE=1 HEADER='Packages with failed logs because installation failed because no database could be connected.' HELPTEXT='

Just how buggy these packages really are is discussed in #595652 at the moment.

' piuparts/known_problems/debsums_mismatch_error.conf0000644000000000000000000000052313036653026020265 0ustar # # detect packages which modify shipped files # PATTERN='FAIL: debsums reports modifications inside the chroot|MISSING OBJECT' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with modified files before removal' HELPTEXT='

Modifying conffiles is forbidden by Policy 10.7.3. Modifying other shipped files is a stupid idea.

' piuparts/known_problems/debsums_mismatch_issue.conf0000644000000000000000000000050313036653026020262 0ustar # # detect packages which modify shipped files # PATTERN='FAIL: debsums reports modifications inside the chroot|MISSING OBJECT' WHERE='pass' ISSUE=1 HEADER='Packages with modified files before removal' HELPTEXT='

Modifying conffiles is forbidden by Policy 10.7.3. Modifying other shipped files is a stupid idea.

' piuparts/known_problems/dependency_error.conf0000644000000000000000000000103512567171321017054 0ustar # # detect packages with unsatisfied dependencies # PATTERN='E: Broken packages|E: Unable to correct problems, you have held broken packages.|E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because of unsatisfied dependencies' HELPTEXT='

Usually this is caused by some unsatisfied (versioned) Depends/Conflicts/Replaces. These packages will be automatically rescheduled for testing seven days after they failed.

' piuparts/known_problems/disappeared_files_after_purge_error.conf0000644000000000000000000000043512567171321022767 0ustar # # detect packages with disappeared files after purge # PATTERN='FAIL: After purging files have disappeared:' EXCLUDE_PATTERN=',|usr/local' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with files which disappeared after purge' HELPTEXT='

This is obviously broken.

' piuparts/known_problems/diversion_error.conf0000644000000000000000000000051612567171321016743 0ustar # # detect packages which mishandle diversions # PATTERN='ERROR: FAIL: (Existing|Installed) diversions' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages that leave a modified diversion state' HELPTEXT='

Packages need to remove all the diversions they introduced and may not modify diversions setup by other packages.

' piuparts/known_problems/files_in_usr_local_error.conf0000644000000000000000000000066612567171321020602 0ustar # # detect packages which leave stuff in /usr/local (see policy 9.1.2) # PATTERN='usr/local.+not owned' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with files and/or directories in /usr/local/ after purge' HELPTEXT='

This is a violation of policy 9.1.2: see https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.1.2.

' piuparts/known_problems/immediate_configuration_error.conf0000644000000000000000000000055712567171321021633 0ustar # # detect packages which failed because apt could not perform immediate configuration # PATTERN='E: Could not perform immediate configuration on' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because apt could not perform immediate configuration' HELPTEXT='

This is a bug in apt, but it has to be worked around in some packages.

' piuparts/known_problems/inadequate_exit_issue.conf0000644000000000000000000000074512567171321020115 0ustar # # detect packages which have the string "Exit code from adequate was" in their logs # PATTERN='(WARN|FAIL): Exit code from adequate was' WHERE='pass fail bugged affected' ISSUE=0 HEADER='Packages which have logs with the string "Exit code from adequate was"' HELPTEXT='

Running adequate resulted in an exit code not equal zero, which indicates a severe problem with adequate. Please investigate and report.

' piuparts/known_problems/incompatible_licenses_inadequate_issue.conf0000644000000000000000000000065612567171321023500 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* incompatible-licenses' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'incompatible-licenses' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'incompatible-licenses' which indicates a bug.

" piuparts/known_problems/initdscript_lsb_header_issue.conf0000644000000000000000000000261512762537303021451 0ustar # # detect packages with an update-rc.d warning # PATTERN='update-rc.d: warning.*do not match LSB|service.*already provided' WHERE='fail bugged affected pass' ISSUE=1 HEADER='Packages with logs with the string "update-rc.d: warning.*do not match LSB"' HELPTEXT='

Some packages have inconsistency between the init.d script headers used with dependency based boot sequencing and the runlevels specified on the update-rc.d command line and used by the legacy boot ordering. Such inconsistency is most likely a bug in the package, as the two ways of ordering init.d scripts should enable and disable the scripts in the same runlevels while Debian migrate to dependency based boot sequencing.

Such inconsinstency is reported like this when a postinst script call update-rc.d 

  update-rc.d: warning: initdscript start runlevel arguments (2 3 4 5) do not match LSB Default-Start values (S)
  update-rc.d: warning: initdscript stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (none)

Such reports are most likely bugs in the package calling update-rc.d, and should be reported and fixed in the individual packages.

See the paragraph "How to solve migration problems" at https://wiki.debian.org/LSBInitScripts/DependencyBasedBoot for information how to fix these issues.

' piuparts/known_problems/insserv_error.conf0000644000000000000000000000241012762537303016430 0ustar # # detect packages with fail due to insserv error # PATTERN='insserv: exiting now|service.*already provided' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs with the string "insserv: exiting now"' HELPTEXT='

Some packages fail to install because their init.d scripts have headers with bugs. There are several classes of bugs.

Some packages fail to install because the init.d script have conflicting provide. This is normally reported like this: 

  insserv: script clvm: service lvm already provided!
  insserv: exiting now!

Other packages fail to install because their dependencies are missing. This is normally reported like this: 

  insserv: Service portmap has to be enabled to start service quotarpc
  insserv: exiting now!

Last, some packages introduce dependency loops, this is normally reported like this: 

  insserv: There is a loop between service script1 and script2 if started
  insserv: exiting without changing boot order!

See the paragraph "How to solve migration problems" at https://wiki.debian.org/LSBInitScripts/DependencyBasedBoot for information how to fix these issues.

' piuparts/known_problems/installs_over_symlink_error.conf0000644000000000000000000000115212567171321021370 0ustar # # detect packages which have the string "dirname part contains a symlink" in their logs # PATTERN='silently overwrites files via directory symlinks|dirname part contains a symlink' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages that install something over existing symlinks' HELPTEXT='

Installing anything over a symlink opens a can of worms - this causes problems on upgrades while switching between directory and symlink or if the symlink is ever changed.
Piuparts looks at all $pathname known to dpkg and checks for 

$(dirname $pathname) != $(readlink $(dirname $pathname))

' piuparts/known_problems/installs_over_symlink_issue.conf0000644000000000000000000000113212567171321021365 0ustar # # detect packages which have the string "dirname part contains a symlink" in their logs # PATTERN='silently overwrites files via directory symlinks|dirname part contains a symlink' WHERE='pass' ISSUE=1 HEADER='Packages that install something over existing symlinks' HELPTEXT='

Installing anything over a symlink opens a can of worms - this causes problems on upgrades while switching between directory and symlink or if the symlink is ever changed.
Piuparts looks at all $pathname known to dpkg and checks for 

$(dirname $pathname) != $(readlink $(dirname $pathname))

' piuparts/known_problems/ldd_inadequate_issue.conf0000644000000000000000000000062012762537303017702 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* ldd-failure' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'ldd-failure' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'ldd-failure' which indicates a bug.

" piuparts/known_problems/library_not_found_inadequate_issue.conf0000644000000000000000000000064212567171321022657 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* library-not-found' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'library-not-found' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'library-not-found' which indicates a bug.

" piuparts/known_problems/logrotate_error_after_removal_error.conf0000644000000000000000000000107212567171321023056 0ustar # # detect packages with logrotate errors after the package has been removed # PATTERN='(FAIL: Logrotate file .* has output with package removed|ERROR: Command failed.*'/etc/logrotate\.d/.*'])' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because a logrotate script has output and/or exits with error after the package has been removed' HELPTEXT='

Most of these packages are probably not buggy but rather affected by #582630. It is being considered to disable this check...

' piuparts/known_problems/maintainer_script_issue.conf0000644000000000000000000000054712567171321020457 0ustar # # detect packages in successfully-tested state that had maintainer script failures # PATTERN='subprocess .*(pre|post)-(installation|removal) script returned error' WHERE='pass' ISSUE=0 HEADER='Packages in state successfully-tested but logfile contains a maintainer script failure' HELPTEXT='

This is a somewhat unclassified issue at the moment.

' piuparts/known_problems/missing-symbol-version-information_inadequate_issue.conf0000644000000000000000000000072512567171321026124 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* missing-symbol-version-information' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'missing-symbol-version-information' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'missing-symbol-version-information' which indicates a bug.

" piuparts/known_problems/missing_alternative_inadequate_issue.conf0000644000000000000000000000120312567171321023201 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* missing-alternative' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'missing-alternative' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'missing-alternative' which indicates a bug similar to this situation: a package is a provider of the virtual package 'x-terminal-emulator', but it doesn't register itself as an alternative for '/usr/bin/x-terminal-emulator'. See debian-policy 11.8.3 and 11.8.4.

" piuparts/known_problems/missing_copyright_file_inadequate_issue.conf0000644000000000000000000000070212567171321023675 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .*missing-copyright-file|MISSING COPYRIGHT FILE' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'missing-copyright-file' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'missing-copyright-file' which indicates a bug. " piuparts/known_problems/missing_md5sums_error.conf0000644000000000000000000000040712567171321020066 0ustar # # detect packages that ship files without md5sums # PATTERN='(FILE WITHOUT MD5SUM|MD5SUM FILE NOT FOUND)' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages containing files without md5sum' HELPTEXT='

Packages that ship files that have no md5sum.

' piuparts/known_problems/missing_md5sums_issue.conf0000644000000000000000000000036712567171321020072 0ustar # # detect packages that ship files without md5sums # PATTERN='(FILE WITHOUT MD5SUM|MD5SUM FILE NOT FOUND)' WHERE='pass' ISSUE=1 HEADER='Packages containing files without md5sum' HELPTEXT='

Packages that ship files that have no md5sum.

' piuparts/known_problems/missing_pkgconfig-dependency_issue.conf0000644000000000000000000000067512762537303022565 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .*missing-pkgconfig-dependency' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'missing-pkgconfig-dependency' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'missing-pkgconfig-dependency' which indicates a bug. " piuparts/known_problems/modified_files_after_purge_error.conf0000644000000000000000000000042312567171321022263 0ustar # # detect packages with modified files after purge # PATTERN='FAIL: After purging files have been modified:' EXCLUDE_PATTERN=',|usr/local' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with modified files after purge' HELPTEXT='

This is obviously broken.

' piuparts/known_problems/module_build_error_issue.conf0000644000000000000000000000052712567171321020617 0ustar # # detect dkms module source packages that fail to build for a current kernel # PATTERN='Error! Bad return status for module build on kernel' WHERE='pass fail bugged affected' ISSUE=1 HEADER="DKMS-packages failing to build a module for a default kernel" HELPTEXT="

Kernel module source may not be compatible with the latest kernel.

" piuparts/known_problems/needs_rebuild_issue.conf0000644000000000000000000000154713036653026017550 0ustar # # detect packages which have the string "Please rebuild the package" or "package ... should be rebuilt" in their logs # PATTERN="Please rebuild the package|should be rebuilt|should be rebuild|warning: maintainer scripts should not call install-info anymore|called emacs-package-install as a new-style add-on, but has no compat file" WHERE='pass' ISSUE=1 HEADER='Packages which have logs with the string "Please rebuild the package" or "package ... should be rebuilt"' HELPTEXT='

This is a recommendation to rebuild some packages with updated debhelper to enable new features, e.g. trigger support.

Please identify the correct package causing this warning and retest the rdepends after that package was fixed.

This issue may be a false positive during upgrade tests, since it can be triggered on an old package before (or while) it gets upgraded.

' piuparts/known_problems/obsolete_conffiles_error.conf0000644000000000000000000000100712567171321020601 0ustar # # detect packages that leave obsolete conffiles after upgrades # PATTERN='OBSOLETE CONFFILE' WHERE='fail bugged affected' ISSUE=1 HEADER='Packages leaving obsolete conffiles after upgrade' HELPTEXT='

Packages that leave obsolete conffiles after upgrade. Using 

dpkg-maintscript-helper rm_conffile
via dh_installdeb package.maintscript files is the recommended way to clean them up. There may be false positives, e.g. if a conffile was converted to a maintainer script managed configuration file.

' piuparts/known_problems/obsolete_conffiles_issue.conf0000644000000000000000000000076712567171321020614 0ustar # # detect packages that leave obsolete conffiles after upgrades # PATTERN='OBSOLETE CONFFILE' WHERE='pass' ISSUE=1 HEADER='Packages leaving obsolete conffiles after upgrade' HELPTEXT='

Packages that leave obsolete conffiles after upgrade. Using 

dpkg-maintscript-helper rm_conffile
via dh_installdeb package.maintscript files is the recommended way to clean them up. There may be false positives, e.g. if a conffile was converted to a maintainer script managed configuration file.

' piuparts/known_problems/overwrite_other_packages_files_error.conf0000644000000000000000000000145412567171321023212 0ustar # # detect packages which try to overwrite other packages files # PATTERN='trying to overwrite (.*) which is also in package' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because they tried to overwrite other packages files' HELPTEXT='

This is because the package tries to overwrite another packages files without declaring a replaces relation. See policy 7.6 at https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces.

According to the thread started at 200908071233.02813.holger@layer-acht.org these bugs are to be filed with severity serious.

' piuparts/known_problems/owned_files_after_purge_error.conf0000644000000000000000000000114412567171321021620 0ustar # # detect packages with owned files after purge (policy 6.8) # PATTERN='owned by:' EXCLUDE_PATTERN=',|usr/local/' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with owned files after purge' HELPTEXT='

WARNING: this list might contain false positives.
#316521: dpkg: incomplete cleanup of empty directories

Packages need to remove owned files after purge, see https://www.debian.org/doc/debian-policy/ch-files.html#s10.7.3

' piuparts/known_problems/owned_files_after_purge_issue.conf0000644000000000000000000000106512567171321021621 0ustar # # detect packages with owned files after purge (policy 6.8) # PATTERN='owned by:' WHERE='pass' ISSUE=1 HEADER='Packages with owned files after purge' HELPTEXT='

WARNING: this list might contain false positives.
#316521: dpkg: incomplete cleanup of empty directories

Packages need to remove owned files after purge, see https://www.debian.org/doc/debian-policy/ch-files.html#s10.7.3

' piuparts/known_problems/owned_files_by_many_packages_error.conf0000644000000000000000000000113612567171321022612 0ustar # # detect conffiles owned by two or more packages which are not removed by dpkg correctly # PATTERN='owned by.+,' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs which have conffiles owned by two or more packages which are not removed by dpkg correctly' HELPTEXT='

So these might be ok packages (or not, as there might be other problems), but we cannot know until #454694 in dpkg is fixed.
#316521: dpkg: incomplete cleanup of empty directories

' piuparts/known_problems/packages_have_been_kept_back_error.conf0000644000000000000000000000051512567171321022515 0ustar # # detect possible dependency issues # PATTERN='packages have been kept back' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with logs with the string "packages have been kept back"' HELPTEXT='

"packages have been kept back" usually indicates some dependency issue that caused apt to not fully upgrade the system.

' piuparts/known_problems/packages_have_been_kept_back_issue.conf0000644000000000000000000000047512567171321022521 0ustar # # detect possible dependency issues # PATTERN='packages have been kept back' WHERE='pass' ISSUE=1 HEADER='Packages with logs with the string "packages have been kept back"' HELPTEXT='

"packages have been kept back" usually indicates some dependency issue that caused apt to not fully upgrade the system.

' piuparts/known_problems/piuparts-depends-dummy_error.conf0000644000000000000000000000060413042113740021345 0ustar # # detect problems where piuparts-depends-dummy.deb was not installed # PATTERN='ERROR: Installation of piuparts-depends-dummy FAILED|ERROR: Command failed.*apt-get.*install.*piuparts-depends-dummy.deb' WHERE='fail bugged affected' ISSUE=0 HEADER='Problems installing piuparts-depends-dummy.deb' HELPTEXT='

apt chose to remove piuparts-depends-dummy.deb instead of fixing it.

' piuparts/known_problems/piuparts-depends-dummy_issue.conf0000644000000000000000000000046012567171321021356 0ustar # # detect problems where piuparts-depends-dummy.deb was not installed # PATTERN='ERROR: Installation of piuparts-depends-dummy FAILED' WHERE='pass' ISSUE=1 HEADER='Problems installing piuparts-depends-dummy.deb' HELPTEXT='

apt chose to remove piuparts-depends-dummy.deb instead of fixing it.

' piuparts/known_problems/post_installation_script_error.conf0000644000000000000000000000053612567171321022075 0ustar # # detect packages which failed because post-installation maintainer script failed # PATTERN='subprocess .*post-installation script returned error' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because post-installation maintainer script failed' HELPTEXT='

This is a somewhat unclassified failure at the moment.

' piuparts/known_problems/post_removal_script_error.conf0000644000000000000000000000051712567171321021040 0ustar # # detect packages which failed because post-removal maintainer script failed # PATTERN='subprocess .*post-removal script returned error' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because post-removal maintainer script failed' HELPTEXT='

This is a somewhat unclassified failure at the moment.

' piuparts/known_problems/pre_depends_error.conf0000644000000000000000000000142712567171321017233 0ustar # # detect packages which failed because of a problem with pre-depends # PATTERN='E: Couldn.t configure pre-depend .* for .*, probably a dependency cycle.' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because of a problem with pre-depends ' HELPTEXT='

The package(s) in question fail(s) to install or upgrade properly, because a pre-dependent package could not be configured. This is likely due to a dependency cycle.
Note that it is possible, that aptitude can deal with (some of) such situations (ie upgrades), (by removing the packages first), while apt-get cannot. While it can be argued this is a problem in apt-get, it is also a problem in the package(s) listed below, as both aptitude and apt-get can be used for installing packages.

' piuparts/known_problems/pre_installation_script_error.conf0000644000000000000000000000053312567171321021673 0ustar # # detect packages which failed because pre-installation maintainer script failed # PATTERN='subprocess .*pre-installation script returned error' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because pre-installation maintainer script failed' HELPTEXT='

This is a somewhat unclassified failure at the moment.

' piuparts/known_problems/pre_removal_script_error.conf0000644000000000000000000000051412567171321020636 0ustar # # detect packages which failed because pre-removal maintainer script failed # PATTERN='subprocess .*pre-removal script returned error' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because pre-removal maintainer script failed' HELPTEXT='

This is a somewhat unclassified failure at the moment.

' piuparts/known_problems/problems_and_no_force_error.conf0000644000000000000000000000135613042113740021251 0ustar # # detect packages with problems because of not enough force # PATTERN='E: There are problems and -y was used without --force-yes|E: Essential packages were removed and -y was used without --allow-remove-essential' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because of not enough force' HELPTEXT='

The piuparts logfile for these packages contains the string 

E: There are problems and -y was used without --force-yes
, which is usually an indication, that an essential package needs to be removed to install this package. As piuparts does not use that much force, the piuparts test fails.

This is usually not an error in the package and it needs to be seen how piuparts should deal with it.

' piuparts/known_problems/problems_and_no_force_issue.conf0000644000000000000000000000131513061276553021260 0ustar # # detect packages with problems because of not enough force # PATTERN='E: Essential packages were removed and -y was used without --allow-remove-essential' WHERE='pass' ISSUE=1 HEADER='Packages with logs reporting attempted removal of essential packages' HELPTEXT='

The piuparts logfile for these packages contains the string 

E: Essential packages were removed and -y was used without --allow-remove-essential
, which is usually an indication, that an essential package needs to be removed to install this package. As piuparts does not use that much force, the piuparts test fails.

This is usually not an error in the package and it needs to be seen how piuparts should deal with it.

' piuparts/known_problems/processes_running_error.conf0000644000000000000000000000205412567171321020506 0ustar # # detect packages which failed because of processes are running inside chroot # PATTERN='ERROR: FAIL: Processes are running inside chroot' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because of processes left behind' HELPTEXT='

There were processes running inside the chroot at the end of the piuparts run. This is probably due to directly calling /etc/rc.d/ scripts in packages maintainer scripts, which is a violation of policy 9.3.3.2 and must be replaced by using invoke-rc.d (which will respect an optionally existing policy-rc.d) - see https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.3.3, /usr/share/doc/sysv-rc/README.invoke-rc.d.gz and /usr/share/doc/sysv-rc/README.policy-rc.d.gz.

According to the thread started at 200908061127.35727.holger@layer-acht.org these bugs are to be filed with severity serious.

' piuparts/known_problems/program_name_collision_inadequate_issue.conf0000644000000000000000000000103612567171321023660 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* program-name-collision' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'program-name-collision' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'program-name-collision' which indicates that this package ships a program with the same name as another program. This is a violation of debian-policy 10.1.

" piuparts/known_problems/py_file_not_bytecompiled_inadequate_issue.conf0000644000000000000000000000066712567171321024216 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* py-file-not-bytecompiled' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'py-file-not-bytecompiled' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'py-file-not-bytecompiled' which indicates a bug.

" piuparts/known_problems/pyshared_file_not_bytecompiled_inadequate_issue.conf0000644000000000000000000000071112567171321025373 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* pyshared-file-not-bytecompiled' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'pyshared-file-not-bytecompiled' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'pyshared-file-not-bytecompiled' which indicates a bug.

" piuparts/known_problems/resource_violation_error.conf0000644000000000000000000000067712567171321020664 0ustar # # detect packages violating resource limits during the test # PATTERN='Command was terminated after exceeding|Process KILLED - exceed maximum run time' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages where the test was aborted after exceeding time or output limit' HELPTEXT='

Usually caused by seriously misbehaving maintainer scripts that go into endless loops or try to get user input in DEBIAN_FRONTEND=noninteractive mode.

' piuparts/known_problems/symbol-size-mismatch_inadequate_issue.conf0000644000000000000000000000065312567171321023222 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* symbol-size-mismatch' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'symbol-size-mismatch' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'symbol-size-mismatch' which indicates a bug.

" piuparts/known_problems/test_was_skipped_issue.conf0000644000000000000000000000115613036653026020310 0ustar # # detect possible dependency issues # PATTERN='SKIP: All tests. Package cannot be tested with piuparts:' WHERE='pass' ISSUE=1 HEADER='Packages with logs with the string "SKIP: All tests. Package cannot be tested with piuparts"' HELPTEXT='

The package is not installable by piuparts and the test has therefore been skipped. This usually means that either the package depends on external ressources that are not (or no longer) publically available or that the package (from an archived release) is broken beyond repair. There is no point in doing distupgrade tests if the package cannot be installed initially.

' piuparts/known_problems/unclassified_failures.conf0000644000000000000000000000047112567171321020073 0ustar # # report for failures that didn't match any known problem # PATTERN='QQQQQ_d195eab29de40b96b44ee8646c0d10d97171d454_dummy_pattern_QQQQQ' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with unclassified failures detected' HELPTEXT='

Please investigate and improve detection of known error types!

' piuparts/known_problems/undefined_symbol_inadequate_issue.conf0000644000000000000000000000063712567171321022472 0ustar # # detect packages with some inadequate tag from adequate # PATTERN='(FAIL|WARN): Running adequate resulted in .* undefined-symbol' WHERE='pass fail bugged affected' ISSUE=1 HEADER="Packages tagged 'undefined-symbol' by adequate" HELPTEXT="

Running adequate resulted in the package being tagged 'undefined-symbol' which indicates a bug.

" piuparts/known_problems/unknown_inadequate_issue.conf0000644000000000000000000000117712567171321020643 0ustar # # detect packages which have the string "Found unknown tags running adequate" in their logs # PATTERN='(FAIL|WARN): Found unknown tags running adequate' WHERE='pass fail bugged affected' ISSUE=1 HEADER='Packages which have logs with the string "Found unknown tags running adequate"' HELPTEXT='

Sometimes new types of problems are detected by adequate, which classifies them using tags. When this happens these new tags need to be made known to piuparts. Please notify piuparts-devel@lists.alioth.debian.org.

' piuparts/known_problems/unknown_purge_error.conf0000644000000000000000000000107412567171321017642 0ustar # # detect packages which try to overwrite other packages files # PATTERN='ERROR: Command failed .*.dpkg., .--purge., .--pending.]' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with failed logs because dpkg --purge --pending failed' HELPTEXT='

This is often because some dependencies have to be removed together and not seperatedly, and thus rather a bug in piuparts... but be careful, this list also includes failures due to "command not found"-error, which made the purge fail... so file those bugs first.

' piuparts/known_problems/unowned_files_after_purge_error.conf0000644000000000000000000000146012567171321022164 0ustar # # detect packages with unowned files after purge (policy 6.8) # PATTERN='not owned' EXCLUDE_PATTERN='/usr/share/mime/|usr/local/' WHERE='fail bugged affected' ISSUE=0 HEADER='Packages with unowned files after purge' HELPTEXT='

WARNING: this list might contain false positives. One group of them are packages with files in /usr/share/mime - those are bugs from shared-mime-info (#527063) and have been ignored for this list. There are probably others like this as well.

Packages with unowned files after purge (violating policy 6.8) see https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails

' piuparts/known_problems/unowned_files_after_purge_issue.conf0000644000000000000000000000144012567171321022161 0ustar # # detect packages with unowned files after purge (policy 6.8) # PATTERN='not owned' EXCLUDE_PATTERN="/usr/share/mime/|usr/local/" WHERE='pass' ISSUE=1 HEADER='Packages with unowned files after purge' HELPTEXT='

WARNING: this list might contain false positives. One group of them are packages with files in /usr/share/mime - those are bugs from shared-mime-info (#527063) and have been ignored for this list. There are probably others like this as well.

Packages with unowned files after purge (violating policy 6.8) see https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails

' piuparts/known_problems/unowned_lib_symlink_error.conf0000644000000000000000000000034712567171321021016 0ustar # # detect unowned symlinks in library directories # PATTERN='UNOWNED SYMLINK' WHERE='fail bugged affected' ISSUE=0 HEADER='Unowned symlinks is library directories' HELPTEXT='

These should rather be shipped in packages.

' piuparts/known_problems/unowned_lib_symlink_issue.conf0000644000000000000000000000032712567171321021013 0ustar # # detect unowned symlinks in library directories # PATTERN='UNOWNED SYMLINK' WHERE='pass' ISSUE=1 HEADER='Unowned symlinks is library directories' HELPTEXT='

These should rather be shipped in packages.

' piuparts/known_problems/used_exception_issue.conf0000644000000000000000000000040612567171321017754 0ustar # # report packages that used a piuparts exception to pass a test # PATTERN='piuparts exception for package' WHERE='pass' ISSUE=1 HEADER='Packages that used a piuparts exception' HELPTEXT='

These packages needed a piuparts exception to pass the test.

' piuparts/lib/0000755000000000000000000000000012567171321010366 5ustar piuparts/lib/read_config.sh0000644000000000000000000000376012567171321013170 0ustar # Copyright © 2011, 2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # Helper function for getting values from piuparts.conf. # Used by several master and slave scripts. # PIUPARTS_CONF=${PIUPARTS_CONF:-/etc/piuparts/piuparts.conf} [ -f "$PIUPARTS_CONF" ] || exit 0 # usage: get_config_value VARIABLE section key [default] get_config_value() { local section key value test -n "$1" && test "$1" = "$(echo "$1" | tr -c -d '[:alnum:]_')" || exit 1 section="$2" key="$3" # First select the [$section] block (\#^\[$section\]#) (use # as # marker because $section may contain slashes) up to the start of the # next section (/^\[/). The select the $key=value, this may be wrapped # with indented lines and comment lines embedded. The $key=value is # over once we hit the next key (or any line not starting with # or # whitespace. Throw away comments (/^#/d), the following key, remove # our $key= part, trim the value, remove empty lines, and print it. value="$(sed -rn '\#^\['"$section"'\]#,/^\[/ {/^'"$key"'\s*=/,/^[^ \t#]/ {/^#/d; /^'"$key"'\s*=|^\s/!d; s/^'"$key"'\s*=\s*//; s/^\s*//; s/\s*$//; /^$/d; p}}' "$PIUPARTS_CONF")" if [ -z "$value" ]; then if [ -n "${4+set}" ]; then value="$4" else echo "'$key' not set in section [$section] of $PIUPARTS_CONF, exiting." >&2 exit 1 fi fi eval "$1"='"$value"' } piuparts/master-bin/0000755000000000000000000000000013061276553011664 5ustar piuparts/master-bin/archive_old_logs.in0000755000000000000000000000435313061276553015527 0ustar #!/bin/sh # Copyright 2009-2011 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections # # archive old log files # TOTAL=0 OLDPWD=$(pwd) for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue cd $MASTER/$SECTION mkdir -p pass bugged affected fail mkdir -p archive/pass archive/bugged archive/affected archive/fail test -f archive/stamp || touch -d @0 archive/stamp # start at the epoch touch -d yesterday archive/stamp.new # look back one more day the next time we will be run OUTPUT="" # loop through all packages logs for PACKAGE in $(find pass/ fail/ bugged/ affected/ -name '*.log' -newer archive/stamp | cut -d"_" -f1 | cut -d"/" -f2 | sort -u) ; do # all logs except the last one (|sed '$d' deletes the last line) OLDLOGS=$( ls -tr1 --color=none bugged/${PACKAGE}_*.log affected/${PACKAGE}_*.log fail/${PACKAGE}_*.log pass/${PACKAGE}_*.log 2>/dev/null|sed '$d' ) if [ ! -z "$OLDLOGS" ] ; then if [ -z "$OUTPUT" ] ; then OUTPUT="yes" echo $SECTION fi for LOG in $OLDLOGS ; do TOTAL=$(($TOTAL + 1)) mv -v $LOG archive/$(echo $LOG|cut -d"/" -f1)/ done fi done NUM_CPU=$(grep -c ^processor /proc/cpuinfo) find archive/ -name '*.log' | xargs -P $NUM_CPU -n 1 -r xz -f if [ -n "$OUTPUT" ] ; then echo echo fi mv archive/stamp.new archive/stamp cd "$OLDPWD" done if [ "$TOTAL" -gt "0" ]; then echo "Archived $TOTAL old logfiles." fi piuparts/master-bin/detect_archive_issues.in0000755000000000000000000000537613061276553016576 0ustar #!/bin/sh # Copyright © 2009-2017 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections get_config_value URLBASE global urlbase https://piuparts.debian.org get_config_value DAYS global reschedule-untestable-days 7 STARTDATE=$(date -u +%s) # # detect packages which are/were untestable due to archive issue and mark them as such # LOGS=`mktemp` URLS=`mktemp` for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue mkdir -p $MASTER/$SECTION/fail/ $MASTER/$SECTION/untestable/ find $MASTER/$SECTION/fail -name '*.log' -mtime +1 | xargs -r \ grep -l -E "E: Broken packages|E: Unable to correct problems, you have held broken packages|E: Error, pkgProblemResolver::Resolve generated breaks" 2>/dev/null > $LOGS if [ -s $LOGS ]; then for package_log in $(cat $LOGS) do mv $package_log $MASTER/$SECTION/untestable/ done sed "s#$MASTER/$SECTION/fail#$URLBASE/$SECTION/untestable#" $LOGS >> $URLS fi done if [ -s $URLS ]; then date >> $MASTER/archive_issues.txt cat $URLS >> $MASTER/archive_issues.txt FINALDATE=$(date -u +%s) RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T) echo "Runtime: $RUNTIME" echo echo "Broken packages detected!" echo "(By grep'ing for" echo " 'E: Broken packages'," echo " 'E: Unable to correct problems, you have held broken packages'," echo " 'E: Error, pkgProblemResolver::Resolve generated breaks'" echo "in failed logs.)" echo echo 'The following packages have been moved to $section/untestable and will be' echo "tested again in $DAYS days." echo echo "Broken packages are usually a temporary problem in the archive and are" echo "caught by other tools like britney or https://qa.debian.org/dose/debcheck.html" echo echo "If it is always the same package failing, it's likely to be an issue in the" echo "package." echo grep -f $URLS $MASTER/archive_issues.txt | sort | uniq -c | sort -rn echo fi rm $LOGS $URLS piuparts/master-bin/detect_network_issues.in0000755000000000000000000000753313061276553016643 0ustar #!/bin/sh # Copyright 2009-2017 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2017 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections STARTDATE=$(date -u +%s) # # detect network/mirror problems # FILE=`mktemp` for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue for subdir in fail bugged affected ; do test -d $MASTER/$SECTION/$subdir || continue rgrep -l "Cannot initiate the connection to" $MASTER/$SECTION/$subdir >> $FILE rgrep -l "Hash Sum mismatch" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "Failed to fetch.*Could not resolve" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "Failed to fetch.*Something wicked happened resolving" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "Failed to fetch.*Unable to connect to" $MASTER/$SECTION/$subdir >> $FILE rgrep -l "E: Method http has died unexpectedly" $MASTER/$SECTION/$subdir >> $FILE rgrep -l "Some index files failed to download, they have been ignored, or old ones used instead." $MASTER/$SECTION/$subdir >> $FILE rgrep -l "Some index files failed to download. They have been ignored, or old ones used instead." $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "W: GPG error: .* Release: The following signatures were invalid: BADSIG" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "W: GPG error: .* InRelease: The following signatures were invalid: NODATA" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "E: The repository '.*' does no longer have a Release file." $MASTER/$SECTION/$subdir >> $FILE rgrep -l "E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "E: Version '.*' for '.*' was not found" $MASTER/$SECTION/$subdir >> $FILE rgrep -l 'E: Method file has died unexpectedly!' $MASTER/$SECTION/$subdir >> $FILE rgrep -l "E: Sub-process rred received a segmentation fault." $MASTER/$SECTION/$subdir >> $FILE rgrep -l -E "ERROR: Command failed \(status=-7\):.*'apt-get', 'update'" $MASTER/$SECTION/$subdir >> $FILE rgrep -l 'WARNING: The following packages cannot be authenticated!' $MASTER/$SECTION/$subdir >> $FILE # modified changelogs are usually caused by mirror pushes during the piuparts test rgrep -l -E ' /usr/share/doc/(.*)/changelog.*owned by: \1' $MASTER/$SECTION/$subdir >> $FILE done done if [ -s $FILE ] ; then FINALDATE=$(date -u +%s) RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T) echo "Runtime: $RUNTIME" echo echo "Network problems on detected!" echo "(By grep'ing for" echo " 'Cannot initiate the connection to'," echo " 'Failed to fetch'," echo " 'Some index files failed to download'," echo " 'Hash Sum mismatch'," echo " 'The following signatures were invalid'" echo "in failed logs.)" echo "Test failures due to modified /usr/share/doc/*/changelog.*" echo "are caused by mirror updates during the test." echo echo "The following logfiles have been deleted:" echo echo "----------------------------------------------------------------------" echo for LOG in $(sort -u $FILE) ; do rm -v $LOG done echo fi rm $FILE piuparts/master-bin/detect_piuparts_issues.in0000755000000000000000000001235313061276553017015 0ustar #!/bin/sh # Copyright © 2009-2017 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2017 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections STARTDATE=$(date -u +%s) # # detect piuparts problems # FILE=`mktemp` for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue for subdir in fail bugged affected ; do test -d $MASTER/$SECTION/$subdir || continue rgrep -l 'PIUPARTS OUTPUT INCOMPLETE' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -E "tar( \(child\))?: .*.tar.gz: Cannot open: No such file or directory" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "tar: Error is not recoverable: exiting now" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "tar: .*: Wrote only .* of .* bytes" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "gzip: stdin: invalid compressed data--crc error" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "ERROR:.*:Temporary directory is not a directory" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -i -e "No space left on device" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "E: You don't have enough free space in /var/cache/apt/archives/" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "E: Failed to write temporary StateFile /var/lib/apt/extended_states.tmp" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "passwd: Authentication token manipulation error" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -E "(chfn|groupadd|useradd): cannot lock /etc/(group|gshadow|passwd|shadow|subuid); try again later." $MASTER/$SECTION/$subdir >> $FILE rgrep -l -E "(groupadd|useradd): failure while writing changes to /etc/(group|passwd|shadow)" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -E "(groupadd|useradd): (unable to lock|cannot rewrite) (group|password) file" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "E: Sub-process Popen returned an error code (2)" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "ERROR: Command failed (status=-7): .* 'apt-cache'" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "^RuntimeError: maximum recursion depth exceeded while calling a Python object" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "^Traceback" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "^OSError:" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "^IndentationError:" $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e "^SyntaxError:" $MASTER/$SECTION/$subdir >> $FILE # Python errors from maintainer scripts etc. would be indented by 2 spaces, so don't match here. rgrep -l -e 'update-binfmts: warning: unable to close /proc/sys/fs/binfmt_misc/register: Invalid argument' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'userdel: user .* is currently logged in' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -i -E 'invoke-rc.d: initscript mysql, action "(start|restart)" failed.' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -i -e 'invoke-rc.d: initscript postgresql, action "start" failed.' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'The PostgreSQL server failed to start.' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -i -e 'FATAL: could not create shared memory segment: No space left on device' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'No database found online on port 5432' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'unable to connect to postgresql server' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'Failed to connect to postgres.' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'psql: FATAL: password authentication failed for user' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'createuser: could not connect to database postgres: could not connect to server: No such file or directory' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'Non-default port in /etc/postgresql/.*/main/postgresql.conf' $MASTER/$SECTION/$subdir >> $FILE rgrep -l -e 'Firebird .* server already running.' $MASTER/$SECTION/$subdir >> $FILE find $MASTER/$SECTION/$subdir -name '*.log' -size 0 >> $FILE done done if [ -s $FILE ] ; then FINALDATE=$(date -u +%s) RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T) echo "Runtime: $RUNTIME" echo echo "piuparts problem detected!" echo "(By grep'ing for 'tar: .*.tar.gz: Cannot open: No such file or directory'" echo "and for some python errors and tracebacks in failed logs.)" echo echo "Please review the following logfiles/packages." echo "If it is always the same package failing, it's likely to be an issue in the" echo "package." echo sort -u $FILE | xargs rm -v echo echo "----------------------------------------------------------------------" echo fi rm $FILE piuparts/master-bin/detect_well_known_errors.py0000755000000000000000000001507112762537303017350 0ustar #!/usr/bin/python # -*- coding: utf-8 -*- # Copyright 2013 David Steele (dsteele@gmail.com) # Copyright © 2014 Andreas Beckmann (anbe@debian.org) # # This file is part of Piuparts # # Piuparts is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # Piuparts is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, see . import ConfigParser import os import sys import time import logging import argparse import piupartslib from piupartslib.conf import MissingSection from piupartslib.dwke import * CONFIG_FILE = "/etc/piuparts/piuparts.conf" KPR_DIRS = ('pass', 'bugged', 'affected', 'fail') class WKE_Config(piupartslib.conf.Config): """Configuration parameters for Well Known Errors""" def __init__(self): self.section = 'global' piupartslib.conf.Config.__init__(self, self.section, { "sections": "report", "master-directory": ".", "known-problem-directory": "@sharedir@/piuparts/known_problems", }, "") def setup_logging(log_level): logger = logging.getLogger() logger.setLevel(log_level) handler = logging.StreamHandler(sys.stdout) logger.addHandler(handler) def write_file(filename, contents): with file(filename, "w") as f: f.write(contents) def mtime(path): return os.path.getmtime(path) def clean_cache_files(logdict, cachedict, recheck=False, recheck_failed=False, skipnewer=False): """Delete files in cachedict if the corresponding logdict file is missing or newer""" count = 0 for pkgspec in cachedict: try: if pkgspec not in logdict \ or (mtime(logdict[pkgspec]) > mtime(cachedict[pkgspec]) and not skipnewer)\ or get_where(logdict[pkgspec]) != get_where(cachedict[pkgspec])\ or recheck\ or (recheck_failed and not get_where(cachedict[pkgspec]) in ['pass']): os.remove(cachedict[pkgspec]) count = count + 1 except (IOError, OSError): # logfile may have disappeared pass return count def make_kprs(logdict, kprdict, problem_list): """Create kpr files, as necessary, so every log file has one kpr entries are e.g. fail/xorg-docs_1:1.6-1.log broken_symlinks_error.conf""" needs_kpr = set(logdict.keys()).difference(set(kprdict.keys())) for pkg_spec in needs_kpr: logpath = logdict[pkg_spec] try: lb = open(logpath, 'r') logbody = lb.read() lb.close() where = get_where(logpath) kprs = "" for problem in problem_list: if problem.has_problem(logbody, where): kprs += "%s/%s.log %s\n" % (where, pkg_spec, problem.name) if not where in ['pass'] and not len(kprs): kprs += "%s/%s.log %s\n" % (where, pkg_spec, "unclassified_failures.conf") write_file(get_kpr_path(logpath), kprs) except IOError: logging.error("File error processing %s" % logpath) return len(needs_kpr) def process_section(section, config, problem_list, recheck=False, recheck_failed=False, pkgsdb=None): """ Update .bug and .kpr files for logs in this section """ sectiondir = os.path.join(config['master-directory'], section) workdirs = [os.path.join(sectiondir, x) for x in KPR_DIRS] if not os.access(sectiondir, os.F_OK): raise MissingSection("", section) [os.mkdir(x) for x in workdirs if not os.path.exists(x)] logdict = get_file_dict(workdirs, LOG_EXT) kprdict = get_file_dict(workdirs, KPR_EXT) bugdict = get_file_dict(workdirs, BUG_EXT) del_cnt = clean_cache_files(logdict, kprdict, recheck, recheck_failed) clean_cache_files(logdict, bugdict, skipnewer=True) kprdict = get_file_dict(workdirs, KPR_EXT) add_cnt = make_kprs(logdict, kprdict, problem_list) failures = FailureManager(logdict) return (del_cnt, add_cnt, failures) def detect_well_known_errors(sections, config, problem_list, recheck, recheck_failed): for section in sections: try: logging.info(time.strftime("%a %b %2d %H:%M:%S %Z %Y", time.localtime())) logging.info("%s:" % section) (del_cnt, add_cnt, failures) = \ process_section(section, config, problem_list, recheck, recheck_failed) logging.info("parsed logfiles: %d removed, %d added" % (del_cnt, add_cnt)) for prob in problem_list: pcount = len(failures.filtered(prob.name)) if pcount: logging.info("%7d %s" % (pcount, prob.name)) except MissingSection: pass logging.info(time.strftime("%a %b %2d %H:%M:%S %Z %Y", time.localtime())) if __name__ == '__main__': setup_logging(logging.DEBUG) parser = argparse.ArgumentParser( description="Detect well known errors", epilog=""" This script processes all log files against defined "known_problem" files, caching the problems found, by package, into ".kpr" files. """) parser.add_argument('sections', nargs='*', metavar='SECTION', help="limit processing to the listed SECTION(s)") parser.add_argument('--recheck', dest='recheck', action='store_true', help="recheck all log files (delete cache)") parser.add_argument('--recheck-failed', dest='recheck_failed', action='store_true', help="recheck failed log files (delete cache)") args = parser.parse_args() conf = WKE_Config() conf.read(CONFIG_FILE) sections = args.sections if not sections: sections = conf['sections'].split() problem_list = create_problem_list(conf['known-problem-directory']) detect_well_known_errors(sections, conf, problem_list, args.recheck, args.recheck_failed) # vi:set et ts=4 sw=4 : piuparts/master-bin/gather_bts_stats.in0000755000000000000000000000460712567171321015563 0ustar #!/bin/bash set -e # Copyright 2013,2014 Holger Levsen (holger@layer-acht.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory BTS_STATS="$MASTER/bts_stats.txt" # exit if master-directory doesn't exist or if devscripts package is not installed test -n "$MASTER" || $(which bts) || exit 0 # "bts select" needs libsoap-lite-perl too dpkg -l libsoap-lite-perl >/dev/null 2>&1 || exit 0 # only run once a day TODAY=$(date +%Y%m%d) if $(grep -q ^$TODAY $BTS_STATS 2>/dev/null) ; then exit 0 fi # query bts RC="severity:serious severity:grave severity:critical" OTHER="severity:wishlist severity:minor severity:normal severity:important" DONE_RC=$(bts select usertag:piuparts users:debian-qa@lists.debian.org status:done archive:both $RC 2>/dev/null|wc -l) OPEN_RC=$(bts select usertag:piuparts users:debian-qa@lists.debian.org status:open status:forwarded $RC 2>/dev/null|wc -l) DONE_OTHER=$(bts select usertag:piuparts users:debian-qa@lists.debian.org status:done archive:both $OTHER 2>/dev/null|wc -l) OPEN_OTHER=$(bts select usertag:piuparts users:debian-qa@lists.debian.org status:open status:forwarded $OTHER 2>/dev/null|wc -l) # test if both values are integers if ! ( [[ $DONE_RC =~ ^-?[0-9]+$ ]] && [[ $OPEN_RC =~ ^-?[0-9]+$ ]] && [[ $DONE_OTHER =~ ^-?[0-9]+$ ]] && [[ $OPEN_OTHER =~ ^-?[0-9]+$ ]] ) ; then echo "Non-integer value detected, exiting." echo "DONE_RC: $DONE_RC OPEN_RC: $OPEN_RC" echo "DONE_OTHER: $DONE_OTHER OPEN_OTHER: $OPEN_OTHER" exit 1 fi # init file if needed if [ ! -f $BTS_STATS ] ; then echo "date, non-RC done, non-RC open, RC done, RC open" > $BTS_STATS fi # finally, write stats echo "$TODAY, $DONE_OTHER, $OPEN_OTHER, $DONE_RC, $OPEN_RC" >> $BTS_STATS piuparts/master-bin/generate_daily_report.in0000755000000000000000000000637513061276553016601 0ustar #!/bin/sh # Copyright © 2011-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections get_config_value HTDOCS global output-directory get_config_value URLBASE global urlbase https://piuparts.debian.org get_config_value PYTHONPATH global PYTHONPATH '' export PYTHONPATH LOCKFILE=$HTDOCS/daily.lock if [ -e $LOCKFILE ]; then echo "daily.lock exists:" ls -l $LOCKFILE exit 1 fi date > $LOCKFILE R_STARTDATE=$(date -u +%s) # # various scripts to detect issues and cleanup # OUTPUT=$(mktemp) for SCRIPT in detect_network_issues detect_piuparts_issues archive_old_logs report_newly_bugged_packages ; do STARTDATE=$(date -u +%s) @sharedir@/piuparts/master/$SCRIPT 2>&1 > $OUTPUT FINALDATE=$(date -u +%s) if [ -s $OUTPUT ] ; then RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T) ( echo "Runtime: $RUNTIME" ; echo ; cat $OUTPUT ) | mail -s $SCRIPT $LOGNAME fi done rm $OUTPUT # # piuparts-report starts here # DAILYREPORT=$HTDOCS/dailyreport.txt >$DAILYREPORT # Failures of the day it should be, the rest is boring. FAILURESTAMP=$HTDOCS/last-failure-report.stamp test -f $FAILURESTAMP || touch -d @0 $FAILURESTAMP # start at the epoch touch $FAILURESTAMP.new # for the next report echo "New failures:" >> $DAILYREPORT for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue for DIRECTORY in fail bugged affected untestable ; do mkdir -p $MASTER/$SECTION/$DIRECTORY done find $MASTER/$SECTION/fail $MASTER/$SECTION/bugged $MASTER/$SECTION/affected $MASTER/$SECTION/untestable \ -type f -name '*.log' -newer $FAILURESTAMP -exec ls -1 {} + 2>/dev/null done | sed s#^$MASTER#$URLBASE# >> $DAILYREPORT echo "" >> $DAILYREPORT date >> $DAILYREPORT @sharedir@/piuparts/master/detect_well_known_errors 2>&1 >> $DAILYREPORT echo "" >> $DAILYREPORT date >> $DAILYREPORT nice \ @sharedir@/piuparts/piuparts-report \ >> $DAILYREPORT 2>&1 date >> $DAILYREPORT echo "" >> $DAILYREPORT date >> $DAILYREPORT echo "Expiring old .html files:" >> $DAILYREPORT find $HTDOCS -name '* *.html' -mtime +30 -ls -delete >> $DAILYREPORT expire=$(mktemp) find $HTDOCS -name '*.html' -mtime +30 | head -n 500 > $expire if [ -s $expire ]; then ls -ld $(cat $expire) >> $DAILYREPORT rm -f $(cat $expire) >> $DAILYREPORT fi rm $expire FINALDATE=$(date -u +%s) RUNTIME=$(date -u -d "0 $FINALDATE seconds - $R_STARTDATE seconds" +%T) ( echo "Runtime: $RUNTIME" ; echo ; cat $DAILYREPORT ) | mail -s piuparts-report $LOGNAME mv $FAILURESTAMP.new $FAILURESTAMP rm -f $LOCKFILE piuparts/master-bin/master_cleanup.in0000755000000000000000000000217512567171321015223 0ustar #!/bin/sh set -e # Copyright © 2012 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # cleanup $HTDOCS/daily.lock # # - this should only be run (automatically) on boot # . @sharedir@/piuparts/lib/read_config.sh get_config_value HTDOCS global output-directory LOCKFILE=$HTDOCS/daily.lock if [ -e $LOCKFILE ]; then if pgrep -f generate_daily_report || pgrep -f piuparts-report ; then echo "daily processing is running" else rm -f -v $LOCKFILE fi fi piuparts/master-bin/prepare_backup.in0000755000000000000000000000331512762537303015204 0ustar #!/bin/sh set -e # Copyright 2009,2014 Holger Levsen (holger@layer-acht.org) # Copyright © 2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # backup statistiscs files to a single directory for DSA to backup # piuparts.d.o's state is not backupped # (currently it takes one month to run a full piuparts test on a suite # which produces a gigabyte of logfiles to be saved, for basically # not much value) # . @sharedir@/piuparts/lib/read_config.sh get_config_value SECTIONS global sections get_config_value MASTER global master-directory get_config_value HTDOCS global output-directory get_config_value BACKUPDIR global backup-directory '' copy_to_backupdir() { if [ -f $MASTER/$1 ]; then cp $MASTER/$1 $BACKUPDIR/$1 else echo "Warning: $MASTER/$1 does not exist." fi } test -n "$BACKUPDIR" || exit 0 copy_to_backupdir archive_issues.txt copy_to_backupdir bts_stats.txt for SECTION in $SECTIONS ; do mkdir -p $BACKUPDIR/$SECTION copy_to_backupdir $SECTION/submissions.txt copy_to_backupdir $SECTION/counts.txt done piuparts/master-bin/reclassify_bugged.in0000755000000000000000000000244112567171321015676 0ustar #!/bin/sh set -e # Copyright © 2012 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections OLDPWD=$(pwd) for SECTION in $SECTIONS ; do get_config_value KEEP_BUGGED $SECTION keep-bugged no if [ "$KEEP_BUGGED" = "no" ] && [ -d $MASTER/$SECTION/fail ] && [ -d $MASTER/$SECTION/bugged ]; then cd $MASTER/$SECTION mv bugged/*.log bugged/*.bug fail/ 2>/dev/null mv affected/*.log affected/*.bug fail/ 2>/dev/null cd "$OLDPWD" fi done @sharedir@/piuparts/master/report_newly_bugged_packages piuparts/master-bin/report_newly_bugged_packages.in0000755000000000000000000000262512567171321020125 0ustar #!/bin/sh # Copyright © 2011-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections get_config_value PYTHONPATH global PYTHONPATH '' get_config_value DEBEMAIL global bts-from '' export PYTHONPATH OLDPWD=$(pwd) OUTPUT=$(mktemp) for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue cd $MASTER/$SECTION get_config_value DISTROS $SECTION upgrade-test-distros '' distro=${DISTROS##* } distro=${distro%-proposed} timeout 60m @sharedir@/piuparts/piuparts-analyze $distro 2>&1 > $OUTPUT if [ -s $OUTPUT ]; then echo $SECTION cat $OUTPUT echo fi cd "$OLDPWD" done rm $OUTPUT piuparts/master-bin/report_stale_reserved_packages.in0000755000000000000000000000271512567171321020461 0ustar #!/bin/sh # Copyright 2009,2010 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2012 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections # # find packages which have been reserved for more than one day and unschedule them # DAYS=1 LOGS=`mktemp` for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION/reserved || continue find $MASTER/$SECTION/reserved/ -mtime +$DAYS -name "*.log" 2>/dev/null >> $LOGS done if [ -s $LOGS ] ; then echo "Stale reserved packages detected, which have been reserved more then $DAYS days ago!" echo "These packages have been unscheduled." echo for package_log in $(cat $LOGS) ; do rm -fv $package_log done fi rm $LOGS piuparts/master-bin/report_untestable_packages.in0000755000000000000000000000304212567171321017612 0ustar #!/bin/sh # Copyright 2009 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2012 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections get_config_value DAYS global reschedule-untestable-days 7 # # find packages which have been in untestable for more than $DAYS days and reschedule them for testing # LOGS=`mktemp` for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION/untestable || continue find $MASTER/$SECTION/untestable/ -mtime +$DAYS -name "*.log" 2>/dev/null >> $LOGS done if [ -s $LOGS ] ; then echo "Untestable packages detected, which have been tested more than $DAYS days ago!" echo "These packages have been rescheduled for piuparts testing." echo for package_log in $(cat $LOGS) ; do rm -fv $package_log done fi rm $LOGS piuparts/master-bin/reschedule_oldest_logs.in0000755000000000000000000001363513061276553016750 0ustar #!/bin/sh # Copyright © 2009-2017 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2012 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections STARTDATE=$(date -u +%s) OUTPUT=$(mktemp) # # reschedule 200 oldest log files, if they are older than 180 days # reschedule 25 oldest fail log files, if they are older than 30 days # delete $reschedule-old-count oldest logfiles, if they are scheduled # for recycling and older than $expire-old-days # and $expire-old-days > $reschedule-old-days # get_config_value EXPIRE_AGE global expire-old-days 0 get_config_value AGE global reschedule-old-days 180 get_config_value COUNT global reschedule-old-count 200 get_config_value EXPIRE_FAIL_AGE global expire-fail-days 0 get_config_value FAIL_AGE global reschedule-fail-days 30 get_config_value FAIL_COUNT global reschedule-fail-count 25 get_config_value AUTO_RESCHEDULE global auto-reschedule yes RECENT=$(mktemp) touch -d "2 days ago" "$RECENT" is_recent() { local stamp stamp="$1" if [ -f "$stamp" ] && [ "$stamp" -nt "$RECENT" ]; then return 0 fi return 1 } TMPLIST=$(mktemp) list_logs() { __AGE="$1" __COUNT="$2" shift 2 find "$@" -name "*.log" -mtime +$__AGE \ | tac \ | xargs --no-run-if-empty -n99999 -s1999999 ls -dt \ > $TMPLIST tail -n $__COUNT $TMPLIST } TOTAL=0 TOTAL_EXPIRED=0 UNSCHEDULE=0 TOTAL_QUEUED=0 LOGS=$(mktemp) OBSOLETE=$(mktemp) EXPIRED=$(mktemp) UNSORTED=$(mktemp) QUEUED=$(mktemp) OLDPWD=$(pwd) for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue cd $MASTER/$SECTION mkdir -p pass fail bugged affected recycle recycle_was_idle= find recycle/ -name '*.log' > $QUEUED if [ ! -s $QUEUED ]; then recycle_was_idle="empty" elif [ -f "recycle.stamp" ] && [ "recycle.stamp" -nt "recycle" ]; then recycle_was_idle="idle" fi # Clean up obsolete rescheduling requests for log in $(find recycle/ -name '*.log' | cut -d"/" -f2) ; do for dir in pass bugged affected fail ; do test ! -e "$dir/$log" || continue 2 done echo "recycle/$log" done | sort > $OBSOLETE # Reschedule old logs >$LOGS >$EXPIRED get_config_value RESCHEDULE $SECTION auto-reschedule $AUTO_RESCHEDULE if [ "$RESCHEDULE" = "yes" ]; then get_config_value _EXPIRE_AGE $SECTION expire-old-days $EXPIRE_AGE get_config_value _AGE $SECTION reschedule-old-days $AGE get_config_value _COUNT $SECTION reschedule-old-count $COUNT get_config_value _EXPIRE_FAIL_AGE $SECTION expire-fail-days $EXPIRE_FAIL_AGE get_config_value _FAIL_AGE $SECTION reschedule-fail-days $FAIL_AGE get_config_value _FAIL_COUNT $SECTION reschedule-fail-count $FAIL_COUNT # FIXME: we ignore bugged here - ptyhon-bts is really the way to go >$UNSORTED if [ "$_EXPIRE_AGE" -gt "$_AGE" ]; then if [ -n "$recycle_was_idle" ]; then list_logs $_EXPIRE_AGE $_COUNT pass fail affected >> $UNSORTED fi fi if [ "$_EXPIRE_FAIL_AGE" -gt "$_FAIL_AGE" ]; then list_logs $_EXPIRE_FAIL_AGE $_FAIL_COUNT fail affected >> $UNSORTED fi for log in $(sort -u $UNSORTED) ; do # the log needs to be scheduled for recycling before it gets expired test -f "recycle/${log#*/}" && echo "$log" done > $EXPIRED >$UNSORTED if is_recent idle.stamp ; then list_logs $_AGE $_COUNT pass fail affected >> $UNSORTED fi list_logs $_FAIL_AGE $_FAIL_COUNT fail affected >> $UNSORTED for log in $(sort -u $UNSORTED) ; do # skip if already scheduled test -f "recycle/${log#*/}" || echo "$log" done > $LOGS fi find recycle/ -name '*.log' > $QUEUED if [ -s $LOGS ] || [ -s $OBSOLETE ] || [ -s $EXPIRED ] || [ -s $QUEUED ] ; then echo "#########################################################" >> $OUTPUT echo >> $OUTPUT RCOUNT=$(wc -l $LOGS | awk '{ print $1 }') TOTAL=$(($TOTAL + $RCOUNT)) ECOUNT=$(wc -l $EXPIRED | awk '{ print $1 }') TOTAL_EXPIRED=$(($TOTAL_EXPIRED + $ECOUNT)) UCOUNT=$(wc -l $OBSOLETE | awk '{ print $1 }') UNSCHEDULE=$(($UNSCHEDULE + $UCOUNT)) echo "$SECTION: $RCOUNT rescheduled, $ECOUNT expired, $UCOUNT obsolete${recycle_was_idle:+ (recycle-was-${recycle_was_idle})}" >> $OUTPUT if [ -s $LOGS ]; then ls -dtl $(cat $LOGS) >> $OUTPUT ln -f $(cat $LOGS) recycle/ fi if [ -s $EXPIRED ]; then ls -dtl $(cat $EXPIRED) >> $OUTPUT rm -fv $(cat $EXPIRED) >> $OUTPUT fi if [ -s $OBSOLETE ]; then rm -fv $(cat $OBSOLETE) >> $OUTPUT fi find recycle/ -name '*.log' > $QUEUED NUM_QUEUED=$(wc -l < $QUEUED) TOTAL_QUEUED=$(($TOTAL_QUEUED + $NUM_QUEUED)) echo "queued: $NUM_QUEUED$(is_recent idle.stamp || echo " not-idle")" >> $OUTPUT echo >> $OUTPUT fi cd $OLDPWD done rm $RECENT rm $TMPLIST rm $LOGS rm $OBSOLETE rm $EXPIRED rm $UNSORTED rm $QUEUED FINALDATE=$(date -u +%s) RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T) echo "Runtime: $RUNTIME" echo if [ "$TOTAL" -gt "0" ]; then echo "Rescheduled $TOTAL logs." fi if [ "$TOTAL_EXPIRED" -gt "0" ]; then echo "Deleted $TOTAL_EXPIRED expired logs." fi if [ "$UNSCHEDULE" -gt "0" ]; then echo "Cancelled $UNSCHEDULE outdated rescheduling requests." fi if [ "$TOTAL_QUEUED" -gt "0" ]; then echo "Queued logs: $TOTAL_QUEUED" fi echo cat $OUTPUT rm -f $OUTPUT piuparts/master-bin/reschedule_piuparts_tests.in0000755000000000000000000000473513042113740017507 0ustar #!/bin/sh set -e # Copyright © 2013-2017 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value SECTIONS global sections TESTING="stretch" dryrun="" current="" longterm="" pass="" for arg in "$@" do case "$arg" in --pass) shift pass="pass" ;; --current) shift current="yes" ;; --longterm) shift longterm="yes" ;; --dryrun) shift dryrun="yes" ;; *) break ;; esac done test -n "$1" || exit 1 subdirs="$pass fail bugged affected" pattern=' ('"$1"')\.conf$' # # mark logs matching kpr tag $1 for recycling # TOTAL=0 OLDPWD=$(pwd) LIST=$(mktemp) for SECTION in $SECTIONS ; do test -d $MASTER/$SECTION || continue go= if [ -z "$current$longterm" ]; then go=yes else case "$SECTION" in # don't catch old*stable222testing *$TESTING*|testing*|*sid*|*experimental*) if [ "$current" = "yes" ]; then go=yes fi ;; *222testing*) if [ "$longterm" = "yes" ]; then go=yes fi ;; esac fi if [ "$go" != "yes" ]; then echo "Skip: $SECTION" continue fi cd $MASTER searchdirs="" for dir in $subdirs ; do searchdirs="$searchdirs $SECTION/$dir" done mkdir -p $searchdirs $SECTION/recycle grep -rlE "$pattern" --include '*.kpr' $searchdirs | sed 's/\.kpr$/.log/' > $LIST for log in $(cat $LIST) do if [ -f "$log" ] && [ ! -f "$SECTION/recycle/$(basename $log)" ]; then TOTAL=$(( $TOTAL + 1 )) if [ -n "$dryrun" ]; then echo "candidate: $log" else ln -fv "$log" $SECTION/recycle/ || true fi fi done cd "$OLDPWD" done rm -f $LIST if [ "$TOTAL" -gt "0" ]; then if [ -n "$dryrun" ]; then echo "Found $TOTAL candidate logfiles for recycling." else echo "Marked $TOTAL logfiles for recycling." fi fi piuparts/nosetests.cfg0000644000000000000000000000035112567171321012327 0ustar [nosetests] # create test coverage report with-coverage=1 with-coverage=1 cover-html=1 cover-html-dir=build/html cover-package=piupartslib,piuparts #cover-inclusive=1 # run doctests with-doctest=1 # catch all tests # all-modules=1 piuparts/piuparts-analyze.py0000644000000000000000000002414613061276553013514 0ustar #!/usr/bin/python # -*- coding: utf-8 -*- # # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright 2011 Mika Pflüger (debian@mikapflueger.de) # Copyright © 2012-2017 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """Analyze failed piuparts logs and move them around when the errors are known. This program looks at piuparts log files in ./fail, and queries the bts to find out if bugs have been filed already. If so, it moves them to ./bugged. It tries to detect if new versions of bugged packages are uploaded without solving the bug and will then update the bts with the new found versions, and copy the headers of the log in ./fail to the one in ./bugged and vice versa. It will then move the failed log to ./bugged as well. """ import os import sys import re import shutil import subprocess import debianbts import apt_pkg apt_pkg.init_system() error_pattern = re.compile(r"(?<=\n).*error.*\n?", flags=re.IGNORECASE) chroot_pattern = re.compile(r"tmp/tmp.*?'") distro = "?" def find_logs(directory): return [os.path.join(directory, x) for x in os.listdir(directory) if x.endswith(".log")] def find_bugged_logs(failed_log): package = package_name(failed_log) pat = "/" + package + "_" return [x for x in find_logs("bugged") + find_logs("affected") if pat in x] def package_name(log): return os.path.basename(log).split("_", 1)[0] def package_version(log): return os.path.basename(log).split("_", 1)[1].rstrip('.log') def package_source_version(log): version = package_version(log) possible_binnmu_part = version.rsplit('+', 1)[-1] if possible_binnmu_part.startswith('b') and possible_binnmu_part[1:].isdigit(): # the package version contains a binnmu-part which is not part of the source version # and therefore not accepted/tracked by the bts. Remove it. version = version.rsplit('+', 1)[0] return version def extract_errors(log): """This pretty stupid implementation is basically just 'grep -i error', and then removing the timestamps and the name of the chroot and the package version itself.""" f = open(log) data = f.read() f.close() whole = '' pversion = package_version(log) for match in error_pattern.finditer(data): text = match.group() # Get rid of timestamps if text[:1].isdigit(): text = text.split(" ", 1)[1] # Get rid of chroot names if 'tmp/tmp' in text: text = re.sub(chroot_pattern, "chroot'", text) # Get rid of the package version text = text.replace(pversion, '') whole += text return whole def extract_headers(log): f = open(log) data = f.read() f.close() headers = [] headers = data.partition("\nExecuting:")[0] if headers and not headers.endswith("\n"): headers += "\n" return headers def prepend_to_file(filename, data): f = file(filename, "r") old_data = f.read() f.close() f = file(filename + ".tmp", "w") f.write(data) f.write(old_data) f.close() shutil.copymode(filename, filename + ".tmp") os.rename(filename, filename + "~") os.rename(filename + ".tmp", filename) os.remove(filename + "~") def get_bug_versions(bug): """Gets a list of only the version numbers for which the bug is found. Newest versions are returned first.""" # debianbts returns it in the format package/1.2.3 or 1.2.3 which will become 1.2.3 versions = [] for found_version in debianbts.get_status(bug)[0].found_versions: v = found_version.rsplit('/', 1)[-1] if v == "None": # only allow the distro-qualified "$distro/None" version if found_version == distro + "/" + v: versions.append(v) else: versions.append(v) return list(reversed(sorted(versions, cmp=apt_pkg.version_compare))) or ['~'] def write_bug_file(failed_log, bugs): if bugs: f = file(os.path.splitext(failed_log)[0] + '.bug', "w") for bug in bugs: f.write('#%s\n' % (bug, bug)) f.close() def move_to_bugged(failed_log, bugged="bugged", bug=None): print("Moving %s to %s (#%s)" % (failed_log, bugged, bug)) os.rename(failed_log, os.path.join(bugged, os.path.basename(failed_log))) if bug is not None: write_bug_file(os.path.join(bugged, os.path.basename(failed_log)), [bug]) def mark_bugged_version(failed_log, bugged_log): """Copies the headers from the old log to the new log and vice versa and moves the new log to bugged. Removes the old log in bugged.""" bugged_headers = extract_headers(bugged_log) failed_headers = extract_headers(failed_log) prepend_to_file(failed_log, bugged_headers) prepend_to_file(bugged_log, failed_headers) move_to_bugged(failed_log) def bts_update_found(bugnr, newversion): if "DEBEMAIL" in os.environ and os.environ["DEBEMAIL"]: # subprocess.check_call(('bts', 'found', bugnr, newversion)) print(' '.join(('bts', 'found', str(bugnr), newversion))) def mark_logs_with_reported_bugs(): for failed_log in find_logs("fail") + find_logs("untestable"): try: pname = package_name(failed_log) pversion = package_source_version(failed_log) try: failed_errors = extract_errors(failed_log) except IOError: print('IOError while processing %s' % failed_log) continue moved = False abugs = piuparts_bugs_affecting(pname) bugs = piuparts_bugs_in(pname) for bug in abugs + bugs: if moved: break if bug in abugs: bugged = "affected" else: bugged = "bugged" found_versions = get_bug_versions(bug) if pversion in found_versions: move_to_bugged(failed_log, bugged, bug) moved = True break for bug_version in found_versions: # print('DEBUG: %s/%s #%d %s' % (pname, pversion, bug, bug_version)) if apt_pkg.version_compare(pversion, bug_version) > 0: # pversion > bug_version bugged_logs = find_bugged_logs(failed_log) if not bugged_logs and not moved: print('%s/%s: Maybe the bug was filed earlier: https://bugs.debian.org/%d against %s/%s' % (pname, pversion, bug, pname, bug_version)) break for bugged_log in bugged_logs: old_pversion = package_source_version(bugged_log) bugged_errors = extract_errors(bugged_log) if (apt_pkg.version_compare(old_pversion, bug_version) == 0 # old_pversion == bug_version and failed_errors == bugged_errors): # a bug was filed for an old version of the package, # and the errors were the same back then - assume it is the same bug. if not moved: mark_bugged_version(failed_log, bugged_log) moved = True bts_update_found(bug, pversion) break if not moved: write_bug_file(failed_log, abugs + bugs) except: print('ERROR processing %s' % failed_log) print sys.exc_info()[0] def report_packages_with_many_logs(): failed_logs = find_logs("fail") packages = {} for failed_log in failed_logs: package = package_name(failed_log) packages[package] = packages.get(package, []) + [failed_log] for package, failed_logs in packages.iteritems(): printed = False if len(failed_logs) > 1: print "Many failures:" for failed_log in failed_logs: print " ", failed_log printed = True bugged_logs = find_bugged_logs(failed_logs[0]) if bugged_logs: print "Already bugged?" for failed_log in failed_logs + bugged_logs: print " ", failed_log printed = True if printed: print piuparts_usertags_cache = None def all_piuparts_bugs(): global piuparts_usertags_cache if piuparts_usertags_cache is None: piuparts_usertags_cache = debianbts.get_usertag("debian-qa@lists.debian.org", 'piuparts')['piuparts'] return piuparts_usertags_cache def piuparts_bugs_in(package): bugs = debianbts.get_bugs('package', package, 'bugs', all_piuparts_bugs(), 'archive', 'both') bugs += debianbts.get_bugs('package', 'src:' + package, 'bugs', all_piuparts_bugs(), 'archive', 'both') return sorted(set(bugs), reverse=True) def piuparts_bugs_affecting(package): bugs = debianbts.get_bugs('affects', package, 'bugs', all_piuparts_bugs(), 'archive', 'both') bugs += debianbts.get_bugs('affects', 'src:' + package, 'bugs', all_piuparts_bugs(), 'archive', 'both') return sorted(set(bugs), reverse=True) def main(): if len(sys.argv) > 1: global distro distro = sys.argv[1] mark_logs_with_reported_bugs() report_packages_with_many_logs() if __name__ == "__main__": main() # vi:set et ts=4 sw=4 : piuparts/piuparts-master-backend.py0000644000000000000000000003461312762537303014731 0ustar #!/usr/bin/python # -*- coding: utf-8 -*- # # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright © 2011-2015 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """Distributed piuparts processing, master program Lars Wirzenius """ import sys import logging import ConfigParser import os import fcntl import time import random from urllib2 import URLError import piupartslib from piupartslib.packagesdb import LogfileExists from piupartslib.conf import MissingSection CONFIG_FILE = "/etc/piuparts/piuparts.conf" DISTRO_CONFIG_FILE = "/etc/piuparts/distros.conf" log_handler = None def setup_logging(log_level, log_file_name): logger = logging.getLogger() global log_handler logger.removeHandler(log_handler) if log_file_name: log_handler = logging.FileHandler(log_file_name) else: log_handler = logging.StreamHandler(sys.stderr) logger.addHandler(log_handler) logger.setLevel(log_level) def timestamp(): return time.strftime("[%Y-%m-%d %H:%M:%S]") class Config(piupartslib.conf.Config): def __init__(self, section="master", defaults_section=None): piupartslib.conf.Config.__init__(self, section, { "log-file": None, "master-directory": ".", "proxy": None, "mirror": None, "distro": None, "area": None, "arch": None, "upgrade-test-distros": None, "depends-sections": None, }, defaults_section=defaults_section) class CommandSyntaxError(Exception): def __init__(self, msg): self.args = msg, class ProtocolError(Exception): def __init__(self): self.args = "EOF, missing space in long part, or other protocol error", class Protocol: def __init__(self, input, output): self._input = input self._output = output def _readline(self): line = self._input.readline() logging.debug(">> " + line.rstrip()) return line def _writeline(self, line): logging.debug("<< " + line) self._output.write(line + "\n") self._output.flush() def _short_response(self, *words): self._writeline(" ".join(words)) def _read_long_part(self): lines = [] while True: line = self._input.readline() if not line: raise ProtocolError() if line == ".\n": break if line[0] != " ": raise ProtocolError() lines.append(line[1:]) return "".join(lines) class Master(Protocol): def __init__(self, input, output): Protocol.__init__(self, input, output) self._commands = { "section": self._switch_section, "recycle": self._recycle, "idle": self._idle, "status": self._status, "reserve": self._reserve, "unreserve": self._unreserve, "pass": self._pass, "fail": self._fail, "untestable": self._untestable, # debug commands, unstable and undocumented interface "_state": self._state, "_depends": self._depends, "_recursive-depends": self._recursive_depends, "_depcycle": self._depcycle, "_list": self._list, } self._section = None self._lock = None self._writeline("hello") def _init_section(self, section): if self._lock: self._lock.close() # clear all settings from a previous section and set defaults self._section = None self._lock = None self._recycle_mode = False self._idle_mode = None self._idle_stamp = os.path.join(section, "idle.stamp") self._package_databases = None self._binary_db = None config = Config(section=section, defaults_section="global") try: config.read(CONFIG_FILE) except MissingSection: return False if not os.path.exists(section): os.makedirs(section) self._lock = open(os.path.join(section, "master.lock"), "we") try: fcntl.flock(self._lock, fcntl.LOCK_EX | fcntl.LOCK_NB) except IOError: return False self._section = section logging.debug(timestamp() + " switching logfile") logfile = config["log-file"] or os.path.join(section, "master.log") setup_logging(logging.DEBUG, logfile) logging.debug(timestamp() + " connected") # start with a dummy _binary_db (without Packages file), sufficient # for submitting finished logs self._binary_db = piupartslib.packagesdb.PackagesDB(prefix=section) return True def _init_db(self): if self._package_databases is not None: return self._package_databases = {} self._load_package_database(self._section) self._binary_db = self._package_databases[self._section] def _load_package_database(self, section): if section in self._package_databases: return config = Config(section=section, defaults_section="global") config.read(CONFIG_FILE) distro_config = piupartslib.conf.DistroConfig(DISTRO_CONFIG_FILE, config["mirror"]) db = piupartslib.packagesdb.PackagesDB(prefix=section) if self._recycle_mode and self._section == section: db.enable_recycling() self._package_databases[section] = db if config["depends-sections"]: deps = config["depends-sections"].split() for dep in deps: self._load_package_database(dep) db.set_dependency_databases([self._package_databases[dep] for dep in deps]) db.load_packages_urls( distro_config.get_packages_urls( config.get_distro(), config.get_area(), config.get_arch())) if config.get_distro() != config.get_final_distro(): # take version numbers (or None) from final distro db.load_alternate_versions_from_packages_urls( distro_config.get_packages_urls( config.get_final_distro(), config.get_area(), config.get_arch())) def _clear_idle(self): if not self._idle_mode is False: self._idle_mode = False if os.path.exists(self._idle_stamp): os.unlink(self._idle_stamp) def _set_idle(self): if not self._idle_mode is True: self._idle_mode = True open(self._idle_stamp, "w").close() os.utime(self._idle_stamp, (-1, self._binary_db._stamp)) def _get_idle_status(self): """ Returns number of seconds a cached idle status is still valid, or 0 if not known to be idle. """ if not os.path.exists(self._idle_stamp): return 0 stamp_mtime = os.path.getmtime(self._idle_stamp) ttl = stamp_mtime + 3600 - time.time() if ttl <= 0: return 0 # stamp expired if stamp_mtime < self._binary_db.get_mtime(): return 0 # stamp outdated return ttl + random.randrange(120) def do_transaction(self): line = self._readline() if line: parts = line.split() if len(parts) > 0: command = parts[0] args = parts[1:] if self._section is None and command != "section": raise CommandSyntaxError("Expected 'section' command, got %s" % command) if command in self._commands: self._commands[command](command, args) return True else: raise CommandSyntaxError("Unknown command %s" % command) return False def _check_args(self, count, command, args): if len(args) != count: raise CommandSyntaxError("Need exactly %d args: %s %s" % (count, command, " ".join(args))) def dump_pkgs(self): for st in self._binary_db.get_states(): for name in self._binary_db.get_pkg_names_in_state(st): logging.debug("%s : %s\n" % (st, name)) def _switch_section(self, command, args): self._check_args(1, command, args) if self._init_section(args[0]): self._short_response("ok") elif self._lock is None: # unknown section self._short_response("error") else: self._short_response("busy") def _recycle(self, command, args): self._check_args(0, command, args) if self._binary_db.enable_recycling(): self._idle_stamp = os.path.join(self._section, "recycle.stamp") self._recycle_mode = True self._short_response("ok") else: self._short_response("error") def _idle(self, command, args): self._check_args(0, command, args) self._short_response("ok", "%d" % self._get_idle_status()) def _status(self, command, args): self._check_args(0, command, args) self._init_db() stats = "" if self._binary_db._recycle_mode: stats += "(recycle) " total = 0 for state in self._binary_db.get_states(): count = len(self._binary_db.get_pkg_names_in_state(state)) total += count stats += "%s=%d " % (state, count) stats += "total=%d" % total self._short_response("ok", stats) def _reserve(self, command, args): self._check_args(0, command, args) self._init_db() package = self._binary_db.reserve_package() if package is None: self._set_idle() self._short_response("error") else: self._clear_idle() self._short_response("ok", package["Package"], package["Version"]) def _unreserve(self, command, args): self._check_args(2, command, args) self._binary_db.unreserve_package(args[0], args[1]) self._short_response("ok") def _pass(self, command, args): self._check_args(2, command, args) log = self._read_long_part() try: self._binary_db.pass_package(args[0], args[1], log) except LogfileExists: logging.info("Ignoring duplicate submission: %s %s %s" % ("pass", args[0], args[1])) self._short_response("ok") def _fail(self, command, args): self._check_args(2, command, args) log = self._read_long_part() try: self._binary_db.fail_package(args[0], args[1], log) except LogfileExists: logging.info("Ignoring duplicate submission: %s %s %s" % ("fail", args[0], args[1])) self._short_response("ok") def _untestable(self, command, args): self._check_args(2, command, args) log = self._read_long_part() try: self._binary_db.make_package_untestable(args[0], args[1], log) except LogfileExists: logging.info("Ignoring duplicate submission: %s %s %s" % ("untestable", args[0], args[1])) self._short_response("ok") # debug command def _state(self, command, args): self._check_args(1, command, args) self._short_response("ok", self._binary_db.get_package_state(args[0]), self._binary_db.get_package(args[0])["Version"]) # debug command def _depends(self, command, args): self._check_args(1, command, args) if self._binary_db.has_package(args[0]): package = self._binary_db.get_package(args[0]) self._short_response("ok", *package.dependencies()) else: self._short_response("error") # debug command def _recursive_depends(self, command, args): self._check_args(1, command, args) if self._binary_db.has_package(args[0]): package = self._binary_db.get_package(args[0]) self._short_response("ok", *self._binary_db._get_recursive_dependencies(package)) else: self._short_response("error") # debug command def _depcycle(self, command, args): self._check_args(1, command, args) if self._binary_db.has_package(args[0]): self._short_response("ok", *self._binary_db._get_dependency_cycle(args[0])) else: self._short_response("error") # debug command def _list(self, command, args): self._check_args(1, command, args) if args[0] in self._binary_db.get_states(): self._short_response("ok", *self._binary_db.get_pkg_names_in_state(args[0])) else: self._short_response("error") def main(): setup_logging(logging.INFO, None) global_config = Config(section="global") global_config.read(CONFIG_FILE) if global_config["proxy"]: os.environ["http_proxy"] = global_config["proxy"] master_directory = global_config["master-directory"] if not os.path.exists(master_directory): os.makedirs(master_directory) os.chdir(master_directory) m = Master(sys.stdin, sys.stdout) try: while m.do_transaction(): pass except URLError as e: logging.error("ABORT: URLError: " + str(e.reason)) logging.debug(timestamp() + " disconnected") if __name__ == "__main__": main() # vi:set et ts=4 sw=4 : piuparts/piuparts-master.in0000755000000000000000000000243712567171321013321 0ustar #!/bin/sh set -e # Copyright © 2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # wrapper for running piuparts-master-backend without needing to know # anything about the master setup and paths # . @sharedir@/piuparts/lib/read_config.sh get_config_value MASTER global master-directory get_config_value PYTHONPATH global PYTHONPATH '' get_config_value LOGFILE global log-file master-error.log export PYTHONPATH LOGFILE="$LOGFILE.$$" # put logfile in a deterministic location cd "$MASTER" trap "test -s ${LOGFILE} || rm -f ${LOGFILE}" EXIT timeout 15m @sharedir@/piuparts/piuparts-master-backend 2> "$LOGFILE" piuparts/piuparts-report.py0000644000000000000000000021364713061276553013372 0ustar #!/usr/bin/python # -*- coding: utf-8 -*- # # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright 2009-2014 Holger Levsen (holger@layer-acht.org) # Copyright © 2011-2016 Andreas Beckmann (anbe@debian.org) # Copyright 2013 David Steele (dsteele@gmail.com) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """Create HTML reports of piuparts log files Lars Wirzenius """ import os import sys import time import logging import ConfigParser import shutil import re import string import yaml import hashlib from urllib2 import HTTPError # if python-rpy2 ain't installed, we don't draw fancy graphs try: from rpy2 import robjects from rpy2.robjects.packages import importr except: pass import piupartslib from piupartslib.conf import MissingSection from piupartslib.dwke import * import piupartslib.pkgsummary as pkgsummary CONFIG_FILE = "/etc/piuparts/piuparts.conf" DISTRO_CONFIG_FILE = "/etc/piuparts/distros.conf" KPR_DIRS = ('pass', 'bugged', 'affected', 'fail') TPL_EXT = '.tpl' PIUPARTS_VERSION = "__PIUPARTS_VERSION__" HTML_HEADER = """ $page_title
$section_navigation
General information
About + News
Overview
FAQ
Source
piuparts.d.o bugs
piuparts bugs / ToDo
Contact us
Documentation
piuparts manpage
README
README_server
piuparts.d.o configuration:
piuparts.conf,
distros.conf
and scripts
How to file bugs
Debian policy
Available reports
Bugs filed
Other Debian QA efforts
Debian QA Group
Dose tools (former: EDOS)
Lintian
Debian Package Tracker (former: PTS)
Ultimate Debian Database
jenkins.debian.net
ci.debian.net
Last update
$time
""" HTML_FOOTER = """
""" LOG_LIST_BODY_TEMPLATE = """ $logrows
$title in $section
$preface The list has $count packages, with $versioncount total versions.
""" STATE_BODY_TEMPLATE = """
Packages in state "$state" in $section $aside
    $list
""" SECTION_INDEX_BODY_TEMPLATE = """ $tablerows
$section statistics
$description
Binary packages per state
URL to Packages file
$packagesurl
""" MAINTAINER_BODY_TEMPLATE = """ $distrolinks $rows
$maintainer
""" SOURCE_PACKAGE_BODY_TEMPLATE = """ $rows
""" ANALYSIS_BODY_TEMPLATE = """ $rows
""" PROB_TPL = \ """ $HEADER in $SECTION, sorted by reverse dependency count. $HELPTEXT

The commandline to find these logs is: 

COMMAND='$COMMAND'

Please file bugs!
    $PACKAGE_LIST

Affected packages in $SECTION: $COUNT

""" PKG_ERROR_TPL = \ """
  • $RDEPS - $LOG$ARCH (PTS) (BTS) $BUG
    • """ title_by_dir = { "pass": "PASSED piuparts logs", "fail": "Failed UNREPORTED piuparts logs", "bugged": "Failed REPORTED piuparts logs", "affected": "Failed AFFECTED piuparts logs", "reserved": "RESERVED packages", "untestable": "UNTESTABLE packages", } desc_by_dir = { "pass": "Log files for packages that have PASSED testing.", "fail": "Log files for packages that have FAILED testing. " + "Bugs have not yet been reported.", "bugged": "Log files for packages that have FAILED testing. " + "Bugs have been reported, but not yet fixed.", "affected": "Log files for packages that have dependencies FAILED testing. " + "Bugs have been reported, but not yet fixed.", "reserved": "Packages that are RESERVED for testing on a node in a " + "distributed piuparts network.", "untestable": "Log files for packages that have are UNTESTABLE with " + "piuparts at the current time.", } state_by_dir = { "pass": "successfully-tested", "fail": "failed-testing", "bugged": "failed-testing", "affected": "failed-testing", "reserved": "waiting-to-be-tested", "untestable": "cannot-be-tested", } # better use XX_name.tpl and get the linktarget from the template # (its a substring of the of the that template # maintaining this list is errorprone and tiresome linktarget_by_template = [ ("initdscript_lsb_header_issue.tpl", "but logfile contains update-rc.d issues"), ("command_not_found_issue.tpl", "but logfile contains 'command not found'"), ("debsums_mismatch_issue.tpl", "but logfile contains modified conffiles or other shipped files"), ("alternatives_after_purge_issue.tpl", "but logfile contains forgotten alternatives"), ("owned_files_after_purge_issue.tpl", "but logfile contains owned files existing after purge"), ("unowned_files_after_purge_issue.tpl", "but logfile contains unowned files after purge"), ("maintainer_script_issue.tpl", "but logfile contains maintainer script failures"), ("db_setup_issue.tpl", "but logfile contains failure to setup a database"), ("problems_and_no_force_issue.tpl", "but logfile reports that not enough force was used"), ("installs_over_symlink_issue.tpl", "but package installs something over existing symlinks"), ("broken_symlinks_issue.tpl", "but logfile contains 'broken symlinks'"), ("unknown_inadequate_issue.tpl", "but logfile contains unknown inadequate issues"), ("boring_obsolete_conffile_file_inadequate_issue.tpl", "...and logfile contains tag from adequate 'obsolete-conffile-file'"), ("boring_broken_symlink_file_inadequate_issue.tpl", "...and logfile contains tag from adequate 'broken-symlink-file'"), ("bin_or_sbin_binary_requires_usr_lib_library_inadequate_issue.tpl", "but adequate found a binary in /bin or /sbin that requires a /usr/lib library"), ("incompatible_licenses_inadequate_issue.tpl", "but adequate found a license incompatibility"), ("broken_binfmt_detector_inadequate_issue.tpl", "but adequate did not find the detector registered with update_binfmts"), ("broken_binfmt_interpreter_inadequate_issue.tpl", "but adequate did not find the interpreter registered with update_binfmts"), ("missing_alternative_inadequate_issue.tpl", "but adequate found a missing alternative"), ("missing_copyright_file_inadequate_issue.tpl", "but adequate couldn't find a copyright file"), ("missing_pkgconfig-dependency_issue.tpl", "but adequate found a missing pkgconfig dependency"), ("program_name_collision_inadequate_issue.tpl", "but adequate found a program name collision"), ("py_file_not_bytecompiled_inadequate_issue.tpl", "but adequate found a .py file that is not byte-compiled"), ("pyshared_file_not_bytecompiled_inadequate_issue.tpl", "but adequate found a .py file in /usr/share/pyshared that is not byte-compiled"), ("ldd_inadequate_issue.tpl", "but adequate failed to run ldd on a binary or library"), ("library_not_found_inadequate_issue.tpl", "but adequate couldn't find a required library"), ("undefined_symbol_inadequate_issue.tpl", "but adequate found an undefined symbol"), ("symbol-size-mismatch_inadequate_issue.tpl", "but adequate found that a symbol has changed size since the package was built"), ("missing-symbol-version-information_inadequate_issue.tpl", "but adequate found that a library is missing symbol version information"), ("unknown_inadequate_issue.tpl", "but an unknown adequate issue was found"), ("inadequate_exit_issue.tpl", "but adequate exited inadequately"), ("packages_have_been_kept_back_issue.tpl", "but logfile contains 'packages have been kept back'"), ("needs_rebuild_issue.tpl", "but logfile recommends to rebuild some packages"), ("module_build_error_issue.tpl", "but logfile contains dkms module build failures"), ("obsolete_conffiles_issue.tpl", "but logfile reports obsolete conffiles"), ("missing_md5sums_issue.tpl", "but logfile reports missing md5sums"), ("unowned_lib_symlink_issue.tpl", "but logfile reports unowned lib symlinks"), ("piuparts-depends-dummy_issue.tpl", "but logfile reports piuparts-depends-dummy.deb could not be installed"), ("used_exception_issue.tpl", "but package used a piuparts exception"), ("test_was_skipped_issue.tpl", "but package test was skipped"), ("dependency_error.tpl", "due to unsatisfied dependencies"), ("packages_have_been_kept_back_error.tpl", "...and logfile also contains 'packages have been kept back'"), ("command_not_found_error.tpl", "due to a 'command not found' error"), ("files_in_usr_local_error.tpl", "due to files in /usr/local"), ("overwrite_other_packages_files_error.tpl", "due to overwriting other packages files"), ("debsums_mismatch_error.tpl", "due to modifying conffiles or other shipped files"), ("alternatives_after_purge_error.tpl", "due to forgotten alternatives after purge"), ("owned_files_by_many_packages_error.tpl", "due to owned files by many packages"), ("owned_files_after_purge_error.tpl", "due to owned files existing after purge"), ("unowned_files_after_purge_error.tpl", "due to unowned files after purge"), ("modified_files_after_purge_error.tpl", "due to files having been modified after purge"), ("disappeared_files_after_purge_error.tpl", "due to files having disappeared after purge"), ("diversion_error.tpl", "due to diversions being modified after purge"), ("processes_running_error.tpl", "due to leaving processes running behind"), ("resource_violation_error.tpl", "due to resource violation"), ("conffile_prompt_error.tpl", "due to prompting due to modified conffiles"), ("db_setup_error.tpl", "due to failing to setup a database"), ("insserv_error.tpl", "due to a problem with insserv"), ("problems_and_no_force_error.tpl", "due to not enough force being used"), ("immediate_configuration_error.tpl", "due to apt could not perform immediate configuration"), ("pre_depends_error.tpl", "due to a problem with pre-depends"), ("pre_installation_script_error.tpl", "due to pre-installation maintainer script failed"), ("post_installation_script_error.tpl", "due to post-installation maintainer script failed"), ("pre_removal_script_error.tpl", "due to pre-removal maintainer script failed"), ("post_removal_script_error.tpl", "due to post-removal maintainer script failed"), ("unknown_purge_error.tpl", "due to purge failed due to an unknown reason"), ("cron_error_after_removal_error.tpl", "due to errors from cronjob after removal"), ("logrotate_error_after_removal_error.tpl", "due to errors from logrotate after removal"), ("installs_over_symlink_error.tpl", "...and package installs something over existing symlinks"), ("broken_symlinks_error.tpl", "...and logfile also contains 'broken symlinks'"), ("obsolete_conffiles_error.tpl", "...and logfile reports obsolete conffiles"), ("missing_md5sums_error.tpl", "...and logfile reports missing md5sums"), ("unowned_lib_symlink_error.tpl", "...and logfile reports unowned lib symlinks"), ("piuparts-depends-dummy_error.tpl", "...and logfile reports piuparts-depends-dummy.deb could not be installed"), ("unclassified_failures.tpl", "due to unclassified failures"), ] class Config(piupartslib.conf.Config): def __init__(self, section="report", defaults_section=None): self.section = section piupartslib.conf.Config.__init__(self, section, { "sections": "report", "output-directory": "html", "master-directory": ".", "depends-sections": None, "description": "", "proxy": None, "mirror": None, "distro": None, "area": None, "arch": None, "upgrade-test-distros": None, "max-reserved": 1, "doc-root": "/", "known-problem-directory": "@sharedir@/piuparts/known_problems", "json-sections": "default", "precedence": 1, "web-host": "piuparts.debian.org", }, defaults_section=defaults_section) def setup_logging(log_level, log_file_name): logger = logging.getLogger() logger.setLevel(log_level) formatter = logging.Formatter(fmt="%(asctime)s %(message)s", datefmt="%H:%M:%S") handler = logging.StreamHandler(sys.stderr) handler.setFormatter(formatter) logger.addHandler(handler) if log_file_name: handler = logging.FileHandler(log_file_name) logger.addHandler(handler) def html_protect(vstr): vstr = "&".join(vstr.split("&")) vstr = "<".join(vstr.split("<")) vstr = ">".join(vstr.split(">")) vstr = """.join(vstr.split('"')) vstr = "'".join(vstr.split("'")) return vstr def is_bad_state(state): bad_states = [ #"successfully-tested", "failed-testing", "cannot-be-tested", # "essential-required", # obsolete #"waiting-to-be-tested", #"waiting-for-dependency-to-be-tested", "dependency-failed-testing", "dependency-cannot-be-tested", "dependency-does-not-exist", "circular-dependency", # obsolete "unknown", "unknown-preferred-alternative", # obsolete "no-dependency-from-alternatives-exists", # obsolete #"foreign:*", "does-not-exist", ] return(state in bad_states) def emphasize_reason(reason): if is_bad_state(reason): reason = "<em>" + reason + "</em>" return reason def source_subdir(source): if source[:3] == "lib": return source[:4] else: return source[:1] def source_summary_url(web_host, doc_root, section, src_pkg): return("https://%s%s/%s/source/%s/%s.html" % ( web_host, doc_root, section, source_subdir(src_pkg), src_pkg, ) ) def maintainer_subdir(maintainer): return maintainer.lower()[:1] def find_files_with_suffix(vdir, suffix): pairs = [] # (mtime, name) for name in os.listdir(vdir): if name.endswith(suffix): try: if os.path.isfile(os.path.join(vdir, name)): mtime = os.path.getmtime(os.path.join(vdir, name)) pairs.append((mtime, name)) except OSError: pass # sort by mtime return [x[1] for x in sorted(pairs)] def update_file(source, target): if os.path.exists(target): try: aa = os.stat(source) bb = os.stat(target) except OSError: pass else: if aa.st_size == bb.st_size and aa.st_mtime <= bb.st_mtime: return try: os.remove(target) except: pass try: os.link(source, target) except OSError: try: shutil.copyfile(source, target) except IOError as xxx_todo_changeme: (errno, strerror) = xxx_todo_changeme.args logging.error("failed to copy %s to %s: I/O error(%d): %s" % (source, target, errno, strerror)) def copy_logs(logs_by_dir, output_dir): for vdir in logs_by_dir: fulldir = os.path.join(output_dir, vdir) if not os.path.exists(fulldir): os.makedirs(fulldir) for basename in logs_by_dir[vdir]: source = os.path.join(vdir, basename) target = os.path.join(fulldir, basename) update_file(source, target) def remove_old_logs(logs_by_dir, output_dir): for vdir in logs_by_dir: fulldir = os.path.join(output_dir, vdir) # convert logs_by_dir array to a dict to avoid linear search logs_dict = {} for log in logs_by_dir[vdir]: logs_dict[log] = 1 if os.path.exists(fulldir): for basename in os.listdir(fulldir): if not basename in logs_dict: os.remove(os.path.join(fulldir, basename)) def write_file(filename, contents): f = file(filename, "w") f.write(contents) f.close() def append_file(filename, contents): f = file(filename, "a") f.write(contents) f.close() def read_file(filename): f = file(filename, "r") l = f.readlines() f.close() return l def write_template_html(filename, body, mapping={}): header = HTML_HEADER footer = HTML_FOOTER htmlpage = string.Template(header + body + footer) mapping = mapping.copy() mapping.update({ "content_md5": "", "piuparts_version": "", "time": "", }) content_md5 = hashlib.md5(htmlpage.safe_substitute(mapping)).hexdigest() mapping.update({ "content_md5": content_md5, "piuparts_version": PIUPARTS_VERSION, "time": time.strftime("%Y-%m-%d %H:%M %Z"), }) write_file(filename, htmlpage.safe_substitute(mapping)) def create_section_navigation(section_names, current_section, doc_root): tablerows = "" for section in section_names: tablerows += ("<tr class=\"normalrow\"><td class=\"contentcell\"><a href='%s/%s'>%s</a></td></tr>\n") % \ (doc_root, html_protect(section), html_protect(section)) tablerows += "<tr><td class=\"contentcell\"><a href=\"%s/%s/maintainer/\">by maintainer / uploader</a></td></tr>\n" \ % (doc_root, current_section) tablerows += "<tr><td class=\"contentcell\"><a href=\"%s/%s/source/\">by source package</a></td></tr>\n" \ % (doc_root, current_section) return tablerows; def get_email_address(maintainer): email = "INVALID maintainer address: %s" % (maintainer) try: m = re.match(r"(.+)(<)(.+@.+)(>)", maintainer) email = m.group(3) except: pass return email def package2id(package_name): # "+" is not a valid identifier char for id=... attributes return package_name.replace("+", "_") # return order preserving list of the first occurrence of an element def unique(stuff): # can't use set() because 'stuff' is a list of lists and list() is not hashable vlist = [] previtem = stuff for item in stuff: if item != previtem: vlist.append(item) previtem = item return vlist class Section: def __init__(self, section, master_directory, doc_root, packagedb_cache={}): self._config = Config(section=section, defaults_section="global") self._config.read(CONFIG_FILE) self._distro_config = piupartslib.conf.DistroConfig( DISTRO_CONFIG_FILE, self._config["mirror"]) logging.debug("-------------------------------------------") logging.debug("Running section " + self._config.section) self._section_directory = os.path.abspath(os.path.join(master_directory, self._config.section)) if not os.path.exists(self._section_directory): logging.debug("Warning: %s did not exist, now created. Did you ever let the slave work?" % self._section_directory) os.makedirs(self._section_directory) self._doc_root = doc_root logging.debug("Loading and parsing Packages file") self._packagedb_cache = packagedb_cache self._package_databases = {} self._load_package_database(section, master_directory) self._binary_db = self._package_databases[section] self._source_db = piupartslib.packagesdb.PackagesDB(prefix=self._section_directory) self._source_db.load_packages_urls( self._distro_config.get_sources_urls( self._config.get_distro(), self._config.get_area())) if self._config.get_distro() != self._config.get_final_distro(): # take version numbers (or None) from final distro self._source_db.load_alternate_versions_from_packages_urls( self._distro_config.get_sources_urls( self._config.get_final_distro(), self._config.get_area())) self._log_name_cache = {} def _load_package_database(self, section, master_directory): if section in self._package_databases: return elif section in self._packagedb_cache: self._package_databases[section] = self._packagedb_cache[section] return config = Config(section=section, defaults_section="global") config.read(CONFIG_FILE) if not config["depends-sections"]: # this is a base database eligible for caching # only cache the most recent base database self._packagedb_cache.clear() sectiondir = os.path.join(master_directory, section) db = piupartslib.packagesdb.PackagesDB(prefix=sectiondir) self._package_databases[section] = db if config["depends-sections"]: deps = config["depends-sections"].split() for dep in deps: self._load_package_database(dep, master_directory) db.set_dependency_databases([self._package_databases[dep] for dep in deps]) else: # only cache the big base databases that don't have additional dependencies self._packagedb_cache[section] = db db.load_packages_urls( self._distro_config.get_packages_urls( config.get_distro(), config.get_area(), config.get_arch())) if config.get_distro() != config.get_final_distro(): # take version numbers (or None) from final distro db.load_alternate_versions_from_packages_urls( self._distro_config.get_packages_urls( config.get_final_distro(), config.get_area(), config.get_arch())) def _write_template_html(self, filename, body, mapping={}): mapping = mapping.copy() mapping.update({ "section_navigation": self._section_navigation, "doc_root": self._doc_root, "section": html_protect(self._config.section), }) write_template_html(filename, body, mapping) def write_log_list_page(self, filename, title, preface, logs): packages = {} for pathname, package, version in logs: packages[package] = packages.get(package, []) + [(pathname, version)] names = sorted(packages.keys()) lines = [] version_count = 0 for package in names: versions = [] for pathname, version in packages[package]: version_count += 1 versions.append("<a href=\"%s\">%s</a>" % (html_protect(pathname), html_protect(version))) line = "<tr class=\"normalrow\"><td class=\"contentcell2\">%s</td><td class=\"contentcell2\">%s</td></tr>" % \ (html_protect(package), ", ".join(versions)) lines.append(line) if "FAIL" in preface: title_style = "alerttitlecell" else: title_style = "titlecell" self._write_template_html( filename, LOG_LIST_BODY_TEMPLATE, { "page_title": html_protect(title + " in " + self._config.section), "title": html_protect(title), "title_style": title_style, "preface": preface, "count": len(packages), "versioncount": version_count, "logrows": "".join(lines), }) def print_by_dir(self, output_directory, logs_by_dir): for vdir in logs_by_dir: vlist = [] for basename in logs_by_dir[vdir]: assert basename.endswith(".log") assert "_" in basename package, version = basename[:-len(".log")].split("_") vlist.append((os.path.join(vdir, basename), package, version)) self.write_log_list_page(os.path.join(output_directory, vdir + ".html"), title_by_dir[vdir], desc_by_dir[vdir], vlist) def find_links_to_logs(self, package_name, dirs, logs_by_dir): links = [] for vdir in dirs: # avoid linear search against log file names by caching in a dict # # this cache was added to avoid a very expensive linear search # against the arrays in logs_by_dir. Note that the use of this cache # assumes that the contents of logs_by_dir is invarient across calls # to find_links_to_logs() # if vdir not in self._log_name_cache: self._log_name_cache[vdir] = {} for basename in logs_by_dir[vdir]: if basename.endswith(".log"): package, version = basename[:-len(".log")].split("_") self._log_name_cache[vdir][package] = version if vdir == "fail": style = " class=\"needs-bugging\"" else: style = "" if package_name in self._log_name_cache[vdir]: basename = package_name \ + "_" \ + self._log_name_cache[vdir][package_name] \ + ".log" links.append("<a href=\"%s/%s\"%s>%s</a>" % ( self._doc_root, os.path.join(self._config.section, vdir, basename), style, html_protect(self._log_name_cache[vdir][package_name]), )) return links def link_to_maintainer_summary(self, maintainer): email = get_email_address(maintainer) return "<a href=\"%s/%s/maintainer/%s/%s.html\">%s</a>" \ % (self._doc_root, self._config.section, maintainer_subdir(email), email, html_protect(maintainer)) def link_to_uploaders(self, uploaders): link = "" for uploader in uploaders.split(","): link += self.link_to_maintainer_summary(uploader.strip()) + ", " return link[:-2] def link_to_source_summary(self, package_name): source_name = self._binary_db.get_source(package_name) link = "<a href=\"%s/%s/source/%s\">%s</a>" % ( self._doc_root, self._config.section, source_subdir(source_name) + "/" + source_name + ".html", html_protect(package_name)) return link def link_to_state_page(self, section, package_name, link_target): if self._binary_db.has_package(package_name): state = self._binary_db.get_package_state(package_name) link = "<a href=\"%s/%s/%s\">%s</a>" % ( self._doc_root, section, "state-" + state + ".html" + "#" + package_name, link_target) else: if link_target == package_name: link = html_protect(package_name) else: link = "unknown-package" return link def links_to_logs(self, package_name, state, logs_by_dir): link = "N/A" dirs = "" if state == "successfully-tested": dirs = ["pass"] elif state == "failed-testing": dirs = ["fail", "bugged", "affected"] elif state == "cannot-be-tested": dirs = ["untestable"] if dirs != "": links = self.find_links_to_logs(package_name, dirs, logs_by_dir) link = ", ".join(links) if "/bugged/" in link or "/affected/" in link: link += " - <a href=\"https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=" \ + package_name \ + "\" target=\"_blank\" class=\"bugged\"> bug filed </a>\n" return link def write_counts_summary(self): logging.debug("Writing counts.txt") header = "date" current_day = "%s" % time.strftime("%Y%m%d") counts = current_day total = 0 for state in self._binary_db.get_states(): count = len(self._binary_db.get_pkg_names_in_state(state)) header += ", %s" % state counts += ", %s" % count logging.debug("%s: %s" % (state, count)) total += count logging.debug("total: %s" % total) logging.debug("source: %s" % len(self._source_db.get_all_packages())) header += "\n" counts += "\n" countsfile = os.path.join(self._section_directory, "counts.txt") if not os.path.isfile(countsfile): logging.debug("writing new file: %s" % countsfile) write_file(countsfile, header) last_line = "" else: last_line = read_file(countsfile)[-1] if not current_day in last_line: append_file(countsfile, counts) logging.debug("appending line: %s" % counts.strip()) shutil.copy(countsfile, self._output_directory) return total def create_maintainer_summaries(self, maintainers, source_data): logging.debug("Writing %d maintainer summaries in %s" % (len(maintainers), self._output_directory)) maintainer_dir = os.path.join(self._output_directory, "maintainer") if not os.path.exists(maintainer_dir): os.mkdir(maintainer_dir) states = ["fail", "unknown", "pass"] for maintainer in sorted(maintainers.keys()): sources = maintainers[maintainer] maintainer_subdir_path = os.path.join(maintainer_dir, maintainer_subdir(maintainer)) if not os.path.exists(maintainer_subdir_path): os.mkdir(maintainer_subdir_path) rows = "" package_rows = {} packages = {} for state in states: packages[state] = [] package_rows[state] = "" for source in sorted(sources): (state, sourcerows, binaryrows) = source_data[source] packages[state].append(source) package_rows[state] += sourcerows + binaryrows for state in states: if len(packages[state]) > 0: links = "" for package in packages[state]: links += "<a href=\"#%s\">%s</a> " % (package, package) else: links = " " rows += "<tr class=\"normalrow\">" \ + "<td class=\"labelcell\">%s:</td>" % state \ + "<td class=\"contentcell2\">%s</td>" % len(packages[state]) \ + "<td class=\"contentcell2\" colspan=\"4\">%s</td>" % links \ + "</tr>\n" distrolinks = "<tr class=\"normalrow\">" \ + "<td class=\"labelcell\">other distributions: </td>" \ + "<td class=\"contentcell2\" colspan=\"5\">" for section in self._section_names: if section != self._config.section: distrolinks += "<a href=\"" \ + self._doc_root \ + "/" \ + section \ + "/maintainer/" \ + maintainer_subdir(maintainer) \ + "/" \ + maintainer \ + ".html\">" \ + html_protect(section) \ + "</a> " distrolinks += "</td></tr>" self._write_template_html( os.path.join(maintainer_subdir_path, maintainer + ".html"), MAINTAINER_BODY_TEMPLATE, { "page_title": html_protect("Status of " + maintainer + " packages in " + self._config.section), "maintainer": html_protect(maintainer + " in " + self._config.section), "distrolinks": distrolinks, "rows": rows + "".join([package_rows[state] for state in states]), }) def create_source_summary(self, source, logs_by_dir): source_version = self._source_db.get_control_header(source, "Version") binaries = self._source_db.get_control_header(source, "Binary") maintainer = self._source_db.get_control_header(source, "Maintainer") uploaders = self._source_db.get_control_header(source, "Uploaders") success = True failed = False binaryrows = "" for binary in sorted([x.strip() for x in binaries.split(",") if x.strip()]): if not self._binary_db.has_package(binary): # udebs or binary packages for other architectures # The latter is a FIXME which needs parsing the Packages files from other archs too binaryrows += "<tr class=\"normalrow\">" \ + "<td class=\"labelcell\">Binary:</td>" \ + "<td class=\"contentcell2\">%s</td>" \ % binary \ + "<td class=\"contentcell2\" colspan=\"4\">unknown package</td>" \ + "</tr>\n" continue state = self._binary_db.get_package_state(binary) if not "waiting" in state and "dependency" in state: state_style = "lightalertlabelcell" elif state == "failed-testing": state_style = "lightlabelcell" else: state_style = "labelcell" binary_version = self._binary_db.get_control_header(binary, "Version") binary_arch = self._binary_db.get_control_header(binary, "Architecture") archsuffix = "" if binary_arch == "all": archsuffix = ":all" binaryrows += "<tr class=\"normalrow\">" \ + "<td class=\"labelcell\">Binary:</td>" \ + "<td class=\"contentcell2\">%s%s</td>" \ % (binary, archsuffix) \ + "<td class=\"%s\">piuparts-result:</td>" \ % state_style \ + "<td class=\"contentcell2\">%s %s</td>" \ % (self.link_to_state_page(self._config.section, binary, state), self.links_to_logs(binary, state, logs_by_dir)) \ + "<td class=\"labelcell\">Version:</td>" \ + "<td class=\"contentcell2\">%s</td>" \ % html_protect(binary_version) \ + "</tr>\n" if state not in ("successfully-tested", "essential-required"): success = False if state in ("failed-testing", "dependency-does-not-exist", "cannot-be-tested"): failed = True if binaryrows != "": source_state = "unknown" if success: source_state = "<img src=\"%s/images/sunny.png\" alt=\"success\">" % self._doc_root if failed: source_state = "<img src=\"%s/images/weather-severe-alert.png\" alt=\"failed\">" % self._doc_root sourcerows = "<tr class=\"titlerow\">" \ + "<td class=\"titlecell\" colspan=\"6\" id=\"%s\">%s in %s</td>" \ % (package2id(source), source, self._config.section) \ + "</tr>\n" sourcerows += "<tr class=\"normalrow\">" \ + "<td class=\"labelcell\">Source:</td>" \ + "<td class=\"contentcell2\">" \ + "<a href=\"https://tracker.debian.org/%s\" target=\"_blank\">%s</a>" \ % (source, html_protect(source)) \ + "</td>" \ + "<td class=\"labelcell\">piuparts summary:</td>" \ + "<td class=\"contentcell2\">%s</td>" \ % source_state \ + "<td class=\"labelcell\">Version:</td>" \ + "<td class=\"contentcell2\">%s</td>" \ % html_protect(source_version) \ + "</tr>\n" sourcerows += "<tr class=\"normalrow\">" \ + "<td class=\"labelcell\">Maintainer:</td>" \ + "<td class=\"contentcell2\" colspan=\"5\">%s</td>" \ % self.link_to_maintainer_summary(maintainer) \ + "</tr>\n" if uploaders: sourcerows += "<tr class=\"normalrow\">" \ + "<td class=\"labelcell\">Uploaders:</td>" \ + "<td class=\"contentcell2\" colspan=\"5\">%s</td>" \ % self.link_to_uploaders(uploaders) \ + "</tr>\n" source_summary_page_path = os.path.join(self._output_directory, "source", source_subdir(source)) if not os.path.exists(source_summary_page_path): os.makedirs(source_summary_page_path) self._write_template_html( os.path.join(source_summary_page_path, (source + ".html")), SOURCE_PACKAGE_BODY_TEMPLATE, { "page_title": html_protect("Status of source package " + source + " in " + self._config.section), "rows": sourcerows + binaryrows, }) # return parsable values if success: source_state = "pass" if failed: source_state = "fail" else: source_state = "udeb" sourcerows = "" return sourcerows, binaryrows, source_state, maintainer, uploaders def create_package_summaries(self, logs_by_dir): logging.debug("Writing source summaries in %s" % self._config.section) maintainers = {} source_binary_rows = {} sources = "" sources_data = [] for source in sorted(self._source_db.get_all_package_names()): (sourcerows, binaryrows, source_state, maintainer, uploaders) = \ self.create_source_summary(source, logs_by_dir) if source_state != "udeb": source_version = self._source_db.get_control_header(source, "Version") source_data = {}; source_data["source"] = source source_data["state"] = source_state source_data["version"] = source_version sources_data.append(source_data) sources += "%s: %s\n" % (source, source_state) source_binary_rows[source] = (source_state, sourcerows, binaryrows) for maint in [maintainer] + uploaders.split(","): if maint.strip(): email = get_email_address(maint.strip()) if not "INVALID" in email: if not email in maintainers: maintainers[email] = [] maintainers[email].append(source) write_file(os.path.join(self._output_directory, "sources.txt"), sources) write_file(os.path.join(self._output_directory, "sources.yaml"), yaml.dump(sources_data, default_flow_style=False)) self.create_maintainer_summaries(maintainers, source_binary_rows) def make_section_stats_graph(self): countsfile = os.path.join(self._section_directory, "counts.txt") pngfile = os.path.join(self._output_directory, "states.png") grdevices = importr('grDevices') grdevices.png(file=pngfile, width=1600, height=900, pointsize=10, res=100, antialias="none") r = robjects.r r('t <- (read.table("' + countsfile + '",sep=",",header=1,row.names=1))') r('cname <- c("date",rep(colnames(t)))') # here we define how many days we wants stats for (163=half a year) # r('v <- t[(nrow(t)-163):nrow(t),0:12]') # make graph since day 1 r('v <- t[0:nrow(t),0:12]') # thanks to http://tango.freedesktop.org/Generic_Icon_Theme_Guidelines for those nice colors r('palette(c("#4e9a06", "#ef2929", "#d3d7cf", "#5c3566", "#c4a000", \ "#fce94f", "#a40000", "#888a85", "#2e3436", "#729fcf", \ "#3465a4", "#204a87", "#555753"))') r('barplot(t(v),col = 1:13, \ main="Binary packages per state in ' + self._config.section + '", \ xlab="", ylab="Number of binary packages", space=0, border=NA)') r('legend(x="bottom",legend=colnames(t), ncol=2,fill=1:13,xjust=0.5,yjust=0,bty="n")') grdevices.dev_off() stats_html = "<tr class=\"normalrow\"> " \ + "<td class=\"contentcell2\" colspan=\"3\">" \ + "<a href=\"%s\">" \ % "states.png" \ + "<img src=\"%s\" height=\"450\" width=\"800\" alt=\"Binary package states in %s\">" \ % ("states.png", self._config.section) \ + "</a>" \ + "<br>(<a href=\"counts.txt\">Source</a>)\n" \ + "</td></tr>\n" return stats_html def create_and_link_to_analysises(self, state): link = "<ul>\n" for template, linktarget in linktarget_by_template: # successful logs only have issues and failed logs only have errors if (state == "failed-testing" and template[-9:] != "issue.tpl") \ or (state == "successfully-tested" and template[-9:] == "issue.tpl"): substats = "" tpl = os.path.join(self._output_directory, template) try: f = file(tpl, "r") rows = file.read(f) f.close() os.unlink(tpl) self._write_template_html( os.path.join(self._output_directory, template[:-len(".tpl")] + ".html"), ANALYSIS_BODY_TEMPLATE, { "page_title": html_protect("Packages in state " + state + " " + linktarget), "rows": rows, }) if state == "failed-testing": count_bugged = string.count(rows, '"bugged/') count_affected = string.count(rows, '"affected/') count_failed = string.count(rows, '"fail/') sep = ": " if count_bugged > 0: substats += sep + "%s bugged" % count_bugged sep = ", " if count_affected > 0: substats += sep + "%s affected" % count_affected sep = ", " if count_failed > 0: substats += sep + "<span class=\"needs-bugging\">%s failed</span>" % count_failed else: count_passed = string.count(rows, '"pass/') if count_passed > 0: substats += ": %s passed" % count_passed link += "<li><a href=%s>%s</a>%s</li>\n" % \ ( template[:-len(".tpl")] + ".html", linktarget, substats, ) except: logging.debug("analysis template %s does not exist." % template) link += "</ul>" if link == "<ul>\n</ul>": link = "" return link def write_section_index_page(self, dirs, total_packages): tablerows = "" for state in self._binary_db.get_active_states(): dir_link = "" analysis = "" for vdir in dirs: if vdir in ("pass", "fail", "bugged", "affected", "untestable") and state_by_dir[vdir] == state: dir_link += "<a href='%s.html'>%s</a> logs<br>" % (vdir, html_protect(vdir)) if state in ("successfully-tested", "failed-testing"): analysis = self.create_and_link_to_analysises(state) tablerows += ("<tr class=\"normalrow\"><td class=\"contentcell2\"><a href='state-%s.html'>%s</a>%s</td>" + "<td class=\"contentcell2\">%d</td><td class=\"contentcell2\">%s</td></tr>\n") % \ (html_protect(state), html_protect(state), analysis, len(self._binary_db.get_pkg_names_in_state(state)), dir_link) try: tablerows += self.make_section_stats_graph(); except: logging.debug("Error generating the graph images, probably python-rpy2 is not installed, disabling graphs.") tablerows += "<tr class=\"normalrow\"> <td class=\"labelcell2\">Total</td> <td class=\"labelcell2\" colspan=\"2\">%d</td></tr>\n" % total_packages vendor = "Debian" if len(self._config.get_distros()) > 1: description = "%s %s: package installation in %s" % ( vendor, self._config.get_area(), self._config.get_start_distro()) for distro in self._config.get_distros()[1:]: description += ", dist-upgrade to %s" % distro description += ", removal, and purge test." else: description = "%s %s / %s: package installation, removal, and purge test." % ( vendor, self._config.get_distro(), self._config.get_area()) if self._config["description"].startswith("+"): description = description + " " + self._config["description"][1:] elif self._config["description"].endswith("+"): description = self._config["description"][:-1] + " " + description elif self._config["description"]: description = self._config["description"] self._write_template_html( os.path.join(self._output_directory, "index.html"), SECTION_INDEX_BODY_TEMPLATE, { "page_title": html_protect(self._config.section + " statistics"), "description": html_protect(description), "tablerows": tablerows, "packagesurl": "<br>".join([html_protect(url) for url in self._binary_db.get_urls()]), }) def _show_providers(self, dep): providers = self._binary_db.get_providers(dep) vlist = "" if providers: vlist += "\n<ul>\n" for provider in providers: vlist += "<li>provider %s is %s</li>\n" % \ (self.link_to_state_page(self._config.section, provider, provider), emphasize_reason(html_protect(self._binary_db.get_package_state(provider)))) vlist += "</ul>\n" return vlist def write_state_pages(self): for state in self._binary_db.get_active_states(): logging.debug("Writing page for %s" % state) vlist = "" if state in self._binary_db.get_error_states(): with_counts = True aside = " (reverse deps, blocked pkgs)" sort_key = lambda x: (-self._binary_db.block_count(x["Package"]), x["Package"]) else: with_counts = False aside = "" sort_key = lambda x: x["Package"] names = self._binary_db.get_pkg_names_in_state(state) packages = [self._binary_db.get_package(name) for name in names] packages.sort(key=sort_key) for package in packages: vlist += "<li id=\"%s\">%s" % ( package2id(package["Package"]), self.link_to_source_summary(package["Package"])) if package["Architecture"] == "all": vlist += ":all" if with_counts: vlist += " (%d, %d)" % (self._binary_db.rrdep_count(package["Package"]), self._binary_db.block_count(package["Package"])) vlist += " (%s)" % html_protect(package["Maintainer"]) all_deps = unique(package.all_dependencies()) if all_deps: vlist += "\n<ul>\n" for alternatives in all_deps: dep = alternatives[0] vlist += "<li>dependency %s is %s" % \ (self.link_to_state_page(self._config.section, dep, dep), emphasize_reason(html_protect(self._binary_db.get_package_state(dep, resolve_virtual=False)))) vlist += self._show_providers(dep) if len(alternatives) > 1: vlist += "\n<ul>\n" for dep in alternatives[1:]: vlist += "<li>alternative dependency %s is %s" % \ (self.link_to_state_page(self._config.section, dep, dep), emphasize_reason(html_protect(self._binary_db.get_package_state(dep, resolve_virtual=False)))) vlist += self._show_providers(dep) vlist += "</li>\n" vlist += "</ul>\n" vlist += "</li>\n" vlist += "</ul>\n" vlist += "</li>\n" self._write_template_html( os.path.join(self._output_directory, "state-%s.html" % state), STATE_BODY_TEMPLATE, { "page_title": html_protect("Packages in state " + state + " in " + self._config.section), "state": html_protect(state), "list": vlist, "aside": aside, }) def archive_logfile(self, vdir, log): archivedir = os.path.join("archive", vdir) if not os.path.exists(archivedir): os.makedirs(archivedir) os.rename(os.path.join(vdir, log), os.path.join("archive", vdir, log)) def cleanup_removed_packages(self, logs_by_dir): vdirs = logs_by_dir.keys() vdirs.remove("reserved") for vdir in vdirs: for log in sorted(logs_by_dir[vdir]): if log.endswith(".log"): package, version = log[:-len(".log")].split("_") if not self._binary_db.has_package(package): logging.debug("Package %s was removed, archiving %s/%s" % (package, vdir, log)) self.archive_logfile(vdir, log) logs_by_dir[vdir].remove(log) else: current = self._binary_db.get_version(package) if version != current: logging.debug("Archiving %s/%s, package is outdated (%s)" % (vdir, log, current)) self.archive_logfile(vdir, log) logs_by_dir[vdir].remove(log) def generate_html(self): logging.debug("Finding log files") dirs = ["pass", "fail", "bugged", "affected", "reserved", "untestable"] logs_by_dir = {} for vdir in dirs: logs_by_dir[vdir] = find_files_with_suffix(vdir, ".log") logging.debug("Archiving logs of obsolete packages") self.cleanup_removed_packages(logs_by_dir) logging.debug("Copying log files") copy_logs(logs_by_dir, self._output_directory) logging.debug("Removing old log files") remove_old_logs(logs_by_dir, self._output_directory) logging.debug("Writing per-dir HTML pages") self.print_by_dir(self._output_directory, logs_by_dir) total_packages = self.write_counts_summary() self.create_package_summaries(logs_by_dir) logging.debug("generate known problem *.tpl") dwke_process_section(self._config.section, '.', self._output_directory, self._problem_list, self._binary_db) logging.debug("Writing section index page") self.write_section_index_page(dirs, total_packages) logging.debug("Writing stats pages for %s" % self._config.section) self.write_state_pages() def generate_summary(self, web_host): summary_path = os.path.join(self._output_directory, "summary.json") if os.path.isfile(summary_path): os.unlink(summary_path) reporting_sections = self._config['json-sections'].split() if not reporting_sections or reporting_sections[0] == 'default': reporting_sections = [self._config.get_std_distro()] if reporting_sections[0] == 'none': logging.debug("Skipping summary") else: logging.debug("Generating summary") summary = pkgsummary.new_summary() for reporting_section in reporting_sections: for binpkg in self._binary_db.get_all_packages(): pkgname = binpkg["Package"] state = self._binary_db.get_package_state(pkgname) flag = pkgsummary.get_flag(state) block_cnt = 0 if flag == 'F': block_cnt = self._binary_db.block_count(pkgname) srcpkg = self._binary_db.get_source(pkgname) url = source_summary_url( web_host, self._doc_root, self._config.section, srcpkg) pkgsummary.add_summary( summary, reporting_section, srcpkg, flag, block_cnt, url) pkgsummary.write_summary(summary, summary_path) def generate_output(self, output_directory, section_names, problem_list, web_host): # skip output generation for disabled sections if int(self._config["max-reserved"]) == 0: return self._section_names = section_names self._section_navigation = create_section_navigation( self._section_names, self._config.section, self._doc_root) self._output_directory = os.path.abspath(os.path.join(output_directory, self._config.section)) if not os.path.exists(self._output_directory): os.makedirs(self._output_directory) self._problem_list = problem_list oldcwd = os.getcwd() os.chdir(self._section_directory) self.generate_html() os.chdir(oldcwd) self.generate_summary(web_host) def sections_by_precedence(sections): precedence = {} count = 0 for section in sections: config = Config(section=section, defaults_section="global") config.read(CONFIG_FILE) precedence[section] = (config["precedence"], count) count += 1 return sorted(sections, key=lambda x: precedence[x]) def generate_global_summary(dir, sections): json_name = "summary.json" logging.debug("Generating global summary") summary = pkgsummary.new_summary() for section in sections_by_precedence(sections): sec_path = os.path.join(dir, section, json_name) if os.path.isfile(sec_path): sec_summ = pkgsummary.read_summary(sec_path) summary = pkgsummary.merge_summary(summary, sec_summ) summary_path = os.path.join(dir, json_name) pkgsummary.write_summary(summary, summary_path) # START detect_well_known_errors def get_bug_text(logpath): bugpath = replace_ext(logpath, BUG_EXT) txt = "" if os.path.exists(bugpath): bf = open(bugpath, 'r') txt = bf.read() bf.close() return txt def populate_tpl(tmpl, vals): for key in vals: tmpl = re.sub("\$%s" % key, str(vals[key]), tmpl) return tmpl def update_tpl(basedir, section, problem, failures, logdict, ftpl, ptpl, pkgsdb): pkg_text = "" bugged_section = False for failure in failures: bin_pkg = get_pkg(failure.pkgspec) src_pkg = pkgsdb.get_source(bin_pkg) rdep_cnt = pkgsdb.rrdep_count(bin_pkg) bin_arch = "" if pkgsdb.get_control_header(bin_pkg, "Architecture") == "all": bin_arch = " [all]" if bugged_section is False and failure.where != 'fail': bugged_section = True pkg_text += "</ul><ul>\n" log = os.path.join(failure.where, failure.pkgspec + LOG_EXT) pkg_text += populate_tpl(ftpl, { 'LOG': log, 'PACKAGE': bin_pkg, 'ARCH': bin_arch, 'BUG': get_bug_text(log), 'RDEPS': rdep_cnt, 'SPKG': src_pkg, }) if len(pkg_text): return populate_tpl(ptpl, { 'HEADER': problem.HEADER, 'SECTION': section, 'HELPTEXT': problem.HELPTEXT, 'COMMAND': problem.get_command(), 'PACKAGE_LIST': pkg_text, 'COUNT': len(failures), }) return "" def update_html(section, html_dir, logdict, problem_list, failures, pkgsdb): for problem in problem_list: tpl_text = update_tpl(html_dir, section, problem, failures.filtered(problem.name), logdict, PKG_ERROR_TPL, PROB_TPL, pkgsdb) if len(tpl_text): with open(os.path.join(html_dir, problem.name[:-5] + TPL_EXT), 'w') as pf: pf.write(tpl_text) def dwke_process_section(section, sectiondir, htmldir, problem_list, pkgsdb): workdirs = [os.path.join(sectiondir, x) for x in KPR_DIRS] logdict = get_file_dict(workdirs, LOG_EXT) failures = FailureManager(logdict) failures.sort_by_bugged_and_rdeps(pkgsdb) update_html(section, htmldir, logdict, problem_list, failures, pkgsdb) # END detect_well_known_errors def make_bts_stats_graph(master_dir, out_dir): countsfile = os.path.join(master_dir, "bts_stats.txt") pngfile = os.path.join(out_dir, "images", "bts_stats.png") grdevices = importr('grDevices') grdevices.png(file=pngfile, width=1600, height=900, pointsize=10, res=100) r = robjects.r r('t <- (read.table("' + countsfile + '",sep=",",header=1,row.names=1))') r('cname <- c("date",rep(colnames(t)))') # make graph since day 1 r('v <- t[0:nrow(t),0:4]') # tango colors again: r('palette(c("#8ae234", "#ef2929", "#4e9a06", "#a40000"))') r('barplot(t(v),col = 1:5, \ main="Bugs with usertag=piuparts and user=debian-qa@lists.debian.org", \ xlab="", ylab="Total number of RC and non-RC bugs submitted and closed", space=0, border=NA)') r('legend(x="bottom",legend=colnames(t), ncol=2,fill=1:5,xjust=0.5,yjust=0,bty="n")') grdevices.dev_off() def main(): setup_logging(logging.DEBUG, None) global_config = Config(section="global") global_config.read(CONFIG_FILE) if global_config["proxy"]: os.environ["http_proxy"] = global_config["proxy"] section_names = global_config["sections"].split() process_section_names = section_names if len(sys.argv) > 1: process_section_names = sys.argv[1:] master_directory = global_config["master-directory"] output_directory = global_config["output-directory"] web_host = global_config["web-host"] doc_root = global_config["doc-root"].strip() if not doc_root.startswith("/"): doc_root = "/" + doc_root if doc_root.endswith("/"): doc_root = doc_root[:-1] problem_list = create_problem_list(global_config['known-problem-directory']) if os.path.exists(master_directory): packagedb_cache = {} write_file(os.path.join(output_directory, "sections.yaml"), yaml.dump(section_names, default_flow_style=False)) for section_name in process_section_names: try: section = Section(section_name, master_directory, doc_root, packagedb_cache=packagedb_cache) section.generate_output(output_directory, section_names, problem_list, web_host) except MissingSection as e: logging.error("Configuration Error in section '%s': %s" % (section_name, e)) except HTTPError as e: logging.error("Failed to fetch Packages files for section '%s' : %s" % (section_name, e)) # static pages logging.debug("Writing static pages") for page in ("index", "bug_howto"): tpl = os.path.join(output_directory, page + ".tpl") INDEX_BODY = "".join(read_file(tpl)) write_template_html( os.path.join(output_directory, page + ".html"), INDEX_BODY, { "page_title": "About piuparts.debian.org and News", "section_navigation": create_section_navigation(section_names, "sid", doc_root), "doc_root": doc_root, }) # create graph about bugs filed try: make_bts_stats_graph(master_directory, output_directory); except: logging.debug("Error generating the graph images, probably python-rpy2 is not installed, disabling graphs.") generate_global_summary(output_directory, section_names) logging.debug("Done") else: logging.debug("Warning: %s does not exist!?! Creating it for you now." % master_directory) os.makedirs(master_directory) if __name__ == "__main__": main() # vi:set et ts=4 sw=4 : �����������������������������������������������������������������������������������������piuparts/piuparts-slave.py��������������������������������������������������������������������������0000644�0000000�0000000�00000107340�13061276553�013161� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # -*- coding: utf-8 -*- # # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright © 2011-2017 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """Distributed piuparts processing, slave program Lars Wirzenius <liw@iki.fi> """ import os import sys import stat import time import logging from signal import alarm, signal, SIGALRM, SIGINT, SIGKILL, SIGHUP import subprocess import fcntl import random import ConfigParser import apt_pkg import pipes import piupartslib.conf import piupartslib.packagesdb from piupartslib.conf import MissingSection apt_pkg.init_system() CONFIG_FILE = "/etc/piuparts/piuparts.conf" DISTRO_CONFIG_FILE = "/etc/piuparts/distros.conf" MAX_WAIT_TEST_RUN = 90 * 60 interrupted = False old_sigint_handler = None got_sighup = False def setup_logging(log_level, log_file_name): logger = logging.getLogger() logger.setLevel(log_level) formatter = logging.Formatter(fmt="%(asctime)s %(message)s", datefmt="%H:%M:%S") handler = logging.StreamHandler(sys.stderr) handler.setFormatter(formatter) logger.addHandler(handler) if log_file_name: handler = logging.FileHandler(log_file_name) logger.addHandler(handler) class Config(piupartslib.conf.Config): def __init__(self, section="slave", defaults_section=None): self.section = section piupartslib.conf.Config.__init__(self, section, { "sections": "slave", "basetgz-sections": "", "idle-sleep": 300, "max-tgz-age": 2592000, "min-tgz-retry-delay": 21600, "master-host": None, "master-user": None, "master-command": None, "proxy": None, "mirror": None, "piuparts-command": "sudo piuparts", "piuparts-flags": "", "tmpdir": None, "distro": None, "area": None, "components": None, "chroot-tgz": None, "upgrade-test-distros": None, "basetgz-directory": ".", "max-reserved": 1, "debug": "no", "keep-sources-list": "no", "arch": None, "precedence": "1", "slave-load-max": None, }, defaults_section=defaults_section) class Alarm(Exception): pass def alarm_handler(signum, frame): raise Alarm def sigint_handler(signum, frame): global interrupted interrupted = True print '\nSlave interrupted by the user, waiting for the current test to finish.' print 'Press Ctrl-C again to abort now.' signal(SIGINT, old_sigint_handler) def sighup_handler(signum, frame): global got_sighup got_sighup = True print 'SIGHUP: Will flush finished logs.' class MasterIsBusy(Exception): def __init__(self): self.args = "Master is busy, retry later", class MasterNotOK(Exception): def __init__(self): self.args = "Master did not respond with 'ok'", class MasterDidNotGreet(Exception): def __init__(self): self.args = "Master did not start with 'hello'", class MasterCommunicationFailed(Exception): def __init__(self): self.args = "Communication with master failed", class MasterIsCrazy(Exception): def __init__(self): self.args = "Master said something unexpected", class MasterCantRecycle(Exception): def __init__(self): self.args = "Master has nothing to recycle", class Slave: def __init__(self): self._to_master = None self._from_master = None self._master_host = None self._master_user = None self._master_command = None self._section = None def _readline(self): try: line = self._from_master.readline() except IOError: raise MasterCommunicationFailed() logging.debug("<< " + line.rstrip()) return line def _writeline(self, *words): line = " ".join(words) logging.debug(">> " + line) try: self._to_master.write(line + "\n") self._to_master.flush() except IOError: raise MasterCommunicationFailed() def set_master_host(self, host): logging.debug("Setting master host to %s" % host) if self._master_host != host: self.close() self._master_host = host def set_master_user(self, user): logging.debug("Setting master user to %s" % user) if self._master_user != user: self.close() self._master_user = user def set_master_command(self, cmd): logging.debug("Setting master command to %s" % cmd) if self._master_command != cmd: self.close() self._master_command = cmd def set_section(self, section): logging.debug("Setting section to %s" % section) self._section = section def connect_to_master(self): if not self._is_connected(): self._initial_connect() self._select_section() def _is_connected(self): return self._to_master and self._from_master def _initial_connect(self): logging.info("Connecting to %s" % self._master_host) ssh_command = ["ssh", "-x"] if self._master_user: ssh_command.extend(["-l", self._master_user]) ssh_command.append(self._master_host) ssh_command.append(self._master_command or "command-is-set-in-authorized_keys") p = subprocess.Popen(ssh_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE) self._to_master = p.stdin self._from_master = p.stdout line = self._readline() if line != "hello\n": raise MasterDidNotGreet() def _select_section(self): self._writeline("section", self._section) line = self._readline() if line == "busy\n": raise MasterIsBusy() elif line != "ok\n": raise MasterNotOK() logging.debug("Connected to master") def close(self): if self._from_master is None and self._to_master is None: return logging.debug("Closing connection to master") if self._from_master is not None: self._from_master.close() if self._to_master is not None: self._to_master.close() self._from_master = self._to_master = None logging.info("Connection to master closed") def send_log(self, section, pass_or_fail, filename): logging.info("Sending log file %s/%s" % (section, filename)) basename = os.path.basename(filename) package, rest = basename.split("_", 1) version = rest[:-len(".log")] self._writeline(pass_or_fail, package, version) f = file(filename, "r") for line in f: if line.endswith("\n"): line = line[:-1] self._writeline(" " + line) f.close() self._writeline(".") line = self._readline() if line != "ok\n": raise MasterNotOK() def get_status(self, section): self._writeline("status") line = self._readline() words = line.split() if words and words[0] == "ok": logging.info("Master " + section + " status: " + " ".join(words[1:])) else: raise MasterIsCrazy() def enable_recycling(self): self._writeline("recycle") line = self._readline() words = line.split() if line != "ok\n": raise MasterCantRecycle() def get_idle(self): self._writeline("idle") line = self._readline() words = line.split() if words and words[0] == "ok" and len(words) == 2: return int(words[1]) else: raise MasterIsCrazy() def reserve(self): self._writeline("reserve") line = self._readline() words = line.split() if words and words[0] == "ok": logging.info("Reserved for us: %s %s" % (words[1], words[2])) self.remember_reservation(words[1], words[2]) return True elif words and words[0] == "error": logging.info("Master didn't reserve anything (more) for us") return False else: raise MasterIsCrazy() def unreserve(self, filename): basename = os.path.basename(filename) package, rest = basename.split("_", 1) version = rest[:-len(".log")] logging.info("Unreserve: %s %s" % (package, version)) self._writeline("unreserve", package, version) line = self._readline() if line != "ok\n": raise MasterNotOK() def _reserved_filename(self, name, version): return os.path.join("reserved", "%s_%s.log" % (name, version)) def remember_reservation(self, name, version): create_file(self._reserved_filename(name, version), "") def get_reserved(self): vlist = [] for basename in os.listdir("reserved"): if "_" in basename and basename.endswith(".log"): name, version = basename[:-len(".log")].split("_", 1) vlist.append((name, version)) return vlist def forget_reserved(self, name, version): try: os.remove(self._reserved_filename(name, version)) except os.error: pass class Section: def __init__(self, section, slave=None): self._config = Config(section=section, defaults_section="global") self._config.read(CONFIG_FILE) self._distro_config = piupartslib.conf.DistroConfig( DISTRO_CONFIG_FILE, self._config["mirror"]) self._error_wait_until = 0 self._idle_wait_until = 0 self._recycle_wait_until = 0 self._tarball_wait_until = 0 self._slave_directory = os.path.abspath(section) if not os.path.exists(self._slave_directory): os.makedirs(self._slave_directory) if self._config["debug"] in ["yes", "true"]: self._logger = logging.getLogger() self._logger.setLevel(logging.DEBUG) if int(self._config["max-reserved"]) > 0: self._check_tarball() for rdir in ["new", "pass", "fail", "untestable", "reserved"]: rdir = os.path.join(self._slave_directory, rdir) if not os.path.exists(rdir): os.mkdir(rdir) self._slave = slave or Slave() def _throttle_if_overloaded(self): global interrupted if interrupted or got_sighup: return if self._config["slave-load-max"] is None: return load_max = float(self._config["slave-load-max"]) if load_max < 1.0: return if os.getloadavg()[0] <= load_max: return load_resume = max(load_max - 1.0, 0.9) secs = random.randrange(30, 90) self._slave.close() while True: load = os.getloadavg()[0] if load <= load_resume: break logging.info("Sleeping due to high load (%.2f)" % load) try: time.sleep(secs) except KeyboardInterrupt: interrupted = True if interrupted or got_sighup: break if secs < 300: secs += random.randrange(30, 90) def _connect_to_master(self, recycle=False): self._slave.set_master_host(self._config["master-host"]) self._slave.set_master_user(self._config["master-user"]) self._slave.set_master_command(self._config["master-command"]) self._slave.set_section(self._config.section) self._slave.connect_to_master() if recycle: self._slave.enable_recycling() def _get_tarball(self): basetgz = self._config["chroot-tgz"] or \ self._distro_config.get_basetgz(self._config.get_start_distro(), self._config.get_arch()) return os.path.join(self._config["basetgz-directory"], basetgz) def _check_tarball(self): if int(self._config["max-tgz-age"]) < 0: return oldcwd = os.getcwd() os.chdir(self._slave_directory) tgz = self._get_tarball() max_tgz_age = int(self._config["max-tgz-age"]) min_tgz_retry_delay = int(self._config["min-tgz-retry-delay"]) ttl = 0 needs_update = not os.path.exists(tgz) if not needs_update and max_tgz_age > 0: # tgz exists and age is limited, so check age now = time.time() age = now - os.path.getmtime(tgz) ttl = max_tgz_age - age logging.info("Check-replace %s: age=%d vs. max=%d" % (tgz, age, max_tgz_age)) if ttl < 0: if os.path.exists(tgz + ".log"): age = now - os.path.getmtime(tgz + ".log") ttl = min_tgz_retry_delay - age logging.info("Limit-replace %s: last-retry=%d vs. min=%d" % (tgz, age, min_tgz_retry_delay)) if ttl < 0: needs_update = True logging.info("%s too old. Forcing re-creation" % tgz) if needs_update: create_chroot(self._config, tgz, self._config.get_start_distro()) ttl = min_tgz_retry_delay self._tarball_wait_until = time.time() + ttl os.chdir(oldcwd) def _count_submittable_logs(self): files = 0 subdirs = ["pass", "fail", "untestable"] if interrupted: subdirs += ["reserved", "new"] for logdir in subdirs: for basename in os.listdir(os.path.join(self._slave_directory, logdir)): if basename.endswith(".log"): files += 1 return files def precedence(self): return int(self._config["precedence"]) def sleep_until(self, recycle=False): if recycle: return max(self._error_wait_until, self._recycle_wait_until) return max(self._error_wait_until, self._idle_wait_until) def run(self, do_processing=True, recycle=False): if time.time() < self.sleep_until(recycle=recycle): return 0 self._throttle_if_overloaded() self._config = Config(section=self._config.section, defaults_section="global") try: self._config.read(CONFIG_FILE) except MissingSection: logging.info("unknown section " + self._config.section) self._error_wait_until = time.time() + 3600 return 0 self._distro_config = piupartslib.conf.DistroConfig( DISTRO_CONFIG_FILE, self._config["mirror"]) if interrupted or got_sighup: do_processing = False if do_processing and time.time() > self._tarball_wait_until: self._check_tarball() if self._config.get_distro() == "None": # section is for tarball creation only self._idle_wait_until = self._tarball_wait_until + 60 self._recycle_wait_until = self._tarball_wait_until + 3600 return 0 if interrupted or got_sighup: do_processing = False if not do_processing and self._count_submittable_logs() == 0: return 0 logging.info("-------------------------------------------") action = "Running" if recycle: action = "Recycling" if not do_processing: action = "Flushing" logging.info("%s section %s (precedence=%d)" % (action, self._config.section, self.precedence())) if int(self._config["max-reserved"]) == 0: logging.info("disabled") self._error_wait_until = time.time() + 12 * 3600 return 0 if not self._config.get_distro() and not self._config.get_distros(): logging.error("neither 'distro' nor 'upgrade-test-distros' configured") self._error_wait_until = time.time() + 3600 return 0 with open(os.path.join(self._slave_directory, "slave.lock"), "we") as lock: oldcwd = os.getcwd() os.chdir(self._slave_directory) try: fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) except IOError: logging.info("busy") self._error_wait_until = time.time() + 900 else: if self._talk_to_master(fetch=do_processing, recycle=recycle, unreserve=interrupted): if do_processing: if not self._slave.get_reserved(): self._idle_wait_until = time.time() + int(self._config["idle-sleep"]) if recycle: self._recycle_wait_until = self._idle_wait_until + 3600 else: processed = self._process() if got_sighup and self._slave.get_reserved(): # keep this section at the front of the round-robin runnable queue self._idle_wait_until = 0 self._recycle_wait_until = 0 else: # put this section at the end of the round-robin runnable queue self._idle_wait_until = time.time() self._recycle_wait_until = time.time() return processed finally: os.chdir(oldcwd) return 0 def _talk_to_master(self, fetch=False, unreserve=False, recycle=False): flush = self._count_submittable_logs() > 0 fetch = fetch and not self._slave.get_reserved() if not flush and not fetch: return True try: self._connect_to_master(recycle=recycle) except KeyboardInterrupt: raise except MasterIsBusy: logging.error("master is busy") self._error_wait_until = time.time() + random.randrange(60, 180) except MasterCantRecycle: logging.error("master has nothing to recycle") self._recycle_wait_until = max(time.time(), self._idle_wait_until) + 3600 except (MasterDidNotGreet, MasterIsCrazy, MasterCommunicationFailed, MasterNotOK): logging.error("connection to master failed") self._error_wait_until = time.time() + 900 self._slave.close() else: try: for logdir in ["pass", "fail", "untestable"]: for basename in os.listdir(logdir): if basename.endswith(".log"): fullname = os.path.join(logdir, basename) self._slave.send_log(self._config.section, logdir, fullname) os.remove(fullname) if unreserve: for logdir in ["new", "reserved"]: for basename in os.listdir(logdir): if basename.endswith(".log"): fullname = os.path.join(logdir, basename) self._slave.unreserve(fullname) os.remove(fullname) if fetch: max_reserved = int(self._config["max-reserved"]) idle = self._slave.get_idle() if idle > 0: idle = min(idle, int(self._config["idle-sleep"])) logging.info("idle (%d)" % idle) if not recycle: self._idle_wait_until = time.time() + idle else: self._recycle_wait_until = time.time() + idle return 0 while len(self._slave.get_reserved()) < max_reserved and self._slave.reserve(): pass self._slave.get_status(self._config.section) except MasterNotOK: logging.error("master did not respond with 'ok'") self._error_wait_until = time.time() + 900 self._slave.close() except (MasterIsCrazy, MasterCommunicationFailed): logging.error("communication with master failed") self._error_wait_until = time.time() + 900 self._slave.close() else: return True return False def _process(self): global interrupted self._slave.close() packagenames = set([x[0] for x in self._slave.get_reserved()]) packages_files = {} for distro in [self._config.get_distro()] + self._config.get_distros(): if distro not in packages_files: try: pf = piupartslib.packagesdb.PackagesFile() pf.load_packages_urls( self._distro_config.get_packages_urls( distro, self._config.get_area(), self._config.get_arch()), packagenames) packages_files[distro] = pf except IOError: logging.error("failed to fetch packages file for %s" % distro) self._error_wait_until = time.time() + 900 return 0 except KeyboardInterrupt: interrupted = True del packagenames test_count = 0 self._check_tarball() if not os.path.exists(self._get_tarball()): self._error_wait_until = time.time() + 300 for package_name, version in self._slave.get_reserved(): self._throttle_if_overloaded() if interrupted or got_sighup: break if not os.path.exists(self._get_tarball()): logging.error("Missing chroot-tgz %s" % self._get_tarball()) break test_count += 1 self._test_package(package_name, version, packages_files) self._slave.forget_reserved(package_name, version) self._talk_to_master(unreserve=interrupted) return test_count def _test_package(self, pname, pvers, packages_files): global old_sigint_handler old_sigint_handler = signal(SIGINT, sigint_handler) logging.info("Testing package %s/%s %s" % (self._config.section, pname, pvers)) output_name = log_name(pname, pvers) logging.debug("Opening log file %s" % output_name) new_name = os.path.join("new", output_name) output = file(new_name, "we") output.write(time.strftime("Start: %Y-%m-%d %H:%M:%S %Z\n", time.gmtime())) distupgrade = len(self._config.get_distros()) > 1 command = self._config["piuparts-command"].split() if self._config["piuparts-flags"]: command.extend(self._config["piuparts-flags"].split()) if "http_proxy" in os.environ: command.extend(["--proxy", os.environ["http_proxy"]]) if self._config["mirror"]: mirror = self._config["mirror"] if self._config["components"]: mirror += " " + self._config["components"] command.extend(["--mirror", mirror]) if self._config["tmpdir"]: command.extend(["--tmpdir", self._config["tmpdir"]]) command.extend(["--arch", self._config.get_arch()]) command.extend(["-b", self._get_tarball()]) if not distupgrade: command.extend(["-d", self._config.get_distro()]) command.append("--no-upgrade-test") else: for distro in self._config.get_distros(): command.extend(["-d", distro]) if self._config["keep-sources-list"] in ["yes", "true"]: command.append("--keep-sources-list") command.extend(["--apt", "%s=%s" % (pname, pvers)]) subdir = "fail" ret = 0 if not distupgrade: distro = self._config.get_distro() if not pname in packages_files[distro]: output.write("Package %s not found in %s\n" % (pname, distro)) ret = -10001 else: package = packages_files[distro][pname] if pvers != package["Version"]: output.write("Package %s %s not found in %s, %s is available\n" % (pname, pvers, distro, package["Version"])) ret = -10002 output.write("\n") package.dump(output) output.write("\n") else: distros = self._config.get_distros() if distros: # the package must exist somewhere for distro in distros: if pname in packages_files[distro]: break else: output.write("Package %s not found in any distribution\n" % pname) ret = -10003 # the package must have the correct version in the distupgrade target distro distro = distros[-1] if not pname in packages_files[distro]: # the package may "disappear" in the distupgrade target distro if pvers == "None": pass else: output.write("Package %s not found in %s\n" % (pname, distro)) ret = -10004 else: package = packages_files[distro][pname] if pvers != package["Version"]: output.write("Package %s %s not found in %s, %s is available\n" % (pname, pvers, distro, package["Version"])) ret = -10005 for distro in distros: output.write("\n[%s]\n" % distro) if pname in packages_files[distro]: packages_files[distro][pname].dump(output) output.write("\n") if ret == 0: prev = "~" for distro in distros: if pname in packages_files[distro]: v = packages_files[distro][pname]["Version"] if not apt_pkg.version_compare(prev, v) <= 0: output.write("Upgrade to %s requires downgrade: %s > %s\n" % (distro, prev, v)) ret = -10006 prev = v else: ret = -10010 if ret != 0: subdir = "untestable" if ret == 0: output.write("Executing: %s\n" % " ".join(quote_spaces(command))) ret, f = run_test_with_timeout(command, MAX_WAIT_TEST_RUN) if not f or f[-1] != '\n': f += '\n' output.write(f.replace('\033', '[ESC]')) lastline = f.split('\n')[-2] if ret < 0: output.write(" *** Process KILLED - exceed maximum run time ***\n") elif not "piuparts run ends" in lastline: ret += 1024 output.write(" *** PIUPARTS OUTPUT INCOMPLETE ***\n") output.write("\n") output.write("ret=%d\n" % ret) output.write(time.strftime("End: %Y-%m-%d %H:%M:%S %Z\n", time.gmtime())) output.close() if ret == 0: subdir = "pass" os.rename(new_name, os.path.join(subdir, output_name)) logging.debug("Done with %s: %s (%d)" % (output_name, subdir, ret)) signal(SIGINT, old_sigint_handler) def log_name(package, version): return "%s_%s.log" % (package, version) def quote_spaces(vlist): return [pipes.quote(x) for x in vlist] def run_test_with_timeout(cmd, maxwait, kill_all=True): def terminate_subprocess(p, kill_all): pids = [p.pid] if kill_all: ps = subprocess.Popen(["ps", "--no-headers", "-o", "pid", "--ppid", "%d" % p.pid], stdout=subprocess.PIPE) stdout, stderr = ps.communicate() pids.extend([int(pid) for pid in stdout.split()]) if p.poll() is None: print 'Sending SIGINT...' try: os.killpg(os.getpgid(p.pid), SIGINT) except OSError: pass # piuparts has 30 seconds to clean up after Ctrl-C for i in range(60): time.sleep(0.5) if p.poll() is not None: break if p.poll() is None: print 'Sending SIGTERM...' p.terminate() # piuparts has 5 seconds to clean up after SIGTERM for i in range(10): time.sleep(0.5) if p.poll() is not None: break if p.poll() is None: print 'Sending SIGKILL...' p.kill() for pid in pids: if pid > 0: try: os.kill(pid, SIGKILL) print "Killed %d" % pid except OSError: pass logging.debug("Executing: %s" % " ".join(quote_spaces(cmd))) stdout = "" p = subprocess.Popen(cmd, preexec_fn=os.setpgrp, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) if maxwait > 0: signal(SIGALRM, alarm_handler) alarm(maxwait) try: stdout, stderr = p.communicate() alarm(0) except Alarm: terminate_subprocess(p, kill_all) return -1, stdout except KeyboardInterrupt: print '\nSlave interrupted by the user, cleaning up...' try: terminate_subprocess(p, kill_all) except KeyboardInterrupt: print '\nTerminating piuparts was interrupted... manual cleanup still neccessary.' raise raise ret = p.returncode if ret in [124, 137]: # process was terminated by the timeout command ret = -ret return ret, stdout def create_chroot(config, tarball, distro): command = config["piuparts-command"].split() if config["piuparts-flags"]: command.extend(config["piuparts-flags"].split()) if "http_proxy" in os.environ: command.extend(["--proxy", os.environ["http_proxy"]]) if config["mirror"]: mirror = config["mirror"] if config["components"]: mirror += " " + config["components"] command.extend(["--mirror", mirror]) if config["tmpdir"]: command.extend(["--tmpdir", config["tmpdir"]]) command.extend(["--arch", config.get_arch()]) command.extend(["-d", distro]) command.extend(["-s", tarball + ".new"]) command.extend(["--apt", "dpkg"]) output_name = tarball + ".log" with open(output_name, "we") as output: try: fcntl.flock(output, fcntl.LOCK_EX | fcntl.LOCK_NB) except IOError: logging.info("Creation of tarball %s already in progress." % tarball) else: logging.info("Creating new tarball %s" % tarball) output.write(time.strftime("Start: %Y-%m-%d %H:%M:%S %Z\n\n", time.gmtime())) output.write("Executing: " + " ".join(quote_spaces(command)) + "\n\n") logging.debug("Executing: " + " ".join(quote_spaces(command))) p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) for line in p.stdout: output.write(line) logging.debug(">> " + line.rstrip()) p.wait() output.write(time.strftime("\nEnd: %Y-%m-%d %H:%M:%S %Z\n", time.gmtime())) if os.path.exists(tarball + ".new"): os.rename(tarball + ".new", tarball) else: logging.error("Tarball creation failed, see %s" % output_name) def create_file(filename, contents): f = file(filename, "w") f.write(contents) f.close() def main(): setup_logging(logging.INFO, None) signal(SIGHUP, sighup_handler) # For supporting multiple architectures and suites, we take command-line # argument(s) referring to section(s) in the configuration file. # If no argument is given, the "sections" entry from the "global" section # is used. section_names = [] global_config = Config(section="global") global_config.read(CONFIG_FILE) if global_config["proxy"]: os.environ["http_proxy"] = global_config["proxy"] if len(sys.argv) > 1: section_names = sys.argv[1:] else: section_names = global_config["sections"].split() section_names += global_config["basetgz-sections"].split() persistent_connection = Slave() sections = [] for section_name in section_names: try: sections.append(Section(section_name, persistent_connection)) except MissingSection: # ignore unknown sections pass if not sections: logging.error("no sections found") return while True: global got_sighup test_count = 0 for section in sorted(sections, key=lambda section: (section.precedence(), section.sleep_until())): test_count += section.run(do_processing=(test_count == 0)) if test_count == 0 and got_sighup: # clear SIGHUP state after flushing all sections got_sighup = False continue if test_count == 0: # try to recycle old logs # round robin recycling of all sections is ensured by the recycle_wait_until timestamps idle_until = min([section.sleep_until() for section in sections]) for section in sorted(sections, key=lambda section: section.sleep_until(recycle=True)): test_count += section.run(recycle=True) if test_count > 0 and idle_until < time.time(): break if interrupted: raise KeyboardInterrupt if test_count == 0 and not got_sighup: now = time.time() sleep_until = min([now + int(global_config["idle-sleep"])] + [section.sleep_until() for section in sections]) if (sleep_until > now): to_sleep = max(60, sleep_until - now) persistent_connection.close() logging.info("Nothing to do, sleeping for %d seconds." % to_sleep) time.sleep(to_sleep) if __name__ == "__main__": try: main() except KeyboardInterrupt: print '' print 'Slave interrupted by the user, exiting...' sys.exit(1) # vi:set et ts=4 sw=4 : ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piuparts.1.txt�����������������������������������������������������������������������������0000644�0000000�0000000�00000045106�13061276553�012400� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts(1) =========== :doctype: manpage :revdate: 2017-03-06 NAME ---- piuparts - .deb installation, upgrade, and removal testing suite SYNOPSIS -------- *piuparts* ['-apvV'] ['-d' _distro_] ['-i' _filename_] ['-I' _regexp_] ['-l' _logfile_] ['-m' _url_] ['--bindmount' _dir_] [_package_]... [_changes_file_]... DESCRIPTION ----------- *piuparts* tests that Debian packages handle installation, upgrading, and removal correctly. It does this by creating a minimal Debian installation in a chroot, and installing, upgrading, and removing packages in that environment, and comparing the state of the directory tree before and after. *piuparts* reports any files that have been added, removed, or modified during this process. *piuparts* is meant as a quality assurance tool for people who create Debian packages to test them before they upload them to the Debian package archive. By default, piuparts can do three different tests: . A simple install-purge test within one Debian distribution (chosen with the '-d' option, unstable by default). It sets up the chroot with the desired distribution, then installs and purges the packages, and reports problems. . A simple install-upgrade-purge test within one Debian distribution. This test is like the install-purge test, but it installs the packages first via *apt-get* and then from the package files given on the command line. If the command line has package names (option '--apt' used), or no tested package is known to *apt-get* (new packages), this test is skipped, otherwise it is performed automatically. . An upgrade test between Debian releases. This test is enabled by using the '-d' option multiple times and disables the other two tests. It sets up the chroot with the first distribution named, then upgrades it to each successive one, and then remembers the directory tree state at the end. After this, it starts over with the chroot of the first distribution, installs the desired packages (via *apt-get*), and does the successive upgrading (via *apt-get dist-upgrade*). Then, if package files (and not just package names) were given on the command line, it installs them. Finally, it reports problems against the state of the directory tree at the last distribution compared with the state without the packages having been installed. This test can be quite slow to execute. + Note that this does not work with experimental, because *apt-get* does not automatically upgrade to packages in experimental. To test a particular package or group of packages in experimental, use the second test. Command line arguments are the paths to package files (e.g., _piuparts_1.0-1_all.deb_), paths to changes files (e.g., _piuparts_1.0-1_i386.changes_), or names of packages, if the '--apt' option is given. When processing changes files, by default, all packages in a changes file will be processed together with all individual packages given on the command line. Then each package given on the command line is processed in a single group. If the '--single-changes-list' is used, the packages in all changes files are processed together along with any individual packages that were given on the command line. To avoid this behaviour, it is possible to specify '--single-packages'. *piuparts* outputs to the standard output some log messages to show what is going on. If a log file is used, the messages go there as well. *piuparts* needs to be run as root. OPTIONS ------- Options must come before the other command line arguments. *-a*, *--apt*:: The package arguments on the command line are to be treated as package names and installed via *apt-get* instead of being names of package files, to be installed via *dpkg -i*. *--allow-database*:: Allow starting MySQL and PostgreSQL database servers in the chroot for packages requiring database access in their maintainer scripts. Do not use this option if there is already a database server running on the system running piuparts (or piuparts-slave)! In master-slave setups with multiple slaves running on one host collisions may occur, these will be detected by 'detect_piuparts_issues' and the affected packages will be tested again. *--arch*='arch':: Create chroot and run tests for (non-default) architecture 'arch'. The default is the output from 'dpkg --print-architecture'. *-b* 'tarball', *--basetgz*='tarball':: Use tarball as the contents of the initial chroot, instead of building a new one with debootstrap. + The tarball can be created with the '-s' option, or you can use one that *pbuilder* has created (see '-p'). If you create one manually, make sure the root of the chroot is the root of the tarball. *--bindmount*='dir':: Bind-mount a directory inside the chroot. *-d* 'name', *--distribution*='name':: Which Debian distribution to use: a code name (for example jessie, stretch or sid) or experimental. The default is sid (=unstable). *-D* 'flavor', *--defaults*='flavor':: Use default settings suitable for a particular flavor of Debian: either debian or ubuntu. The default is debian. *--do-not-verify-signatures*:: Do not verify signatures from the Release files when running debootstrap. Also set APT::Get::AllowUnauthenticated accordingly in /etc/apt/apt.conf in the chroots. *--dpkg-force-confdef*:: Make dpkg use --force-confdef, which lets dpkg always choose the default action when a modified conffile is found. This option will make piuparts ignore errors it was designed to report and therefore should only be used to hide problems in depending packages. 'This option shall normally not be used.' (See #466118.) *--dpkg-noforce-unsafe-io*:: Prevent running dpkg with --force-unsafe-io. --force-unsafe-io causes dpkg to skip certain file system syncs known to cause substantial performance degradation on some filesystems. Thus, including this option reverts to safe but slower behavior. The --dpkg-noforce-unsafe-io is required for running tests on distributions older than squeeze. *--no-eatmydata*:: Prevent use of eatmydata. The --no-eatmydata option is required for running tests on squeeze or older distributions. *--extra-old-packages*='pkg1[,pkg2]...':: Install additional old packages before upgrading. Allows testing package renames/merges where the old package is no longer available in the new distribution and the new one utilizes Conflicts/Replaces. The argument is a comma separated list of package names and the option can be given multiple times. For install/purge tests these packages will be installed before the package that is to be tested. *-e* 'dirname', *--existing-chroot*='dirname':: Use the specified directory as source for the new chroot, instead of building a new one with debootstrap. This is similar to '--basetgz', but the contents are not archived. See also the --hard-link option. *--distupgrade-to-testdebs*:: Use the "testdebs" repository to override the packages in the distupgrade target distribution. This allows one to test complex upgrade paths before the packages enter the archive. *--extra-repo*='deb-line':: Provide an additional (unparsed) line to be appended to sources.list, e.g. 'deb <URL> <distrib> <components>' or 'deb file:// </bind/mount> ./' Useful for e.g. backports, security or local repositories that cannot be handled by '--mirror'. May be repeated to add more than one line. *--fake-essential-packages*='pkg1[,pkg2]...':: Install additional packages in the base chroot that are not removed after the test. These are available during purge and for checking against mistreatment. Takes a comma separated list of package names and can be given multiple times. *--hard-link*:: When the --existing-chroot option is used, and the source directory is on the same filesystem, hard-link files instead of copying them. This is faster, but any modifications to files will be reflected in the originals. *-i* 'filename', *--ignore*='filename':: Add a filename to the list of filenames to be ignored when comparing changes before and after installation. By default, piuparts ignores files that always change during a package installation and uninstallation, such as *dpkg* status files. The filename should be relative to the root of the chroot (e.g., _var/lib/dpkg/status_). This option can be used as many times as necessary. *-I* 'regexp', *--ignore-regexp*='regexp':: Add a regular expression pattern to the list of patterns for filenames to be ignored when comparing changes before and after installation. This option can be used as many times as necessary. *--install-purge-install*:: Purge package after installation and reinstall. All depedencies are installed during purge. *--install-recommends*:: Enable installation of Recommends. *--install-suggests*:: Enable installation of Suggests. *--install-remove-install*:: Remove package after installation and reinstall. For testing installation in config-files-remaining state. *-k*, *--keep-tmpdir*:: Don't remove the temporary directory for the chroot when the program ends. *-K*, *--keyring*='filename':: Use FILE as the keyring to use with debootstrap when creating chroots. *--keep-sources-list*:: Don't modify the chroot's etc/apt/sources.list. *--list-installed-files*:: List the files added to the chroot after the installation of the package and after the installation of the package dependencies. *--lvm-volume*='lvm-volume':: Use the specified lvm-volume as source for the chroot, instead of building a new one with debootstrap. This creates a snapshot of the given LVM volume and mounts it to the chroot path. *--lvm-snapshot-size*='snapshot-size':: Use the specified snapshot-size as snapshot size when creating a new LVM snapshot (default: 1G) *-l* 'filename', *--log-file*='filename':: Append log file to _filename_ in addition to the standard output. *--log-level*='level':: Display messages from loglevel LEVEL, possible values are: error, info, dump, debug. The default is dump. *-m* 'url', *--mirror*='url':: Which Debian mirror to use. The default is the first mirror named in _/etc/apt/sources.list_ or _http://deb.debian.org/debian_ if none is found. This option may be used multiple times to use multiple mirrors. Only the first mirror is used with *debootstrap*. + The 'components' that are used for a mirror can also be set with this option: a space separated list within the same argument (so you need to quote the entire argument in the shell). If no components are given explicitly, the usual Debian components are used (main, contrib, and non-free). For the mirrors read from _/etc/apt/sources.list_, the components are read from the same place. + Note that file: addresses works if the directories are made accessible from within the chroot with '--bindmount'. *--no-adequate*:: Don't run adequate after installation. The default is to run adequate, provided it is installed. *--no-check-valid-until*:: Set apt option Acquire::Check-Valid-Until=false in the chroot to ignore the expiration of Release files. This is needed for testing archived releases. *--no-diversions*:: Don't check for broken diversions. *-n*, *--no-ignores*:: Forget all built-in and other ignores that have been set so far. Any '-i' or '-I' arguments that come after this one will be obeyed, but none of the ones that come before. *-N*, *--no-symlinks*:: Don't check for broken symlinks. *--fail-if-inadequate*:: Fail on inadequate results from running adequate. The default is to just issue those errors as warnings. *--fail-on-broken-symlinks*:: Fail on broken symlinks. The default is to just issue those errors as warnings. *--no-upgrade-test*:: Skip testing upgrade from an existing version in the archive. *--no-install-purge-test*:: Skip the install and purge test. *-p, *--pbuilder*:: Use _/var/cache/pbuilder/base.tgz_ as the base tarball. This is a shorthand so that you don't need to use '-b' for it. *--pedantic-purge-test*:: Be pedantic when checking if a purged package leaves files behind. If this option is not set, files left in _/tmp_ are ignored.") *--proxy*='URL':: Use the proxy at URL to access the Debian mirror(s). Takes precedence over the 'http_proxy' environment variable. Using a local proxy is recommended because piuparts may use large amounts of bandwidth to repeatedly download the same files. *-s* 'filename', *--save*='filename':: Save the chroot, after it has been set up, as a tarball into _filename_. It can then be used with '-b'. *-B* 'FILE', *--end-meta*='FILE':: Load chroot package selection and file meta data from FILE. See the function install_and_upgrade_between_distros() in piuparts.py for defaults. Mostly useful for large scale distro upgrade tests. *-S* 'FILE', *--save-end-meta*='FILE':: Save chroot package selection and file meta data in FILE for later use. See the function install_and_upgrade_between_distros() in piuparts.py for defaults. Mostly useful for large scale distro upgrade tests. *--scriptsdir*='DIR':: Directory where are custom scripts are placed. By default, this is not set. For more information about this, read README_server.txt *--schroot*='SCHROOT-NAME':: Use schroot session named SCHROOT-NAME for the chroot, instead of building a new one with debootstrap. *--single-changes-list*:: When processing changes files, piuparts will process the packages in each individual changes file seperately. This option will set piuparts to scan the packages of all changes files together along with any individual package files that may have been given on the command line. *--single-packages*:: Process every package file or package name individually, thus piuparts process runs multiple times. This option can be useful with conflicting packages. *--shell-on-error*:: Start an interactive shell in the chroot after an error occurred. This should help debugging failures directly inside the piuparts test environment. The chroot cleanup will continue after the shell terminates. Note: This does not work if the piuparts command is prefixed with 'timeout', which is usually the case in command lines directly copied from logfiles from a master-slave setup. Removing the 'timeout' part is sufficient. *--skip-minimize*:: Allow skip minimize chroot step. This is useful when you want to test several packages with piuparts. You can prepare a tarball already minimized and skip this step in all the tests. This is the default now. *--minimize*:: Minimize the chroot with debfoster. This used to be the default until #539142 was fixed. *--skip-cronfiles-test*:: Skip testing the output from the cron files left in the system after remove a package. *--skip-logrotatefiles-test*:: Skip testing the output from the logrotate files left in the system after remove a package. *--testdebs-repo*='deb-line':: Provide an additional line to be appended to sources.list, e.g. 'deb [ trusted=yes ] <URL> <distrib> <components>' or 'deb [ trusted=yes ] file:// </bind/mount> ./' If only an URL or local path is given as argument, "deb [ trusted=yes ]", "file://", and "./" will be prepended/appended as needed. The "testdebs" repository provides the packages to be tested (and some additional dependencies, if needed, e.g. all packages built from the same source package as the (binary) package being tested) and can be used for testing complex installation and upgrade scenarios involving dependencies that are not yet in the archive. This repository will be available only for installing the target packages. Dependency resolution will be done by apt-get. The packages to be tested can be passed as .debs or as package names (with '--apt'). The "trusted=yes" option causes this (and only this) repository to be trustworthy even if the Packages file is not signed, such that a (globally acting) '--do-not-verify-signatures' will not be needed. *-t directory*, *--tmpdir*='directory':: Use directory as the place where temporary files and directories are created. The default is the environment variable *TMPDIR*, or _/tmp_ if not set. + Note: the temporary directory must *not* be mounted with the _nodev_ or _nosuid_ mount option. *-v*, *--verbose*:: This option no longer has any meaning, but it is still accepted for backwards compatibility. *-V*, *--version*:: Write out the version number of the program. *--warn-on-debsums-errors*:: Print a warning rather than failing if debsums reports modified files. *--warn-on-leftovers-after-purge*:: Print a warning rather than failing if files are left behind after purge. *--warn-on-others*:: Print a warning rather than failing if files are left behind, modified, or removed by a package that was not given on the command-line. + This way, you can basically isolate the purge test to your own packages. If a package that is brought in as a dependency doesn't purge cleanly, the test will not fail because of it (but a warning message will be printed). + Behavior with multiple packages given on the command-line could be problematic, particularly if the dependency tree of one package in the list includes another in the list. Therefore, it is recommended to use this option with one package at a time. EXAMPLES -------- Assume that you have just built a new version of your Debian package, to be uploaded to Debian unstable. It is in _../foo_1.0-2_i386.deb_ and you would like to know whether it installs and uninstalls properly. Here's what you would do: ---- piuparts ../foo_1.0-2_i386.deb ---- If the package exists in the Debian archive already, the above command also tests that it upgrades properly. To do the same test, but using a particular mirror, and only the main component, you would do this: ---- piuparts -m 'http://gytha/debian main' ../foo_1.0-2_i386.deb ---- If you want to do the same as above but for your changes files, pass in your changes files when running piuparts, and piuparts will process each package in the changes files as though you had passed all those packages on the command line to piuparts yourself. For example: ---- piuparts ../foo_1.0-2_i386.changes piuparts -m 'http://gytha/debian main' ../foo_1.0-2_i386.changes ---- If you want to test that a package installs properly in the stable (currently jessie) Debian release, then can be upgraded to the testing (currently stretch) and unstable (sid) versions, and then uninstalled without problems, you would give the following command: ---- piuparts -a -d jessie -d stretch -d sid foo ---- ENVIRONMENT ----------- *TMPDIR* Location for temporary files and directories. If not set, use _/tmp_. See also the '-t' ('--tmpdir') option. NOTES ----- Output of commands run by piuparts is limited to three megabytes. To change this limit, the source code needs to be edited. Commands exceeding this limit will be aborted. SEE ALSO -------- *pbuilder*(1), *debootstrap*(8) AUTHOR ------ Lars Wirzenius (liw@iki.fi) and others // vim: set filetype=asciidoc: ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piuparts.py��������������������������������������������������������������������������������0000644�0000000�0000000�00000403771�13061276553�012060� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # -*- coding: utf-8 -*- # # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright © 2007-2017 Holger Levsen (holger@layer-acht.org) # Copyright © 2010-2017 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """Debian package installation and uninstallation tester. This program sets up a minimal Debian system in a chroot, and installs and uninstalls packages and their dependencies therein, looking for problems. See the manual page (piuparts.1, generated from piuparts.1.txt) for more usage information. Lars Wirzenius <liw@iki.fi> """ VERSION = "__PIUPARTS_VERSION__" import distro_info import time import logging import optparse import sys import commands import tempfile import shutil import os import tarfile import stat import re import pickle import subprocess import urllib import uuid import apt_pkg import pipes from collections import namedtuple from signal import alarm, signal, SIGALRM, SIGTERM, SIGKILL try: from debian import deb822 except ImportError: from debian_bundle import deb822 import piupartslib.conf apt_pkg.init_system() DISTRO_CONFIG_FILE = "/etc/piuparts/distros.conf" class Defaults: """Default settings which depend on flavor of Debian. Some settings, such as the default mirror and distribution, depend on which flavor of Debian we run under: Debian itself, or a derived distribution such as Ubuntu. This class abstracts away the defaults so that the rest of the code can just refer to the values defined herein. """ def get_components(self): """Return list of default components for a mirror.""" def get_mirror(self): """Return default mirror.""" def get_distribution(self): """Return default distribution.""" def get_keyring(self): """Return default keyring.""" class DebianDefaults(Defaults): def get_components(self): return ["main", "contrib", "non-free"] def get_mirror(self): return [("http://deb.debian.org/debian", self.get_components())] def get_distribution(self): return [distro_info.DebianDistroInfo().devel()] def get_keyring(self): return "/usr/share/keyrings/debian-archive-keyring.gpg" class UbuntuDefaults(Defaults): def get_components(self): return ["main", "universe", "restricted", "multiverse"] def get_mirror(self): return [("http://archive.ubuntu.com/ubuntu", self.get_components())] def get_distribution(self): return [distro_info.UbuntuDistroInfo().devel()] def get_keyring(self): return "/usr/share/keyrings/ubuntu-archive-keyring.gpg" class DefaultsFactory: """Instantiate the right defaults class.""" def guess_flavor(self): p = subprocess.Popen(["lsb_release", "-i", "-s"], stdout=subprocess.PIPE) stdout, stderr = p.communicate() return stdout.strip().lower() def new_defaults(self): if not settings.defaults: settings.defaults = self.guess_flavor() if settings.defaults.lower() == "debian": return DebianDefaults() if settings.defaults.lower() == "ubuntu": return UbuntuDefaults() logging.error("Unknown set of defaults: %s" % settings.defaults) panic() class Settings: """Global settings for this program.""" def __init__(self): self.defaults = None self.tmpdir = None self.keep_tmpdir = False self.shell_on_error = False self.max_command_output_size = 8 * 1024 * 1024 # 8 MB (google-android-ndk-installer on install) (daptup on dist-upgrade) self.max_command_runtime = 60 * 60 # 60 minutes (texlive-full and blends metapackages on dist-upgrade) self.single_changes_list = False self.single_packages = False self.args_are_package_files = True # distro setup self.proxy = None self.debian_mirrors = [] self.extra_repos = [] self.testdebs_repo = None self.debian_distros = [] self.keep_sources_list = False self.keyring = None self.do_not_verify_signatures = False self.no_check_valid_until = False self.install_recommends = False self.install_suggests = False self.eatmydata = True self.dpkg_force_unsafe_io = True self.dpkg_force_confdef = False self.scriptsdirs = [] self.bindmounts = [] self.allow_database = False # chroot setup self.arch = None self.basetgz = None self.savetgz = None self.lvm_volume = None self.lvm_snapshot_size = "1G" self.adt_virt = None self.existing_chroot = None self.hard_link = False self.schroot = None self.end_meta = None self.save_end_meta = None self.skip_minimize = True self.minimize = False self.debfoster_options = None # tests and checks self.no_install_purge_test = False self.no_upgrade_test = False self.distupgrade_to_testdebs = False self.install_remove_install = False self.install_purge_install = False self.list_installed_files = False self.fake_essential_packages = [] self.extra_old_packages = [] self.skip_cronfiles_test = False self.skip_logrotatefiles_test = False self.adequate = True self.check_broken_diversions = True self.check_broken_symlinks = True self.warn_broken_symlinks = True self.warn_on_others = False self.warn_on_leftovers_after_purge = False self.warn_on_debsums_errors = False self.warn_on_install_over_symlink = False self.warn_if_inadequate = True self.pedantic_purge_test = False self.ignored_files = [ # /root/.rnd should *not* be listed here, see #750099 # piuparts state "/usr/sbin/policy-rc.d", # system state "/boot/grub/", "/etc/X11/", "/etc/X11/default-display-manager", "/etc/aliases", "/etc/aliases.db", "/etc/crypttab", "/etc/group", "/etc/group-", "/etc/group.org", "/etc/gshadow", "/etc/gshadow-", "/etc/hosts", "/etc/inetd.conf", "/etc/inittab", "/etc/ld.so.cache", "/etc/machine-id", "/etc/mailname", "/etc/mtab", "/etc/network/interfaces", "/etc/news/", "/etc/news/organization", "/etc/news/server", "/etc/news/servers", "/etc/news/whoami", "/etc/nologin", "/etc/passwd", "/etc/passwd-", "/etc/passwd.org", "/etc/shadow", "/etc/shadow-", "/etc/shadow.org", "/etc/subgid", "/etc/subgid-", "/etc/subuid", "/etc/subuid-", "/usr/share/info/dir", "/usr/share/info/dir.old", "/var/cache/ldconfig/aux-cache", "/var/crash/", "/var/games/", # package management "/etc/apt/apt.conf.d/01autoremove-kernels", "/etc/apt/secring.gpg", "/etc/apt/trustdb.gpg", "/etc/apt/trusted.gpg", "/etc/apt/trusted.gpg~", "/var/cache/apt/archives/lock", "/var/cache/apt/archives/partial/", "/var/cache/apt/pkgcache.bin", "/var/cache/apt/srcpkgcache.bin", "/var/cache/debconf/", "/var/cache/debconf/config.dat", "/var/cache/debconf/config.dat.old", "/var/cache/debconf/config.dat-old", "/var/cache/debconf/passwords.dat", "/var/cache/debconf/passwords.dat.old", "/var/cache/debconf/templates.dat", "/var/cache/debconf/templates.dat.old", "/var/cache/debconf/templates.dat-old", "/var/lib/apt/extended_states", "/var/lib/cdebconf/", "/var/lib/cdebconf/passwords.dat", "/var/lib/cdebconf/questions.dat", "/var/lib/cdebconf/questions.dat-old", "/var/lib/cdebconf/templates.dat", "/var/lib/cdebconf/templates.dat-old", "/var/lib/dpkg/arch", "/var/lib/dpkg/available", "/var/lib/dpkg/available-old", "/var/lib/dpkg/diversions", "/var/lib/dpkg/diversions-old", "/var/lib/dpkg/lock", "/var/lib/dpkg/status", "/var/lib/dpkg/status-old", "/var/lib/dpkg/statoverride", "/var/lib/dpkg/statoverride-old", "/var/log/alternatives.log", "/var/log/apt/eipp.log.xz", "/var/log/apt/history.log", "/var/log/apt/term.log", "/var/log/bootstrap.log", "/var/log/dbconfig-common/dbc.log", "/var/log/dpkg.log", # system logfiles "/var/log/auth.log", "/var/log/btmp", "/var/log/daemon.log", "/var/log/debug", "/var/log/faillog", "/var/log/kern.log", "/var/log/lastlog", "/var/log/lpr.log", "/var/log/mail.err", "/var/log/mail.info", "/var/log/mail.log", "/var/log/mail.warn", "/var/log/messages", "/var/log/news/", "/var/log/news/news.crit", "/var/log/news/news.err", "/var/log/news/news.notice", "/var/log/secure", "/var/log/syslog", "/var/log/user.log", # application logfiles # actually, only modification should be permitted here, but not creation/removal "/var/log/fontconfig.log", # home directories of system accounts "/var/lib/gozerbot/", "/var/lib/nagios/", # nagios* (#668756) "/var/lib/onioncat/", # onioncat "/var/lib/pkcs11proxyd/", # caml-crush-server (#810703) "/var/lib/rbldns/", "/var/spool/powerdns/", # pdns-server (#531134), pdns-recursor (#531135) # work around broken symlinks "/etc/modules-load.d/modules.conf", # -> ../modules (target obsoleted by modules-load.d) "/etc/sysctl.d/99-sysctl.conf", # -> ../sysctl.conf (target obsoleted by sysctl.d) "/usr/lib/python2.6/dist-packages/python-support.pth", # 635493 and #385775 "/usr/lib/python2.7/dist-packages/python-support.pth", "/usr/share/texmf/ls-R", # -> /var/lib/texmf/ls-R-TEXMFMAIN (link owned by tex-common, target created with mktexlsr from texlive-binaries) # work around #316521 dpkg: incomplete cleanup of empty directories "/etc/apache2/", "/etc/apache2/conf.d/", "/etc/clamav/", "/etc/cron.d/", "/etc/lighttpd/", "/etc/lighttpd/conf-available/", "/etc/modprobe.d/", "/etc/nagios-plugins/config/", "/etc/network/", "/etc/php/", "/etc/php/7.0/", "/etc/php/7.0/apache2/", "/etc/php/7.0/apache2/conf.d/", "/etc/php/7.0/cli/", "/etc/php/7.0/cli/conf.d/", "/etc/php/7.0/mods-available/", "/etc/php/7.0/phpdbg/", "/etc/php/7.0/phpdbg/conf.d/", "/etc/php5/", "/etc/php5/conf.d/", "/etc/php5/mods-available/", "/etc/sgml/", "/etc/ssl/", "/etc/ssl/certs/", "/etc/ssl/private/", "/etc/xml/", "/usr/share/dh-python/", "/usr/share/dh-python/dhpython/", "/usr/share/dh-python/dhpython/build/", "/usr/share/python3/", "/usr/share/python3/debpython/", "/var/lib/apache2/", # HACKS ] self.ignored_patterns = [ # system state "/dev/.*", "/etc/init.d/\.depend.*", "/run/.*", "/var/backups/.*", "/var/cache/man/.*", "/var/mail/.*", "/var/run/.*", # package management "/etc/apt/trusted.gpg.d/.*.gpg~", "/var/lib/apt/lists/.*", "/var/lib/dpkg/alternatives/.*", "/var/lib/dpkg/triggers/.*", "/var/lib/insserv/run.*.log", "/var/lib/ucf/.*", "/var/lib/update-rc.d/.*", # application data "/srv/.*", # 848186 "/var/lib/citadel/(data/.*)?", "/var/lib/mercurial-server/.*", "/var/lib/onak/.*", "/var/lib/openvswitch/(pki/.*)?", "/var/lib/vmm/(./.*)?", # 682184 "/var/log/exim/.*", "/var/log/exim4/.*", "/var/spool/exim/.*", "/var/spool/exim4/.*", "/var/spool/news/.*", "/var/spool/squid/(../.*)?", "/var/www/.*", # HACKS "/lib/modules/.*/modules.*", ] self.non_pedantic_ignore_patterns = [ "/tmp/.*" ] settings = Settings() on_panic_hooks = {} counter = 0 def do_on_panic(hook): global counter cid = counter counter += 1 on_panic_hooks[cid] = hook return cid def dont_do_on_panic(id): del on_panic_hooks[id] class TimeOffsetFormatter(logging.Formatter): def __init__(self, fmt=None, datefmt=None): self.startup_time = time.time() logging.Formatter.__init__(self, fmt, datefmt) def formatTime(self, record, datefmt): t = time.time() - self.startup_time t_min = int(t / 60) t_sec = t % 60.0 return "%dm%.1fs" % (t_min, t_sec) DUMP = logging.DEBUG - 1 HANDLERS = [] def setup_logging(log_level, log_file_name): logging.addLevelName(DUMP, "DUMP") logger = logging.getLogger() logger.setLevel(log_level) formatter = TimeOffsetFormatter("%(asctime)s %(levelname)s: %(message)s") handler = logging.StreamHandler(sys.stdout) handler.setFormatter(formatter) logger.addHandler(handler) HANDLERS.append(handler) if log_file_name: handler = logging.FileHandler(log_file_name) handler.setFormatter(formatter) logger.addHandler(handler) HANDLERS.append(handler) def dump(msg): logger = logging.getLogger() logger.log(DUMP, msg) for handler in HANDLERS: handler.flush() def panic(exit=1): for i in reversed(range(counter)): if i in on_panic_hooks: on_panic_hooks[i]() logging.error("piuparts run ends.") sys.exit(exit) def indent_string(str): """Indent all lines in a string with two spaces and return result.""" return "\n".join([" " + line for line in str.split("\n")]) def quote_spaces(vlist): return [pipes.quote(x) for x in vlist] def unqualify(packages): if packages: return [p.split("=", 1)[0].strip() for p in packages] return packages class Alarm(Exception): pass def alarm_handler(signum, frame): raise Alarm def run(command, ignore_errors=False, timeout=0): """Run an external command and die with error message if it fails.""" def kill_subprocess(p, reason): logging.error("Terminating command due to %s" % reason) p.terminate() for i in range(10): time.sleep(0.5) if p.poll() is not None: break else: logging.error("Killing command due to %s" % reason) p.kill() p.wait() assert isinstance(command, type([])) logging.debug("Starting command: %s" % command) env = os.environ.copy() for var in ["LANG", "LANGUAGE", "LC_CTYPE", "LC_NUMERIC", "LC_TIME", "LC_COLLATE", "LC_MONETARY", "LC_MESSAGES", "LC_PAPER", "LC_NAME", "LC_ADDRESS", "LC_TELEPHONE", "LC_MEASUREMENT", "LC_IDENTIFICATION", "LC_ALL"]: if var in env: del env[var] env["PIUPARTS_OBJECTS"] = ' '.join(str(vobject) for vobject in settings.testobjects) devnull = open('/dev/null', 'r') p = subprocess.Popen(command, env=env, stdin=devnull, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) output = "" excessive_output = False if timeout > 0: signal(SIGALRM, alarm_handler) alarm(timeout) try: while p.poll() is None: """Read 64 KB chunks, but depending on the output buffering behavior of the command we may get less even if more output is coming later. Abort after reading max_command_output_size bytes.""" output += p.stdout.read(1 << 16) if (len(output) > settings.max_command_output_size): excessive_output = True ignore_errors = False alarm(0) kill_subprocess(p, "excessive output") output += "\n\n***** Command was terminated after exceeding output limit (%.2f MB) *****\n" \ % (settings.max_command_output_size / 1024. / 1024.) break if not excessive_output: output += p.stdout.read(settings.max_command_output_size) alarm(0) except Alarm: ignore_errors = False kill_subprocess(p, "excessive runtime") output += "\n\n***** Command was terminated after exceeding runtime limit (%s s) *****\n" % timeout devnull.close() if output: dump("\n" + indent_string(output.rstrip("\n"))) if p.returncode == 0: logging.debug("Command ok: %s" % repr(command)) elif ignore_errors: logging.debug("Command failed (status=%d), but ignoring error: %s" % (p.returncode, repr(command))) else: logging.error("Command failed (status=%d): %s\n%s" % (p.returncode, repr(command), indent_string(output))) panic() return p.returncode, output def create_temp_file(): """Create a temporary file and return its full path.""" (fd, path) = tempfile.mkstemp(dir=settings.tmpdir) logging.debug("Created temporary file %s" % path) return (fd, path) def create_file(name, contents): """Create a new file with the desired name and contents.""" try: f = file(name, "w") f.write(contents) f.close() except IOError as detail: logging.error("Couldn't create file %s: %s" % (name, detail)) panic() def remove_files(filenames): """Remove some files.""" for filename in filenames: logging.debug("Removing %s" % filename) try: os.remove(filename) except OSError as detail: logging.error("Couldn't remove %s: %s" % (filename, detail)) panic() def make_metapackage(name, depends, conflicts, arch='all'): """Return the path to a .deb created just for satisfying dependencies Caller is responsible for removing the temporary directory containing the .deb when finished. """ # Inspired by pbuilder's pbuilder-satisfydepends-aptitude tmpdir = tempfile.mkdtemp(dir=settings.tmpdir) panic_handler_id = do_on_panic(lambda: shutil.rmtree(tmpdir)) create_file(os.path.join(tmpdir, ".piuparts.tmpdir"), "metapackage creation") old_umask = os.umask(0) os.makedirs(os.path.join(tmpdir, name, 'DEBIAN'), mode=0o755) os.umask(old_umask) control = deb822.Deb822() control['Package'] = name control['Version'] = '0.invalid.0' control['Architecture'] = arch control['Maintainer'] = ('piuparts developers team ' '<piuparts-devel@lists.alioth.debian.org>') control['Description'] = ('Dummy package to satisfy dependencies - ' 'created by piuparts\n' ' This package was created automatically by ' 'piuparts and can safely be removed') if depends: control['Depends'] = depends if conflicts: control['Conflicts'] = conflicts create_file(os.path.join(tmpdir, name, 'DEBIAN', 'control'), control.dump()) logging.debug("metapackage:\n" + indent_string(control.dump())) run(['dpkg-deb', '-b', '-Zgzip', '--nocheck', os.path.join(tmpdir, name)]) dont_do_on_panic(panic_handler_id) return os.path.join(tmpdir, name + '.deb') def split_path(pathname): parts = [] while pathname: (head, tail) = os.path.split(pathname) # print "split '%s' => '%s' + '%s'" % (pathname, head, tail) if tail: parts.append(tail) elif not head: break elif head == pathname: parts.append(head) break pathname = head return parts def canonicalize_path(root, pathname, report_links=False): """Canonicalize a path name, simulating chroot at 'root'. When resolving the symlink, pretend (similar to chroot) that 'root' is the root of the filesystem. Also resolve '..' and '.' components. This should not escape the chroot below 'root', but for security concerns, use chroot and have the kernel resolve symlinks instead. Returns the final canonical path or a list of (path, target) tuples, one for each symlink encountered. """ # print "\nCANONICALIZE %s %s" % (root, pathname) links = [] seen = [] parts = split_path(pathname) # print "PARTS ", list(reversed(parts)) path = "/" while parts: tag = "\n".join(parts + [path]) # print "TEST '%s' + " % path, list(reversed(parts)) if tag in seen or len(seen) > 1024: fullpath = os.path.join(path, *reversed(parts)) # print "LOOP %s" % fullpath path = fullpath logging.error("ELOOP: Too many symbolic links in '%s'" % path) break seen.append(tag) part = parts.pop() # Using normpath() to cleanup '.', '..' and multiple slashes. # Removing a suffix 'foo/..' is safe here since it can't change the # meaning of 'path' because it contains no symlinks - they have been # resolved already. newpath = os.path.normpath(os.path.join(path, part)) rootedpath = os.path.join(root, newpath[1:]) if newpath == "/": path = "/" elif os.path.islink(rootedpath): target = os.readlink(rootedpath) # print "LINK to '%s'" % target links.append((newpath, target)) if os.path.isabs(target): path = "/" parts.extend(split_path(target)) else: path = newpath # print "FINAL '%s'" % path if report_links: return links return path def is_broken_symlink(root, dirpath, filename): """Is symlink dirpath+filename broken?""" if dirpath[:len(root)] == root: dirpath = dirpath[len(root):] pathname = canonicalize_path(root, os.path.join(dirpath, filename)) pathname = os.path.join(root, pathname[1:]) # The symlink chain, if any, has now been resolved. Does the target # exist? # print "EXISTS ", pathname, os.path.exists(pathname) return not os.path.exists(pathname) FileInfo = namedtuple('FileInfo', ['st', 'target', 'user', 'group']) class Chroot: """A chroot for testing things in.""" def __init__(self): self.name = None self.bootstrapped = False self.mounts = [] def create_temp_dir(self): """Create a temporary directory for the chroot.""" self.name = tempfile.mkdtemp(dir=settings.tmpdir) create_file(os.path.join(self.name, ".piuparts.tmpdir"), "chroot") os.chmod(self.name, 0o755) logging.debug("Created temporary directory %s" % self.name) def create(self, temp_tgz=None): """Create a chroot according to user's wishes.""" self.panic_handler_id = do_on_panic(self.remove) if not settings.schroot: self.create_temp_dir() if temp_tgz: self.unpack_from_tgz(temp_tgz) elif settings.basetgz: self.unpack_from_tgz(settings.basetgz) elif settings.lvm_volume: self.setup_from_lvm(settings.lvm_volume) elif settings.existing_chroot: self.setup_from_dir(settings.existing_chroot) elif settings.schroot: self.setup_from_schroot(settings.schroot) else: self.setup_minimal_chroot() if not settings.schroot: self.mount_proc() self.configure_chroot() # Copy scripts dirs into the chroot, merging all dirs together, # later files overwriting earlier ones. if settings.scriptsdirs: self.mkdir_p("tmp/scripts/") dest = self.relative("tmp/scripts/") for sdir in settings.scriptsdirs: logging.debug("Copying scriptsdir %s to %s" % (sdir, dest)) for sfile in os.listdir(sdir): if (sfile.startswith("post_") or sfile.startswith("pre_") or sfile.startswith("is_testable_")) \ and not ".dpkg-" in sfile \ and os.path.isfile(os.path.join(sdir, sfile)): shutil.copy(os.path.join(sdir, sfile), dest) # Run custom scripts after chroot has been unpacked/debootstrapped # Useful for adjusting apt configuration e.g. for internal mirror usage self.run_scripts("post_chroot_unpack") if settings.basetgz or settings.schroot or settings.existing_chroot: self.run(["apt-get", "-yf", "dist-upgrade"]) self.minimize() # Run custom scripts after creating the chroot. self.run_scripts("post_setup") self.install_packages_by_name(settings.fake_essential_packages) if settings.savetgz and not temp_tgz: self.pack_into_tgz(settings.savetgz) def remove(self): """Remove a chroot and all its contents.""" if not settings.keep_tmpdir and os.path.exists(self.name): self.terminate_running_processes() self.unmount_all() if settings.lvm_volume: logging.debug('Unmounting and removing LVM snapshot %s' % self.lvm_snapshot_name) run(['umount', self.name]) run(['lvremove', '-f', self.lvm_snapshot]) if settings.schroot: logging.debug("Terminate schroot session '%s'" % self.name) run(['schroot', '--end-session', '--chroot', "session:" + self.schroot_session]) if not settings.schroot: run(['rm', '-rf', '--one-file-system', self.name]) if os.path.exists(self.name): create_file(os.path.join(self.name, ".piuparts.tmpdir"), "removal failed") logging.debug("Removed directory tree at %s" % self.name) elif settings.keep_tmpdir: if settings.schroot: logging.debug("Keeping schroot session %s at %s" % (self.schroot_session, self.name)) else: logging.debug("Keeping directory tree at %s" % self.name) dont_do_on_panic(self.panic_handler_id) def was_bootstrapped(self): return self.bootstrapped def create_temp_tgz_file(self): """Return the path to a file to be used as a temporary tgz file""" # Yes, create_temp_file() would work just as well, but putting it in # the interface for Chroot allows the VirtServ hack to work. (fd, temp_tgz) = create_temp_file() os.close(fd) return temp_tgz def remove_temp_tgz_file(self, temp_tgz): """Remove the file that was used as a temporary tgz file""" # Yes, remove_files() would work just as well, but putting it in # the interface for Chroot allows the VirtServ hack to work. remove_files([temp_tgz]) def pack_into_tgz(self, result): """Tar and compress all files in the chroot.""" self.run(["apt-get", "clean"]) logging.debug("Saving %s to %s." % (self.name, result)) (fd, tmpfile) = tempfile.mkstemp(dir=os.path.dirname(result)) os.close(fd) cleanup_tmpfile = lambda: os.remove(tmpfile) panic_handler_id = do_on_panic(cleanup_tmpfile) run(['tar', '-czf', tmpfile, '--one-file-system', '--exclude', 'tmp/scripts', '-C', self.name, './']) os.chmod(tmpfile, 0o644) os.rename(tmpfile, result) dont_do_on_panic(panic_handler_id) def unpack_from_tgz(self, tarball): """Unpack a tarball to a chroot.""" logging.debug("Unpacking %s into %s" % (tarball, self.name)) prefix = [] if settings.eatmydata and os.path.isfile('/usr/bin/eatmydata'): prefix.append('eatmydata') run(prefix + ["tar", "-C", self.name, "-zxf", tarball]) def setup_from_schroot(self, schroot): self.schroot_session = schroot.split(":", 1)[-1] + "-" + str(uuid.uuid1()) + "-piuparts" run(['schroot', '--begin-session', '--chroot', schroot, '--session-name', self.schroot_session]) ret_code, output = run(['schroot', '--chroot', "session:" + self.schroot_session, '--location']) self.name = output.strip() logging.info("New schroot session in '%s'" % self.name) def setup_from_lvm(self, lvm_volume): """Create a chroot by creating an LVM snapshot.""" self.lvm_base = os.path.dirname(lvm_volume) self.lvm_vol_name = os.path.basename(lvm_volume) self.lvm_snapshot_name = self.lvm_vol_name + "-" + str(uuid.uuid1()) self.lvm_snapshot = os.path.join(self.lvm_base, self.lvm_snapshot_name) logging.debug("Creating LVM snapshot %s from %s" % (self.lvm_snapshot, lvm_volume)) run(['lvcreate', '-n', self.lvm_snapshot, '-s', lvm_volume, '-L', settings.lvm_snapshot_size]) logging.info("Mounting LVM snapshot to %s" % self.name) run(['mount', self.lvm_snapshot, self.name]) def setup_from_dir(self, dirname): """Create chroot from an existing one.""" # if on same device, make hard link cmd = ["cp"] if settings.hard_link and os.stat(dirname).st_dev == os.stat(self.name).st_dev: cmd += ["-al"] logging.debug("Hard linking %s to %s" % (dirname, self.name)) else: cmd += ["-ax"] logging.debug("Copying %s into %s" % (dirname, self.name)) for name in os.listdir(dirname): src = os.path.join(dirname, name) dst = os.path.join(self.name, name) run(cmd + [src, dst]) def interactive_shell(self): logging.info("Entering interactive shell in %s" % self.name) env = os.environ.copy() env['debian_chroot'] = "piuparts:%s" % self.name try: subprocess.call(['chroot', self.name, 'bash', '-l'], env=env) except: pass def run(self, command, ignore_errors=False): prefix = [] if settings.eatmydata and os.path.isfile(os.path.join(self.name, 'usr/bin/eatmydata')): prefix.append('eatmydata') if settings.schroot: return run( ["schroot", "--preserve-environment", "--run-session", "--chroot", "session:" + self.schroot_session, "--directory", "/", "-u", "root", "--"] + prefix + command, ignore_errors=ignore_errors, timeout=settings.max_command_runtime) else: return run(["chroot", self.name] + prefix + command, ignore_errors=ignore_errors, timeout=settings.max_command_runtime) def mkdir_p(self, path): fullpath = self.relative(path) if not os.path.isdir(fullpath): os.makedirs(fullpath) def create_apt_sources(self, distro): """Create an /etc/apt/sources.list with a given distro.""" lines = [] lines.extend(settings.distro_config.get_deb_lines( distro, settings.debian_mirrors[0][1])) for mirror, components in settings.debian_mirrors[1:]: lines.append("deb %s %s %s" % (mirror, distro, " ".join(components))) for repo in settings.extra_repos: lines.append(repo) create_file(self.relative("etc/apt/sources.list"), "\n".join(lines) + "\n") logging.debug("sources.list:\n" + indent_string("\n".join(lines))) def enable_testdebs_repo(self, update=True): if settings.testdebs_repo: if settings.testdebs_repo.startswith("deb"): debline = settings.testdebs_repo elif settings.testdebs_repo.startswith("/"): debline = "deb [ trusted=yes ] file://%s ./" % settings.testdebs_repo else: debline = "deb [ trusted=yes ] %s ./" % settings.testdebs_repo logging.debug("enabling testdebs repository '%s'" % debline) create_file(self.relative("etc/apt/sources.list.d/piuparts-testdebs-repo.list"), debline + "\n") if update: self.run(["apt-get", "update"]) def disable_testdebs_repo(self): if settings.testdebs_repo: logging.debug("disabling testdebs repository") remove_files([self.relative("etc/apt/sources.list.d/piuparts-testdebs-repo.list")]) def create_apt_conf(self): """Create /etc/apt/apt.conf.d/piuparts inside the chroot.""" lines = ['APT::Get::Assume-Yes "yes";\n'] lines.append('APT::Install-Recommends "%d";\n' % int(settings.install_recommends)) lines.append('APT::Install-Suggests "%d";\n' % int(settings.install_suggests)) lines.append('APT::Get::AllowUnauthenticated "%s";\n' % settings.apt_unauthenticated) lines.append('Acquire::PDiffs "false";\n') if settings.no_check_valid_until: lines.append('Acquire::Check-Valid-Until "false";\n') if settings.proxy: proxy = settings.proxy elif "http_proxy" in os.environ: proxy = os.environ["http_proxy"] else: proxy = None pat = re.compile(r"^Acquire::http::Proxy\s+\"([^\"]+)\"", re.I) p = subprocess.Popen(["apt-config", "dump"], stdout=subprocess.PIPE) stdout, _ = p.communicate() if stdout: for line in stdout.split("\n"): m = re.match(pat, line) if proxy is None and m: proxy = m.group(1) if proxy: lines.append('Acquire::http::Proxy "%s";\n' % proxy) if settings.dpkg_force_unsafe_io: lines.append('Dpkg::Options {"--force-unsafe-io";};\n') if settings.dpkg_force_confdef: lines.append('Dpkg::Options {"--force-confdef";};\n') create_file(self.relative("etc/apt/apt.conf.d/piuparts"), "".join(lines)) def create_dpkg_conf(self): """Create /etc/dpkg/dpkg.cfg.d/piuparts inside the chroot.""" lines = [] if settings.dpkg_force_unsafe_io: lines.append('force-unsafe-io\n') if settings.dpkg_force_confdef: lines.append('force-confdef\n') logging.info("Warning: dpkg has been configured to use the force-confdef option. This will hide problems, see #466118.") if lines: self.mkdir_p("etc/dpkg/dpkg.cfg.d") create_file(self.relative("etc/dpkg/dpkg.cfg.d/piuparts"), "".join(lines)) def create_policy_rc_d(self): """Create a policy-rc.d that prevents daemons from running.""" full_name = self.relative("usr/sbin/policy-rc.d") policy = "#!/bin/sh\n" if settings.allow_database: policy += 'test "$1" = "mysql" && exit 0\n' policy += 'test "$1" = "postgresql" && exit 0\n' policy += 'test "$1" = "postgresql-8.3" && exit 0\n' policy += 'test "$1" = "firebird2.5-super" && exit 0\n' policy += "exit 101\n" create_file(full_name, policy) os.chmod(full_name, 0o755) logging.debug("Created policy-rc.d and chmodded it.") def create_resolv_conf(self): """Update resolv.conf based on the current configuration in the host system. Strip comments and whitespace.""" full_name = self.relative("etc/resolv.conf") resolvconf = "" with open("/etc/resolv.conf", "r") as f: for line in f: if line.strip() and not line.startswith(('#', ';')): resolvconf += line.strip() + '\n' create_file(full_name, resolvconf) os.chmod(full_name, 0o644) logging.debug("Created resolv.conf.") def setup_minimal_chroot(self): """Set up a minimal Debian system in a chroot.""" logging.debug("Setting up minimal chroot for %s at %s." % (settings.debian_distros[0], self.name)) prefix = [] if settings.eatmydata and os.path.isfile('/usr/bin/eatmydata'): prefix.append('eatmydata') options = [] if settings.do_not_verify_signatures: logging.info("Warning: not using --keyring option when running debootstrap!") else: options.append("--keyring=%s" % settings.keyring) if settings.eatmydata: options.append('--include=eatmydata') options.append('--components=%s' % ','.join(settings.debian_mirrors[0][1])) if settings.arch: options.append('--arch=%s' % settings.arch) run(prefix + ["debootstrap", "--variant=minbase"] + options + [settings.debian_distros[0], self.name, settings.distro_config.get_mirror(settings.debian_distros[0])]) self.bootstrapped = True def minimize(self): """Minimize a chroot by removing (almost all) unnecessary packages""" if settings.skip_minimize or not settings.minimize: return self.run(["apt-get", "install", "debfoster"]) debfoster_command = ["debfoster"] + settings.debfoster_options if settings.eatmydata: debfoster_command.append("eatmydata") self.run(debfoster_command) remove_files([self.relative("var/lib/debfoster/keepers")]) self.run(["dpkg", "--purge", "debfoster"]) def configure_chroot(self): """Configure a chroot according to current settings""" os.environ["PIUPARTS_DISTRIBUTION"] = settings.distro_config.get_distribution(settings.debian_distros[0]) if not settings.keep_sources_list: self.create_apt_sources(settings.debian_distros[0]) self.create_apt_conf() self.create_dpkg_conf() self.create_policy_rc_d() self.create_resolv_conf() for bindmount in settings.bindmounts: self.mount(bindmount, bindmount, opts="bind") self.run(["apt-get", "update"]) def upgrade_to_distros(self, distros, packages): """Upgrade a chroot installation to each successive distro.""" for distro in distros: logging.debug("Upgrading %s to %s" % (self.name, distro)) os.environ["PIUPARTS_DISTRIBUTION_NEXT"] = settings.distro_config.get_distribution(distro) self.create_apt_sources(distro) # Run custom scripts before upgrade self.run_scripts("pre_distupgrade") self.run(["apt-get", "update"]) self.run(["apt-get", "-yf", "dist-upgrade"]) os.environ["PIUPARTS_DISTRIBUTION_PREV"] = os.environ["PIUPARTS_DISTRIBUTION"] os.environ["PIUPARTS_DISTRIBUTION"] = settings.distro_config.get_distribution(distro) # Sometimes dist-upgrade won't upgrade the packages we want # to test because the new version depends on a newer library, # and installing that would require removing the old version # of the library, and we've told apt-get not to remove # packages. So, we force the installation like this. if packages: known_packages = self.get_known_packages(packages + settings.extra_old_packages) self.install_packages_by_name(known_packages) # Run custom scripts after upgrade self.run_scripts("post_distupgrade") self.check_for_no_processes() def get_known_packages(self, packages): """Does apt-get (or apt-cache) know about a set of packages?""" known_packages = [] new_packages = [] for name in packages: (status, output) = self.run(["apt-cache", "show", name], ignore_errors=True) # apt-cache reports status for some virtual packages and packages # in status config-files-remaining state without installation # candidate -- but only real packages have Filename/MD5sum/SHA* if status != 0 or re.search(r'^(Filename|MD5sum|SHA1|SHA256):', output, re.M) is None: new_packages.append(name) else: known_packages.append(name) if not known_packages: logging.info("apt-cache does not know about any of the requested packages") else: logging.info("apt-cache knows about the following packages: " + ", ".join(known_packages)) if new_packages: logging.info("the following packages are not in the archive: " + ", ".join(new_packages)) return known_packages def copy_files(self, source_names, target_name): """Copy files in 'source_name' to file/dir 'target_name', relative to the root of the chroot.""" target_name = self.relative(target_name) logging.debug("Copying %s to %s" % (", ".join(source_names), target_name)) for source_name in source_names: try: shutil.copy(source_name, target_name) except IOError as detail: logging.error("Error copying %s to %s: %s" % (source_name, target_name, detail)) panic() def list_installed_files(self, pre_info, post_info): """List the new files installed, removed and modified between two dir trees. Actually, it is a nice output of the funcion diff_meta_dat.""" (new, removed, modified) = diff_meta_data(pre_info, post_info) file_owners = self.get_files_owned_by_packages() if new: logging.debug("New installed files on system:\n" + file_list(new, file_owners)) else: logging.debug("The package did not install any new file.\n") if removed: logging.debug("The following files have disappeared:\n" + file_list(removed, file_owners)) if modified: logging.debug("The following files have been modified:\n" + file_list(modified, file_owners)) else: logging.debug("The package did not modify any file.\n") def is_installed(self, packages): if not packages: return True retcode, output = self.run(["dpkg-query", "-f", "${Package} ${Status}\n", "-W"] + packages, ignore_errors=True) if retcode != 0: return False installed = True for line in output.splitlines(): pkg, desired, whatever, status = line.split() if status != 'installed': logging.error("Installation of %s failed", pkg) installed = False return installed def install_packages(self, package_files, packages, with_scripts=True, reinstall=False): if package_files: self.install_package_files(package_files, packages, with_scripts=with_scripts) else: self.install_packages_by_name(packages, with_scripts=with_scripts, reinstall=reinstall) def install_package_files(self, package_files, packages=None, with_scripts=False): if packages and settings.testdebs_repo: self.install_packages_by_name(packages, with_scripts=with_scripts) return if package_files: # Check whether apt-get can install debs (supported since apt 1.1) # # If it can, this is preferable to the traditional # `dpkg -i foo.deb && apt-get -yf install` # approach since 'apt-get -yf install' can 'resolve' dependency # problems by removing the package we are trying to install (status, output) = self.run(["dpkg-query", "-f", "${Version}\n", "-W", "apt"], ignore_errors=True) apt_can_install_debs = apt_pkg.version_compare(output.strip(), "1.1") >= 0 # This must look like a local path so that apt-get can # distinguish it from a 'package/suite' request. self.copy_files(package_files, "tmp") tmp_files = [os.path.join("./tmp", os.path.basename(a)) for a in package_files] if with_scripts: self.run_scripts("pre_install") if apt_can_install_debs: # --allow-downgrades is also required in order to permit # installing a deb with the same version as that already # installed apt_get_install = ["apt-get", "-y", "--allow-downgrades"] else: apt_get_install = ["apt-get", "-yf"] apt_get_install.extend(settings.distro_config.get_target_flags( os.environ["PIUPARTS_DISTRIBUTION"])) apt_get_install.append("install") if settings.list_installed_files: pre_info = self.save_meta_data() if apt_can_install_debs: self.run(apt_get_install + tmp_files) else: (ret, out) = self.run(["dpkg", "-i"] + tmp_files, ignore_errors=True) if ret != 0: if "dependency problems - leaving unconfigured" in out: pass else: logging.error("Installation failed") panic() if settings.list_installed_files: self.list_installed_files(pre_info, self.save_meta_data()) self.run(apt_get_install) if settings.list_installed_files: self.list_installed_files(pre_info, self.save_meta_data()) if not self.is_installed(unqualify(packages)): logging.error("Could not install %s.", " ".join(unqualify(packages))) panic() logging.info("Installation of %s ok", tmp_files) if with_scripts: self.run_scripts("post_install") remove_files([self.relative(name) for name in tmp_files]) def install_packages_by_name(self, packages, with_scripts=True, reinstall=False): if packages: if with_scripts: self.run_scripts("pre_install") self.run(["apt-cache", "policy"]) self.run(["apt-cache", "policy"] + unqualify(packages)) if settings.list_installed_files: pre_info = self.save_meta_data() target_flags = settings.distro_config.get_target_flags(os.environ["PIUPARTS_DISTRIBUTION"]) self.apt_get_install(to_install=packages, flags=target_flags, reinstall=reinstall) if settings.list_installed_files: self.list_installed_files(pre_info, self.save_meta_data()) if with_scripts: self.run_scripts("post_install") def apt_get_install(self, to_install=[], to_remove=[], to_purge=[], flags=[], reinstall=False): command = ["apt-get", "-y"] + flags + ["install"] if reinstall: command.append("--reinstall") command.extend(to_install) command.extend(["%s-" % x for x in unqualify(to_remove)]) command.extend(["%s_" % x for x in unqualify(to_purge)]) self.run(command) def get_selections(self): """Get current package selections in a chroot.""" # "${Status}" emits three columns, e.g. "install ok installed" # "${binary:Package}" requires a multi-arch dpkg, so fall back to "${Package}" on older versions (status, output) = self.run(["dpkg-query", "-W", "-f", "${Status}\\t${binary:Package}\\t${Package}\\t${Version}\\n"]) vdict = {} for line in [line for line in output.split("\n") if line.strip()]: token = line.split() status = token[0] name = token[3] if status == "install": version = token[-1] else: version = None vdict[name] = (status, version) return vdict def get_diversions(self): """Get current dpkg-divert --list in a chroot.""" if not settings.check_broken_diversions: return (status, output) = self.run(["dpkg-divert", "--list"]) return output.split("\n") def get_modified_diversions(self, pre_install_diversions, post_install_diversions=None): """Check that diversions in chroot are identical (though potentially reordered).""" if post_install_diversions is None: post_install_diversions = self.get_diversions() removed = [ln for ln in pre_install_diversions if not ln in post_install_diversions] added = [ln for ln in post_install_diversions if not ln in pre_install_diversions] return (removed, added) def check_debsums(self): (status, output) = run(["debsums", "--root", self.name, "-ac"], ignore_errors=True) if status != 0: logging.error("FAIL: debsums reports modifications inside the chroot:\n%s" % indent_string(output.replace(self.name, ""))) if not settings.warn_on_debsums_errors: panic() def check_adequate(self, packages): """Run adequate and categorize output according to our needs. """ packages = unqualify([p for p in packages if not p.endswith("=None")]) if packages and settings.adequate and os.path.isfile('/usr/bin/adequate'): (status, output) = run(["dpkg-query", "-f", "${Version}\n", "-W", "adequate"], ignore_errors=True) logging.info("Running adequate version %s now." % output.strip()) adequate_tags = [ 'bin-or-sbin-binary-requires-usr-lib-library', 'broken-binfmt-detector', 'broken-binfmt-interpreter', 'incompatible-licenses', 'ldd-failure', 'library-not-found', 'missing-alternative', 'missing-copyright-file', 'missing-pkgconfig-dependency', 'missing-symbol-version-information', 'program-name-collision', 'py-file-not-bytecompiled', 'pyshared-file-not-bytecompiled', 'symbol-size-mismatch', 'undefined-symbol', ] boring_tags = [ 'obsolete-conffile', 'broken-symlink', ] ignored_tags = [] (status, output) = run(["adequate", "--root", self.name] + packages, ignore_errors=True) for tag in ignored_tags: # ignore some tags _regex = '^[^:]+: ' + tag + ' .*\n' output = re.compile(_regex, re.MULTILINE).sub('', output) if output: inadequate_results = '' boring_results = '' for tag in adequate_tags: if ' ' + tag + ' ' in output: inadequate_results += ' ' + tag + ' ' for tag in boring_tags: if ' ' + tag + ' ' in output: boring_results += ' ' + tag + ' ' if settings.warn_if_inadequate: error_code = 'WARN' else: error_code = 'FAIL' logging.error("%s: Inadequate results from running adequate!\n%s" % (error_code, indent_string(output.replace(self.name, "")))) if inadequate_results: logging.error("%s: Running adequate resulted in inadequate tags found: %s" % (error_code, inadequate_results)) if boring_results: logging.error("%s: Running adequate resulted in less interesting tags found: %s" % (error_code, boring_results)) if not boring_results and not inadequate_results: logging.error("%s: Found unknown tags running adequate." % error_code) if status != 0: logging.error("%s: Exit code from adequate was %s!" % (error_code, status)) if not settings.warn_if_inadequate: panic() def list_paths_with_symlinks(self): file_owners = self.get_files_owned_by_packages() bad = [] overwrites = False for f in sorted(file_owners.keys()): dn, fn = os.path.split(f) dc = canonicalize_path(self.name, dn) if dn != dc: fc = os.path.join(dc, fn) of = ", ".join(file_owners[f]) if fc in file_owners: overwrites = True ofc = ", ".join(file_owners[fc]) else: ofc = "?" bad.append("%s (%s) != %s (%s)" % (f, of, fc, ofc)) for (link, target) in canonicalize_path(self.name, dn, report_links=True): bad.append(" %s -> %s" % (link, target)) if bad: if overwrites: logging.error("FAIL: silently overwrites files via directory symlinks:\n" + indent_string("\n".join(bad))) else: logging.info("dirname part contains a symlink:\n" + indent_string("\n".join(bad))) if not settings.warn_on_install_over_symlink: panic() def remove_packages(self, packages): """Remove packages in a chroot.""" if packages: self.run(["apt-get", "remove"] + unqualify(packages), ignore_errors=True) def purge_packages(self, packages): """Purge packages in a chroot.""" if packages: self.run(["dpkg", "--purge"] + unqualify(packages), ignore_errors=True) def restore_selections(self, selections, packages_qualified): """Restore package selections in a chroot to the state in 'selections'.""" packages = unqualify(packages_qualified) changes = diff_selections(self, selections) deps = {} nondeps = {} for name, state_version in changes.iteritems(): if name in packages: nondeps[name] = state_version else: deps[name] = state_version deps_to_remove = [name for name, (state, version) in deps.iteritems() if state == "remove"] deps_to_purge = [name for name, (state, version) in deps.iteritems() if state == "purge"] nondeps_to_remove = [name for name, (state, version) in nondeps.iteritems() if state == "remove"] nondeps_to_purge = [name for name, (state, version) in nondeps.iteritems() if state == "purge"] all_to_remove = deps_to_remove + deps_to_purge + nondeps_to_remove + nondeps_to_purge all_to_install = [(name, version) for name, (state, version) in deps.iteritems() if state == "install"] all_to_install += [(name, version) for name, (state, version) in nondeps.iteritems() if state == "install"] self.list_paths_with_symlinks() self.check_debsums() self.check_adequate(packages_qualified) # Run custom scripts before removing all packages. self.run_scripts("pre_remove") # First remove all packages (and reinstall missing ones). self.remove_packages(deps_to_remove) if all_to_install: version_qualified = [name for (name, version) in all_to_install if version is None] version_qualified += ["%s=%s" % (name, version) for (name, version) in all_to_install if version is not None] self.apt_get_install(to_remove=all_to_remove, to_install=version_qualified, flags=["--no-install-recommends", "--force-yes"]) else: self.remove_packages(all_to_remove) # Run custom scripts after removing all packages. self.run_scripts("post_remove") if not settings.skip_cronfiles_test: cronfiles, cronfiles_list = self.check_if_cronfiles(packages) if not settings.skip_cronfiles_test and cronfiles: self.check_output_cronfiles(cronfiles_list) if not settings.skip_logrotatefiles_test: logrotatefiles, logrotatefiles_list = self.check_if_logrotatefiles(packages) if not settings.skip_logrotatefiles_test and logrotatefiles: installed = self.install_logrotate() self.check_output_logrotatefiles(logrotatefiles_list) self.purge_packages(installed) # Then purge all packages being depended on. self.purge_packages(deps_to_purge) # Finally, purge actual packages. self.purge_packages(nondeps_to_purge) # Run custom scripts after purge all packages. self.run_scripts("post_purge") # Now do a final run to see that everything worked. self.run(["dpkg", "--purge", "--pending"]) self.run(["dpkg", "--remove", "--pending"]) def save_meta_data(self): """Return the filesystem meta data for all objects in the chroot.""" self.run(["apt-get", "clean"]) logging.debug("Recording chroot state") root = self.relative(".") uidmap = {} with open(self.relative("etc/passwd"), "r") as passwd: for line in passwd: (usr, x, uid) = line.split(":")[0:3] uidmap[int(uid)] = usr gidmap = {} with open(self.relative("etc/group"), "r") as group: for line in group: (grp, x, gid) = line.split(":")[0:3] gidmap[int(gid)] = grp vdict = {} proc = os.path.join(root, "proc") devpts = os.path.join(root, "dev/pts") for dirpath, dirnames, filenames in os.walk(root): assert dirpath[:len(root)] == root if dirpath[:len(proc) + 1] in [proc, proc + "/"]: continue if dirpath[:len(devpts) + 1] in [devpts, devpts + "/"]: continue for name in [dirpath] + \ [os.path.join(dirpath, f) for f in filenames]: st = os.lstat(name) if stat.S_ISLNK(st.st_mode): target = os.readlink(name) else: target = None if stat.S_ISDIR(st.st_mode): name += "/" if st.st_uid in uidmap: user = uidmap[st.st_uid] else: user = "#%d" % st.st_uid if st.st_gid in gidmap: group = gidmap[st.st_gid] else: group = "#%d" % st.st_gid vdict[name[len(root):]] = FileInfo(st, target, user, group) return vdict def relative(self, pathname): if pathname.startswith('/'): return os.path.join(self.name, pathname[1:]) return os.path.join(self.name, pathname) def get_files_owned_by_packages(self): """Return dict[filename] = [packagenamelist].""" vdir = self.relative("var/lib/dpkg/info") vdict = {} for basename in os.listdir(vdir): if basename.endswith(".list"): pkg = basename[:-len(".list")] f = file(os.path.join(vdir, basename), "r") for line in f: pathname = line.strip() if pathname in vdict: vdict[pathname].append(pkg) else: vdict[pathname] = [pkg] f.close() return vdict def check_for_no_processes(self, fail=None): """Check there are no processes running inside the chroot.""" (status, output) = run(["lsof", "-w", "+D", self.name], ignore_errors=True) count = len(output.split("\n")) - 1 if count > 0: if fail is None: fail = not settings.allow_database logging.error("%s: Processes are running inside chroot:\n%s" % ("FAIL" if fail else "WARN", indent_string(output))) if fail: self.terminate_running_processes() panic() def terminate_running_processes(self): """Terminate all processes running in the chroot.""" seen = [] while True: p = subprocess.Popen(["lsof", "-t", "+D", self.name], stdin=subprocess.PIPE, stdout=subprocess.PIPE) stdout, _ = p.communicate() if not stdout: break pidlist = reversed([int(pidstr) for pidstr in stdout.split("\n") if len(pidstr) and int(pidstr) > 0]) if not pidlist: break for pid in pidlist: try: signo = (SIGTERM, SIGKILL)[pid in seen] os.kill(pid, signo) seen.append(pid) logging.debug("kill -%d %d" % (signo, pid)) time.sleep(0.25) except OSError: pass time.sleep(5) # If /selinux is present, assume that this is the only supported # location by libselinux. Otherwise use the new location. # /selinux was shipped by the libselinux package until wheezy. def selinuxfs_path(self): if os.path.isdir(self.relative('/selinux')): return '/selinux' else: return '/sys/fs/selinux' def mount(self, source, path, fstype=None, opts=None): """Mount something into the chroot and remember it for unmount_all().""" path = canonicalize_path(self.name, path) self.mkdir_p(path) fullpath = self.relative(path) command = ["mount"] if fstype is not None: command.extend(["-t", fstype]) if opts is not None: command.extend(["-o", opts]) command.extend([source, fullpath]) run(command) self.mounts.append(fullpath) def unmount_all(self): """Unmount everything we mount()ed into the chroot.""" # Workaround to unmount /proc/sys/fs/binfmt_misc which is mounted by # update-binfmts but never unmounted # This workaround can be removed once #847788 is fixed binfmt_misc = self.relative("/proc/sys/fs/binfmt_misc") if os.path.ismount(binfmt_misc): self.mounts.append(binfmt_misc) for mountpoint in reversed(self.mounts): run(["umount", mountpoint], ignore_errors=True) def mount_proc(self): """Mount /proc etc. inside chroot.""" self.mount("proc", "/proc", fstype="proc") etcmtab = self.relative("etc/mtab") if not os.path.lexists(etcmtab): os.symlink("../proc/mounts", etcmtab) self.mount("devpts", "/dev/pts", fstype="devpts", opts="gid=5,mode=620") self.mount("tmpfs", "/dev/shm", fstype="tmpfs", opts="size=65536k") if selinux_enabled(): self.mount("/sys/fs/selinux", self.selinuxfs_path(), opts="bind,ro") def is_ignored(self, pathname): """Is a file (or dir or whatever) to be ignored?""" if pathname in settings.ignored_files: return True for pattern in settings.ignored_patterns: if re.search('^' + pattern + '$', pathname): return True return False def check_for_broken_symlinks(self): """Check that all symlinks in chroot are non-broken.""" if not settings.check_broken_symlinks: return broken = [] for dirpath, dirnames, filenames in os.walk(self.name): # Remove /proc within chroot to avoid lots of spurious errors. if dirpath == self.name and "proc" in dirnames: dirnames.remove("proc") for filename in filenames: full_name = name = os.path.join(dirpath, filename) if name.startswith(self.name): name = name[len(self.name):] ret = is_broken_symlink(self.name, dirpath, filename) if ret and not self.is_ignored(name): try: target = os.readlink(full_name) except os.error: target = "<unknown>" broken.append("%s -> %s" % (name, target)) if broken: if settings.warn_broken_symlinks: logging.error("WARN: Broken symlinks:\n%s" % indent_string("\n".join(broken))) else: logging.error("FAIL: Broken symlinks:\n%s" % indent_string("\n".join(broken))) panic() else: logging.debug("No broken symlinks as far as we can find.") def check_if_cronfiles(self, packages): """Check if the packages have cron files under /etc/cron.d and in case positive, it returns the list of files. """ vdir = self.relative("var/lib/dpkg/info") vlist = [] has_cronfiles = False for p in packages: basename = p + ".list" if not os.path.exists(os.path.join(vdir, basename)): continue f = file(os.path.join(vdir, basename), "r") for line in f: pathname = line.strip() if pathname.startswith("/etc/cron."): if os.path.isfile(self.relative(pathname.strip("/"))): st = os.lstat(self.relative(pathname.strip("/"))) mode = st[stat.ST_MODE] # XXX /etc/cron.d/ files are NOT executables if (mode & stat.S_IEXEC): if not has_cronfiles: has_cronfiles = True vlist.append(pathname) logging.info("Package " + p + " contains cron file: " + pathname) f.close() return has_cronfiles, vlist def check_output_cronfiles(self, list): """Check if a given list of cronfiles has any output. Executes cron file as cron would do (except for SHELL)""" failed = False for vfile in list: if not os.path.exists(self.relative(vfile.strip("/"))): continue (retval, output) = self.run([vfile]) if output: failed = True logging.error("FAIL: Cron file %s has output with package removed" % vfile) if failed: panic() def check_if_logrotatefiles(self, packages): """Check if the packages have logrotate files under /etc/logrotate.d and in case positive, it returns the list of files. """ vdir = self.relative("var/lib/dpkg/info") vlist = [] has_logrotatefiles = False for p in packages: basename = p + ".list" if not os.path.exists(os.path.join(vdir, basename)): continue f = file(os.path.join(vdir, basename), "r") for line in f: pathname = line.strip() if pathname.startswith("/etc/logrotate.d/"): if os.path.isfile(self.relative(pathname.strip("/"))): if not has_logrotatefiles: has_logrotatefiles = True vlist.append(pathname) logging.info("Package " + p + " contains logrotate file: " + pathname) f.close() return has_logrotatefiles, vlist def install_logrotate(self): """Install logrotate for check_output_logrotatefiles, and return the list of packages that were installed""" old_selections = self.get_selections() self.run(['apt-get', 'install', '-y', 'logrotate']) diff = diff_selections(self, old_selections) return diff.keys() def check_output_logrotatefiles(self, list): """Check if a given list of logrotatefiles has any output. Executes logrotate file as logrotate would do from cron (except for SHELL)""" failed = False for vfile in list: if not os.path.exists(self.relative(vfile.strip("/"))): continue (retval, output) = self.run(['/usr/sbin/logrotate', vfile]) if output or retval != 0: failed = True logging.error("FAIL: Logrotate file %s exits with error or has output with package removed" % file) if failed: panic() def run_scripts(self, step, ignore_errors=False): """ Run custom scripts to given step post-install|remove|purge""" errorcodes = 0 if not settings.scriptsdirs: return errorcodes logging.info("Running scripts " + step) basepath = self.relative("tmp/scripts/") if not os.path.exists(basepath): logging.error("Scripts directory %s does not exist" % basepath) panic() list_scripts = sorted(os.listdir(basepath)) for vfile in list_scripts: if vfile.startswith(step): script = os.path.join("tmp/scripts", vfile) errorcode, output = self.run([script], ignore_errors=ignore_errors) errorcodes = errorcodes | errorcode return errorcodes class VirtServ(Chroot): # Provides a thing that looks to the rest of piuparts much like # a chroot but is actually provided by an adt virtualisation server. # See /usr/share/doc/autopkgtest/README.virtualisation-server. def __init__(self, cmdline): self._cmdline = cmdline self.name = '/ADT-VIRT' self._vs = None def _awaitok(self, cmd): r = self._vs.stdout.readline().rstrip('\n') l = r.split(' ') if l[0] != 'ok': self._fail('virtserver response to %s: %s' % (cmd, r)) logging.debug('adt-virt << %s', r) return l[1:] def _vs_send(self, cmd): if isinstance(cmd, type([])): def maybe_quote(a): if not isinstance(a, type(())): return a (a,) = a return urllib.quote(a) cmd = ' '.join(map(maybe_quote, cmd)) logging.debug('adt-virt >> %s', cmd) print >>self._vs.stdin, cmd return cmd.split(' ')[0] def _command(self, cmd): # argument forms: complete-command-string # [arg, ...] where arg may be (arg,) to quote it cmdp = self._vs_send(cmd) self._vs.stdin.flush() return self._awaitok(cmdp) def _getfilecontents(self, filename): try: (_, tf) = create_temp_file() self._command(['copyup', (filename,), (tf,)]) f = file(tf) d = f.read() f.close() finally: os.remove(tf) return d def create_temp_dir(self): if self._vs is None: logging.debug('adt-virt || %s' % self._cmdline) self._vs = subprocess.Popen(self._cmdline, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None) self._awaitok('banner') self._caps = self._command('capabilities') def shutdown(self): if self._vs is None: return self._vs_send('quit') self._vs.stdin.close() self._vs.stdout.close() self._vs.wait() self._vs = None def remove(self): self._command('close') dont_do_on_panic(self.panic_handler_id) def _fail(self, m): logging.error("adt-virt-* error: " + m) panic() def _open(self): self._scratch = self._command('open')[0] # this is a hack to make install_and_upgrade_between distros # work; we pretend to save the chroot to a tarball but in # fact we do nothing and then we can `restore' the `tarball' with # adt-virt revert def create_temp_tgz_file(self): return self def remove_temp_tgz_file(self, tgz): if tgz is not self: self._fail('removing a tgz not supported') # FIXME: anything else to do here? def pack_into_tgz(self, tgz): if tgz is not self: self._fail('packing into tgz not supported') if not 'revert' in self._caps: self._fail('testbed cannot revert') def unpack_from_tgz(self, tgz): if tgz is not self: self._fail('unpacking from tgz not supported') self._open() def _execute(self, cmdl, tolerate_errors=False): assert isinstance(cmdl, type([])) prefix = ['sh', '-ec', ''' LC_ALL=C unset LANGUAGES export LC_ALL exec 2>&1 exec "$@" ''', '<command>'] ca = ','.join(map(urllib.quote, prefix + cmdl)) stdout = '%s/cmd-stdout' % self._scratch stderr = '%s/cmd-stderr-base' % self._scratch cmd = ['execute', ca, '/dev/null', (stdout,), (stderr,), '/root', 'timeout=600'] es = int(self._command(cmd)[0]) if es and not tolerate_errors: stderr_data = self._getfilecontents(stderr) logging.error("Execution failed (status=%d): %s\n%s" % (es, repr(cmdl), indent_string(stderr_data))) panic() return (es, stdout, stderr) def _execute_getoutput(self, cmdl): (es, stdout, stderr) = self._execute(cmdl) stderr_data = self._getfilecontents(stderr) if es or stderr_data: logging.error('Internal command failed (status=%d): %s\n%s' % (es, repr(cmdl), indent_string(stderr_data))) panic() (_, tf) = create_temp_file() try: self._command(['copyup', (stdout,), (tf,)]) except: os.remove(tf) raise return tf def run(self, command, ignore_errors=False): cmdl = ['sh', '-ec', 'cd /\n' + ' '.join(command)] (es, stdout, stderr) = self._execute(cmdl, tolerate_errors=True) stdout_data = self._getfilecontents(stdout) print >>sys.stderr, "VirtServ run", repr(command), repr(cmdl), '==>', repr(es), repr(stdout), repr(stderr), '|', stdout_data if es == 0 or ignore_errors: return (es, stdout_data) stderr_data = self._getfilecontents(stderr) logging.error('Command failed (status=%d): %s\n%s' % (es, repr(command), indent_string(stdout_data + stderr_data))) panic() def setup_minimal_chroot(self): self._open() def _tbpath(self, with_junk): if not with_junk.startswith(self.name): logging.error("Un-mangling testbed path `%s' but it does not" "start with expected manglement `%s'" % (with_junk, self.name)) panic() return with_junk[len(self.name):] def chmod(self, path, mode): self._execute(['chmod', ('0%o' % mode), self._tbpath(path)]) def remove_files(self, paths): self._execute(['rm', '--'] + map(self._tbpath, paths)) def copy_file(self, our_src, tb_dest): self._command(['copydown', (our_src,), (self._tbpath(tb_dest) + '/' + os.path.basename(our_src),)]) def create_file(self, path, data): path = self._tbpath(path) try: (_, tf) = create_temp_file() f = file(tf, 'w') f.write(tf) f.close() self._command(['copydown', (tf,), (path,)]) finally: os.remove(tf) class DummyStat: pass def save_meta_data(self): mode_map = { 's': stat.S_IFSOCK, 'l': stat.S_IFLNK, 'f': stat.S_IFREG, 'b': stat.S_IFBLK, 'd': stat.S_IFDIR, 'c': stat.S_IFCHR, 'p': stat.S_IFIFO, } vdict = {} tf = self._execute_getoutput(['find', '/', '-xdev', '-printf', "%y %m %U %G %s %p %l \\n".replace(' ', '\\0')]) try: f = file(tf) while True: line = '' while True: splut = line.split('\0') if len(splut) == 8 and splut[7] == '\n': break if len(splut) >= 8: self._fail('aaargh wrong output from find: %s' % urllib.quote(line), repr(splut)) l = f.readline() if not l: if not line: break self._fail('aargh missing final newline from find' ': %s, %s' % (repr(l)[0:200], repr(splut)[0:200])) line += l if not line: break st = VirtServ.DummyStat() st.st_mode = mode_map[splut[0]] | int(splut[1], 8) (st.st_uid, st.st_gid, st.st_size) = map(int, splut[2:5]) vdict[splut[5]] = (st, splut[6]) f.close() finally: os.remove(tf) return vdict def get_files_owned_by_packages(self): tf = self._execute_getoutput(['bash', '-ec', ''' cd /var/lib/dpkg/info find . -name "*.list" -type f -print0 | \\ xargs -r0 egrep . /dev/null test "${PIPESTATUS[*]}" = "0 0" ''']) vdict = {} try: f = file(tf) for l in f: (lf, pathname) = l.rstrip('\n').split(':', 1) assert lf.endswith('.list') pkg = lf[:-5] if pathname in vdict: vdict[pathname].append(pkg) else: vdict[pathname] = [pkg] f.close() finally: os.remove(tf) return vdict def check_for_broken_symlinks(self): if not settings.check_broken_symlinks: return tf = self._execute_getoutput(['bash', '-ec', ''' find / -xdev -type l -print0 | \\ xargs -r0 -i'{}' \\ find '{}' -maxdepth 0 -follow -type l -ls test "${PIPESTATUS[*]}" = "0 0" ''']) try: f = file(tf) broken = False for l in f: logging.error("FAIL: Broken symlink: " + l) broken = True if broken: panic() logging.debug("No broken symlinks found.") finally: os.remove(tf) def check_for_no_processes(self): pass # ?! def mount_proc(self): pass def selinux_enabled(enabled_test="/usr/sbin/selinuxenabled"): if os.access(enabled_test, os.X_OK): retval, output = run([enabled_test], ignore_errors=True) if retval == 0: return True else: return False def objects_are_different(obj1, obj2): """Are filesystem objects different based on their meta data?""" if (obj1.st.st_mode != obj2.st.st_mode or obj1.user != obj2.user or obj1.group != obj2.group or obj1.target != obj2.target): return True if stat.S_ISREG(obj1.st.st_mode): return obj1.st.st_size != obj2.st.st_size # or obj1.st.st_mtime != obj2.st.st_mtime return False def format_object_attributes(obj): st = obj.st ft = "" if stat.S_ISDIR(st.st_mode): ft += "d" if stat.S_ISCHR(st.st_mode): ft += "c" if stat.S_ISBLK(st.st_mode): ft += "b" if stat.S_ISREG(st.st_mode): ft += "-" if stat.S_ISFIFO(st.st_mode): ft += "p" if stat.S_ISLNK(st.st_mode): ft += "l" if stat.S_ISSOCK(st.st_mode): ft += "s" res = "(%s, %s, %s %o, %d, %s)" % ( obj.user, obj.group, ft, st.st_mode, st.st_size, obj.target) return res def diff_meta_data(tree1, tree2): """Compare two dir trees and return list of new files (only in 'tree2'), removed files (only in 'tree1'), and modified files.""" tree1 = tree1.copy() tree2 = tree2.copy() for name in settings.ignored_files: if name in tree1: del tree1[name] if name in tree2: del tree2[name] for pattern in settings.ignored_patterns: pat = re.compile(pattern) for name in tree1.keys(): m = pat.search(name) if m: del tree1[name] for name in tree2.keys(): m = pat.search(name) if m: del tree2[name] modified = [] for name in tree1.keys()[:]: if name in tree2: if objects_are_different(tree1[name], tree2[name]): logging.debug("Modified(user, group, mode, size, target): %s expected%s != found%s" % (name, format_object_attributes(tree1[name]), format_object_attributes(tree2[name]))) modified.append((name, tree1[name])) del tree1[name] del tree2[name] removed = [x for x in tree1.iteritems()] new = [x for x in tree2.iteritems()] # fix for #586793 # prune rc?.d symlinks renamed by insserv pat1 = re.compile(r"^(/etc/rc.\.d/)[SK][0-9]{2}(.*)$") for name1, data1 in removed[:]: m = pat1.search(name1) if m: pat2 = re.compile(r"^" + m.group(1) + r"[SK][0-9]{2}" + m.group(2) + r"$") for name2, data2 in new[:]: m = pat2.search(name2) if m: logging.debug("File was renamed: %s\t=> %s" % (name1, name2)) removed.remove((name1, data1)) new.remove((name2, data2)) # this is again special casing due to the behaviour of a single package :( # general tracking of moved files would be the better approach, probably. return new, removed, modified def file_list(meta_infos, file_owners): """Return list of indented filenames.""" meta_infos = sorted(meta_infos[:]) vlist = [] for name, obj in meta_infos: info = "" if obj.target is not None: info = " -> %s" % obj.target vlist.append(" %s%s\t" % (name, info)) key = name if key.endswith('/'): key = key[:-1] if key in file_owners: vlist.append(" owned by: %s\n" % ", ".join(file_owners[key])) else: vlist.append(" not owned\n") return "".join(vlist) def offending_packages(meta_infos, file_owners): """Return a Set of offending packages.""" pkgset = set() for name, data in meta_infos: if name in file_owners: for pkg in file_owners[name]: pkgset.add(pkg) return pkgset def prune_files_list(files, depsfiles): """Remove elements from 'files' that are in 'depsfiles', and return the list of removed elements. """ warn = [] depfiles_names = [x[0] for x in depsfiles] for vfile in files[:]: if vfile[0] in depfiles_names: files.remove(vfile) warn.append(vfile) return warn def diff_selections(chroot, selections): """Compare original and current package selection. Return dict where dict[package_name] = original_status, that is, the value in the dict is the state that the package needs to be set to to restore original selections.""" changes = {} current = chroot.get_selections() for name, (value, version) in current.iteritems(): if name not in selections: changes[name] = ("purge", None) elif selections[name][0] != value and \ selections[name][0] in ["purge", "install"]: changes[name] = selections[name] for name, (value, version) in selections.iteritems(): if name not in current or \ current[name][1] != version: changes[name] = selections[name] return changes def get_package_names_from_package_files(package_files): """Return list of package names given list of package file names.""" vlist = [] for filename in package_files: (status, output) = run(["dpkg", "--info", filename]) p = None v = None for line in [line.lstrip() for line in output.split("\n")]: if line.startswith("Package:"): p = line.split(":", 1)[1].strip() if line.startswith("Version:"): v = line.split(":", 1)[1].strip() if p is not None: if v is not None: vlist.append(p + "=" + v) else: vlist.append(p) return vlist # Method to process a changes file, returning a list of all the .deb packages # from the 'Files' stanza. def process_changes(changes): # Determine the path to the changes file, then check if it's readable. dir_path = "" changes_path = "" if not os.path.dirname(changes): changes_path = os.path.basename(changes) else: dir_path = os.path.dirname(changes) + "/" changes_path = os.path.abspath(changes) if not os.access(changes_path, os.R_OK): logging.warn(changes_path + " is not readable. Skipping.") return # Determine the packages in the changes file through the 'Files' stanza. field = 'Files' pattern = re.compile( r'^' + field + r':' + r''' # The field we want the contents from (.*?) # The contents of the field \n([^ ]|$) # Start of a new field or EOF ''', re.MULTILINE | re.DOTALL | re.VERBOSE) f = open(changes_path) file_text = f.read() f.close() matches = pattern.split(file_text) # Append all the packages found in the changes file to a package list. package_list = [] newline_p = re.compile('\n') package_p = re.compile('.*?([^ ]+\.deb)$') for line in newline_p.split(matches[1]): if package_p.match(line): package = dir_path + package_p.split(line)[1] package_list.append(package) # Return the list. return package_list def check_results(chroot, chroot_state, file_owners, deps_info=None): """Check that current chroot state matches 'chroot_state'. If settings.warn_on_others is True and deps_info is not None, then only print a warning rather than failing if the current chroot contains files that are in deps_info but not in root_info. (In this case, deps_info should be the result of chroot.save_meta_data() right after the dependencies are installed, but before the actual packages to test are installed.) """ root_info = chroot_state["tree"] ok = True if settings.check_broken_diversions: (removed, added) = chroot.get_modified_diversions(chroot_state["diversions"]) if added: logging.error("FAIL: Installed diversions (dpkg-divert) not removed by purge:\n%s" % indent_string("\n".join(added))) ok = False if removed: logging.error("FAIL: Existing diversions (dpkg-divert) removed/modified:\n%s" % indent_string("\n".join(removed))) ok = False current_info = chroot.save_meta_data() if settings.warn_on_others and deps_info is not None: (new, removed, modified) = diff_meta_data(root_info, current_info) (depsnew, depsremoved, depsmodified) = diff_meta_data(root_info, deps_info) warnnew = prune_files_list(new, depsnew) warnremoved = prune_files_list(removed, depsremoved) warnmodified = prune_files_list(modified, depsmodified) else: (new, removed, modified) = diff_meta_data(root_info, current_info) if new: if settings.warn_on_leftovers_after_purge: logging.info("Warning: Package purging left files on system:\n" + file_list(new, file_owners)) else: logging.error("FAIL: Package purging left files on system:\n" + file_list(new, file_owners)) ok = False if removed: logging.error("FAIL: After purging files have disappeared:\n" + file_list(removed, file_owners)) ok = False if modified: logging.error("FAIL: After purging files have been modified:\n" + file_list(modified, file_owners)) ok = False if settings.warn_on_others and deps_info is not None: if warnnew: msg = ("Warning: Package purging left files on system:\n" + file_list(warnnew, file_owners) + "These files seem to have been left by dependencies rather " "than by packages\nbeing explicitly tested.\n") logging.info(msg) if warnremoved: msg = ("After purging files have disappeared:\n" + file_list(warnremoved, file_owners) + "This seems to have been caused by dependencies rather " "than by packages\nbeing explicitly tested.\n") logging.info(msg) if warnmodified: msg = ("After purging files have been modified:\n" + file_list(warnmodified, file_owners) + "This seems to have been caused by dependencies rather " "than by packages\nbeing explicitly tested.\n") logging.info(msg) return ok def install_purge_test(chroot, chroot_state, package_files, packages, extra_packages): """Do an install-purge test. Return True if successful, False if not. Assume 'root' is a directory already populated with a working chroot, with packages in states given by 'selections'.""" deps_info = None os.environ["PIUPARTS_TEST"] = "install" chroot.run_scripts("pre_test") # Install packages into the chroot. os.environ["PIUPARTS_PHASE"] = "install" chroot.enable_testdebs_repo() chroot.check_for_no_processes(fail=True) chroot.check_for_broken_symlinks() chroot.run_scripts("pre_install") chroot.install_packages([], extra_packages, with_scripts=False) if settings.warn_on_others or settings.install_purge_install: # Create a metapackage with dependencies from the given packages if package_files: control_infos = [] # We were given package files, so let's get the Depends and # Conflicts directly from the .debs for deb in package_files: returncode, output = run(["dpkg", "-f", deb]) control = deb822.Deb822(output) control_infos.append(control) else: # We have package names. Use apt to get all their control # information. apt_cache_args = ["apt-cache", "show"] if os.environ["PIUPARTS_DISTRIBUTION"] in ["lenny"]: # apt-cache in lenny does not accept version-qualified packages apt_cache_args.extend(unqualify(packages)) else: apt_cache_args.extend(packages) returncode, output = chroot.run(apt_cache_args) control_infos = deb822.Deb822.iter_paragraphs(output.splitlines()) depends = [] conflicts = [] provides = [] arch = 'all' for control in control_infos: if control.get("pre-depends"): depends.extend([x.strip() for x in control["pre-depends"].split(',')]) if control.get("depends"): depends.extend([x.strip() for x in control["depends"].split(',')]) if control.get("conflicts"): conflicts.extend([x.strip() for x in control["conflicts"].split(',')]) if control.get("provides"): provides.extend([x.strip() for x in control["provides"].split(',')]) if control.get("architecture"): a = control["architecture"] if arch == 'all': arch = a if arch != a: logging.info("architecture mismatch: %s != %s)" % (arch, a)) for provided in provides: if provided in conflicts: conflicts.remove(provided) all_depends = ", ".join(depends) all_conflicts = ", ".join(conflicts) metapackage = make_metapackage("piuparts-depends-dummy", depends=all_depends, conflicts=all_conflicts, arch=arch) cleanup_metapackage = lambda: shutil.rmtree(os.path.dirname(metapackage)) panic_handler_id = do_on_panic(cleanup_metapackage) # Install the metapackage chroot.install_package_files([metapackage], with_scripts=False) # Check whether it got installed, the 'dpkg -i p-d-d.deb && apt-get -yf install' approach # may not have installed it, cannot happen with 'apt-get install p-d-d.deb' (since stretch) if not chroot.is_installed(["piuparts-depends-dummy"]): logging.error("Installation of piuparts-depends-dummy FAILED") # don't panic(), too many problems on old distros # Now remove it metapackagename = os.path.basename(metapackage)[:-4] chroot.purge_packages([metapackagename]) cleanup_metapackage() dont_do_on_panic(panic_handler_id) # Save the file ownership information so we can tell which # modifications were caused by the actual packages we are testing, # rather than by their dependencies. deps_info = chroot.save_meta_data() if settings.install_purge_install: # save chroot state with all deps installed chroot_state_with_deps = {} chroot_state_with_deps["tree"] = deps_info chroot_state_with_deps["selections"] = chroot.get_selections() chroot_state_with_deps["diversions"] = chroot.get_diversions() chroot.check_for_no_processes() chroot.check_for_broken_symlinks() chroot.install_packages(package_files, packages, with_scripts=False) chroot.run_scripts("post_install") if settings.install_purge_install: file_owners = chroot.get_files_owned_by_packages() chroot.restore_selections(chroot_state_with_deps["selections"], packages) logging.info("Validating chroot after purge") chroot.check_debsums() chroot.check_for_no_processes() chroot.check_for_broken_symlinks() if not check_results(chroot, chroot_state_with_deps, file_owners, deps_info=deps_info): return False logging.info("Reinstalling after purge") chroot.install_packages(package_files, packages, with_scripts=True) if settings.install_remove_install: chroot.remove_packages(packages) logging.info("Reinstalling after remove") chroot.install_packages(package_files, packages, with_scripts=True) chroot.install_packages(package_files, packages, with_scripts=True, reinstall=True) chroot.check_for_no_processes() chroot.check_for_broken_symlinks() file_owners = chroot.get_files_owned_by_packages() chroot.disable_testdebs_repo() # Remove all packages from the chroot that weren't there initially. chroot.restore_selections(chroot_state["selections"], packages) chroot.check_for_no_processes(fail=True) chroot.check_for_broken_symlinks() return check_results(chroot, chroot_state, file_owners, deps_info=deps_info) def install_upgrade_test(chroot, chroot_state, package_files, packages, old_packages): """Install old_packages via apt-get, then upgrade from package files. Return True if successful, False if not.""" os.environ["PIUPARTS_TEST"] = "upgrade" chroot.run_scripts("pre_test") # First install via apt-get. os.environ["PIUPARTS_PHASE"] = "install" chroot.install_packages_by_name(old_packages) chroot.check_for_no_processes() chroot.check_for_broken_symlinks() if settings.install_remove_install: chroot.remove_packages(packages) # Then from the package files. os.environ["PIUPARTS_PHASE"] = "upgrade" chroot.enable_testdebs_repo() chroot.install_packages(package_files, packages) chroot.check_for_no_processes() chroot.check_for_broken_symlinks() file_owners = chroot.get_files_owned_by_packages() chroot.disable_testdebs_repo() # Remove all packages from the chroot that weren't there initially. chroot.restore_selections(chroot_state["selections"], packages) chroot.check_for_no_processes(fail=True) chroot.check_for_broken_symlinks() return check_results(chroot, chroot_state, file_owners) def save_meta_data(filename, chroot_state): """Save directory tree meta data into a file for fast access later.""" logging.debug("Saving chroot meta data to %s" % filename) f = file(filename, "w") pickle.dump(chroot_state, f) f.close() def load_meta_data(filename): """Load meta data saved by 'save_meta_data'.""" logging.debug("Loading chroot meta data from %s" % filename) f = file(filename, "r") chroot_state = pickle.load(f) f.close() return chroot_state def install_and_upgrade_between_distros(package_files, packages_qualified): """Install package and upgrade it between distributions, then remove. Return True if successful, False if not.""" # this function is a bit confusing at first, because of what it does by default: # 1. create chroot with source distro # 2. upgrade chroot to target distro # 3. remove chroot and recreate chroot with source distro # 4. install depends in chroot # 5. install package in chroot # 6. upgrade chroot to target distro # 7. remove package and depends # 8. compare results # # sounds silly, or? # well, it is a reasonable default (see below for why), but # step 2+3 can be done differently by using --save-end-meta once and # then --end-meta for all following runs - until the target distro # changes again... # # Under normal circumstances the target distro can change anytime, ie. at # the next mirror pulse, so unless the target distro is frozen, this is # a reasonable default behaviour for distro upgrade tests, which are not # done by default anyway. os.environ["PIUPARTS_TEST"] = "distupgrade" packages = unqualify(packages_qualified) chroot = get_chroot() chroot.create() chroot_state = None if settings.end_meta: if os.path.exists(settings.end_meta): # load root_info and selections chroot_state = load_meta_data(settings.end_meta) else: logging.info("Cannot load chroot state from %s - generating it on-the-fly." % settings.end_meta) if chroot_state is None: temp_tgz = None if chroot.was_bootstrapped(): temp_tgz = chroot.create_temp_tgz_file() panic_handler_id = do_on_panic(lambda: chroot.remove_temp_tgz_file(temp_tgz)) chroot.pack_into_tgz(temp_tgz) chroot.upgrade_to_distros(settings.debian_distros[1:], []) chroot.check_for_no_processes(fail=True) # set root_info and selections chroot_state = {} chroot_state["tree"] = chroot.save_meta_data() chroot_state["selections"] = chroot.get_selections() chroot_state["diversions"] = chroot.get_diversions() if settings.save_end_meta: # save root_info and selections save_meta_data(settings.save_end_meta, chroot_state) chroot.remove() # leave indication in logfile why we do what we do logging.info( "Notice: package selections and meta data from target distro saved, now starting over from source distro. See the description of --save-end-meta and --end-meta to learn why this is neccessary and how to possibly avoid it.") chroot = get_chroot() if temp_tgz is None: chroot.create() else: chroot.create(temp_tgz) chroot.remove_temp_tgz_file(temp_tgz) dont_do_on_panic(panic_handler_id) chroot.check_for_no_processes(fail=True) if chroot.run_scripts("is_testable", ignore_errors=True) != 0: logging.info("SKIP: All tests. Package cannot be tested with piuparts: %s.", " ".join(packages)) chroot.remove() return True if settings.shell_on_error: panic_handler_id = do_on_panic(lambda: chroot.interactive_shell()) chroot.run_scripts("pre_test") os.environ["PIUPARTS_PHASE"] = "install" distupgrade_packages = packages known_packages = chroot.get_known_packages(packages + settings.extra_old_packages) chroot.install_packages_by_name(known_packages) if settings.install_remove_install: chroot.remove_packages(packages) distupgrade_packages = [] chroot.check_for_no_processes() os.environ["PIUPARTS_PHASE"] = "distupgrade" chroot.upgrade_to_distros(settings.debian_distros[1:-1], distupgrade_packages) if settings.distupgrade_to_testdebs: chroot.enable_testdebs_repo(update=False) chroot.upgrade_to_distros(settings.debian_distros[-1:], distupgrade_packages) chroot.check_for_no_processes() os.environ["PIUPARTS_PHASE"] = "upgrade" if not settings.distupgrade_to_testdebs: chroot.enable_testdebs_repo() chroot.install_packages(package_files, [p for p in packages_qualified if not p.endswith("=None")]) chroot.disable_testdebs_repo() chroot.check_for_no_processes() file_owners = chroot.get_files_owned_by_packages() # Remove all packages from the chroot that weren't in the reference chroot. chroot.restore_selections(chroot_state["selections"], packages_qualified) chroot.check_for_no_processes(fail=True) result = check_results(chroot, chroot_state, file_owners) if settings.shell_on_error: dont_do_on_panic(panic_handler_id) if not result: chroot.interactive_shell() chroot.remove() return result def parse_mirror_spec(str, defaultcomponents=[]): """Parse a mirror specification from the --mirror option argument. Return (mirror, componentslist).""" parts = str.split() return parts[0], parts[1:] or defaultcomponents[:] def find_default_debian_mirrors(): """Find the default Debian mirrors.""" mirrors = [] try: f = file("/etc/apt/sources.list", "r") for line in f: line = re.sub('\[arch=.*\]', '', line) parts = line.split() if len(parts) > 2 and parts[0] == "deb": mirrors.append((parts[1], parts[3:])) break # Only use the first one, at least for now. f.close() except IOError: return None return mirrors def forget_ignores(option, opt, value, parser, *args, **kwargs): settings.bindmounts = [] parser.values.ignore = [] parser.values.ignore_regex = [] settings.ignored_files = [] settings.ignored_patterns = [] def set_basetgz_to_pbuilder(option, opt, value, parser, *args, **kwargs): parser.values.basetgz = "/var/cache/pbuilder/base.tgz" def parse_command_line(): """Parse the command line, change global settings, return non-options.""" parser = optparse.OptionParser(usage="%prog [options] package ...", version="piuparts %s" % VERSION) parser.add_option("-a", "--apt", action="store_true", default=False, help="Command line arguments are package names " + "to be installed via apt.") parser.add_option("--arch", metavar="ARCH", action="store", help="Create chroot and run tests for (non-default) architecture ARCH.") parser.add_option("--adt-virt", metavar='CMDLINE', default=None, help="Use CMDLINE via autopkgtest (adt-virt-*)" " protocol instead of managing a chroot.") parser.add_option("-b", "--basetgz", metavar="TARBALL", help="Use TARBALL as the contents of the initial " + "chroot, instead of building a new one with " + "debootstrap.") parser.add_option("--bindmount", action="append", metavar="DIR", default=[], help="Directory to be bind-mounted inside the chroot.") parser.add_option("-d", "--distribution", action="append", metavar="NAME", help="Which Debian distribution to use: a code name " + "(for example jessie, stretch, sid) or experimental. The " + "default is sid (=unstable).") parser.add_option("-D", "--defaults", action="store", help="Choose which set of defaults to use " "(debian/ubuntu).") parser.add_option("--debfoster-options", default="-o MaxPriority=required -o UseRecommends=no -f -n apt debfoster", help="Run debfoster with different parameters (default: -o MaxPriority=required -o UseRecommends=no -f -n apt debfoster).") parser.add_option("--no-eatmydata", default=False, action='store_true', help="Default is to use libeatmydata in the chroot") parser.add_option("--dpkg-noforce-unsafe-io", default=False, action='store_true', help="Default is to run dpkg with --force-unsafe-io option, which causes dpkg to skip certain file system syncs known to cause substantial performance degradation on some filesystems. This option turns that off and dpkg will use safe I/O operations.") parser.add_option("--dpkg-force-confdef", default=False, action='store_true', help="Make dpkg use --force-confdef, which lets dpkg always choose the default action when a modified conffile is found. This option will make piuparts ignore errors it was designed to report and therefore should only be used to hide problems in depending packages. (See #466118.)") parser.add_option("--do-not-verify-signatures", default=False, action='store_true', help="Do not verify signatures from the Release files when running debootstrap.") parser.add_option("--no-check-valid-until", default=False, action='store_true', help="Set apt option Acquire::Check-Valid-Until=false for testing archived releases.") parser.add_option("--allow-database", default=False, action='store_true', help="Allow database servers (MySQL, PostgreSQL) to be started in the chroot.") parser.add_option("--distupgrade-to-testdebs", default=False, action='store_true', help="Use the testdebs repository as distupgrade target.") parser.add_option("-e", "--existing-chroot", metavar="DIR", help="Use DIR as the contents of the initial " + "chroot, instead of building a new one with " + "debootstrap") parser.add_option("--hard-link", default=False, action='store_true', help="When using --existing-chroot, and the source dir is on the same" "filesystem, hard-link files instead of copying them.") parser.add_option("-i", "--ignore", action="append", metavar="FILENAME", default=[], help="Add FILENAME to list of filenames to be " + "ignored when comparing changes to chroot.") parser.add_option("-I", "--ignore-regex", action="append", metavar="REGEX", default=[], help="Add REGEX to list of Perl compatible regular " + "expressions for filenames to be " + "ignored when comparing changes to chroot.") parser.add_option("--install-recommends", action="store_true", default=False, help="Enable the installation of Recommends.") parser.add_option("--install-suggests", action="store_true", default=False, help="Enable the installation of Suggests.") parser.add_option("-k", "--keep-tmpdir", action="store_true", default=False, help="Don't remove the temporary directory for the " + "chroot when the program ends.") parser.add_option("-K", "--keyring", action="store", metavar="FILE", help="Use FILE as the keyring to use with debootstrap when creating chroots.") parser.add_option("--keep-sources-list", action="store_true", default=False, help="Don't modify the chroot's " + "etc/apt/sources.list.") parser.add_option("-l", "--log-file", "--logfile", metavar="FILENAME", help="Write log file to FILENAME in addition to " + "the standard output.") parser.add_option("--list-installed-files", action="store_true", default=False, help="List files added to the chroot after the " + "installation of the package.") parser.add_option("--lvm-volume", metavar="LVM-VOL", action="store", help="Use LVM-VOL as source for the chroot, instead of building " + "a new one with debootstrap. This creates a snapshot of the " + "given LVM volume and mounts it to the chroot path") parser.add_option("--lvm-snapshot-size", metavar="SNAPSHOT-SIZE", action="store", default="1G", help="Use SNAPSHOT-SIZE as snapshot size when creating " + "a new LVM snapshot (default: 1G)") parser.add_option("--schroot", metavar="SCHROOT-NAME", action="store", help="Use schroot session named SCHROOT-NAME for the chroot, instead of building " + "a new one with debootstrap.") parser.add_option("-m", "--mirror", action="append", metavar="URL", default=[], help="Which Debian mirror to use.") parser.add_option("--extra-repo", action="append", default=[], help="Additional (unparsed) lines to be appended to sources.list, e.g. " + "'deb <URL> <distrib> <components>' or 'deb file://</bind/mount> ./'") parser.add_option("--testdebs-repo", help="A repository that contains the packages to be tested, e.g. " + "'deb <URL> <distrib> <components>...' or 'deb file://</bind/mount> ./'," + "plain URLs or local paths are permitted, too.") parser.add_option("--no-adequate", default=False, action='store_true', help="Don't run adequate after installation.") parser.add_option("--no-diversions", action="store_true", default=False, help="Don't check for broken diversions.") parser.add_option("-n", "--no-ignores", action="callback", callback=forget_ignores, help="Forget all ignores set so far, including " + "built-in ones.") parser.add_option("-N", "--no-symlinks", action="store_true", default=False, help="Don't check for broken symlinks.") parser.add_option("--no-upgrade-test", action="store_true", default=False, help="Skip testing the upgrade from an existing version " + "in the archive.") parser.add_option("--no-install-purge-test", action="store_true", default=False, help="Skip install and purge test.") parser.add_option("--install-purge-install", action="store_true", default=False, help="Purge package after installation and reinstall.") parser.add_option("--install-remove-install", action="store_true", default=False, help="Remove package after installation and reinstall. For testing installation in config-files-remaining state.") parser.add_option("--fake-essential-packages", action="append", default=[], help="Install additional packages in the base chroot that are not removed after the test. " + "Takes a comma separated list of package names and can be given multiple times. " + "Useful for packages that can be used during purge of the package to be tested " + "or to test whether the package to be tested mishandles these packages.") parser.add_option("--extra-old-packages", action="append", default=[], help="Install these additional packages along with the old packages from the archive. " + "Useful to test Conflicts/Replaces of packages that will disappear during the update. " + "Takes a comma separated list of package names and can be given multiple times. " + "For install/purge tests these packages will be installed before the package that is to be tested.") parser.add_option("-p", "--pbuilder", action="callback", callback=set_basetgz_to_pbuilder, help="Use /var/cache/pbuilder/base.tgz as the base " + "tarball.") parser.add_option("--pedantic-purge-test", action="store_true", default=False, help="Be pedantic when checking if a purged package leaves files behind. If this option is not set, files left in /tmp are ignored.") parser.add_option("--proxy", metavar="URL", help="Use the proxy at URL for accessing the mirrors.") parser.add_option("-s", "--save", metavar="FILENAME", help="Save the chroot into FILENAME.") parser.add_option("-B", "--end-meta", metavar="FILE", help="Load chroot package selection and file meta data from FILE. See the function install_and_upgrade_between_distros() in piuparts.py for defaults. Mostly useful for large scale distro upgrade tests.") parser.add_option("-S", "--save-end-meta", metavar="FILE", help="Save chroot package selection and file meta data in FILE for later use. See the function install_and_upgrade_between_distros() in piuparts.py for defaults. Mostly useful for large scale distro upgrade tests.") parser.add_option("--single-changes-list", default=False, action="store_true", help="test all packages from all changes files together.") parser.add_option("--single-packages", default=False, action="store_true", help="test all packages from the command line individually.") parser.add_option("--skip-cronfiles-test", action="store_true", default=False, help="Skip testing the output from the cron files.") parser.add_option("--skip-logrotatefiles-test", action="store_true", default=False, help="Skip testing the output from the logrotate files.") parser.add_option("--skip-minimize", action="store_true", default=True, help="Skip minimize chroot step. This is the default now.") parser.add_option("--minimize", action="store_true", default=False, help="Minimize chroot with debfoster. This used to be the default until #539142 was fixed.") parser.add_option("--scriptsdir", metavar="DIR", action="append", default=[], help="Directory where are placed the custom scripts. Can be given multiple times.") parser.add_option("--shell-on-error", default=False, action='store_true', help="Execute an interactive shell in the chroot if an error occurred.") parser.add_option("-t", "--tmpdir", metavar="DIR", help="Use DIR for temporary storage. Default is " + "$TMPDIR or /tmp.") parser.add_option("-v", "--verbose", action="store_true", default=False, help="No meaning anymore.") parser.add_option("--warn-on-others", action="store_true", default=False, help="Print a warning rather than failing if " "files are left behind, modified, or removed " "by a package that was not given on the " "command-line. Behavior with multiple packages " "given could be problematic, particularly if the " "dependency tree of one package in the list " "includes another in the list. Therefore, it is " "recommended to use this option with one package " "at a time.") parser.add_option("--warn-on-leftovers-after-purge", action="store_true", default=False, help="Print a warning rather than failing if " "files are left behind after purge.") parser.add_option("--warn-on-debsums-errors", action="store_true", default=False, help="Print a warning rather than failing if " "debsums reports modified files.") parser.add_option("--warn-on-install-over-symlink", action="store_true", default=False, help="Print a warning rather than failing if " "files are installed over existing symlinks.") parser.add_option("--fail-if-inadequate", action="store_true", default=False, help="Fail on inadequate results from running adequate.") parser.add_option("--fail-on-broken-symlinks", action="store_true", default=False, help="Fail if broken symlinks are detected.") parser.add_option("--log-level", action="store", metavar='LEVEL', default="dump", help="Displays messages from LEVEL level, possible values are: error, info, dump, debug. The default is dump.") (opts, args) = parser.parse_args() settings.defaults = opts.defaults defaults = DefaultsFactory().new_defaults() settings.tmpdir = opts.tmpdir settings.keep_tmpdir = opts.keep_tmpdir settings.shell_on_error = opts.shell_on_error settings.single_changes_list = opts.single_changes_list settings.single_packages = opts.single_packages settings.args_are_package_files = not opts.apt # distro setup settings.proxy = opts.proxy if settings.proxy: os.environ["http_proxy"] = settings.proxy settings.debian_mirrors = [parse_mirror_spec(x, defaults.get_components()) for x in opts.mirror] settings.extra_repos = opts.extra_repo settings.testdebs_repo = opts.testdebs_repo settings.debian_distros = opts.distribution settings.keep_sources_list = opts.keep_sources_list if opts.keyring: settings.keyring = opts.keyring else: settings.keyring = defaults.get_keyring() settings.do_not_verify_signatures = opts.do_not_verify_signatures if settings.do_not_verify_signatures: settings.apt_unauthenticated = "Yes" else: settings.apt_unauthenticated = "No" settings.no_check_valid_until = opts.no_check_valid_until settings.install_recommends = opts.install_recommends settings.install_suggests = opts.install_suggests settings.eatmydata = not opts.no_eatmydata settings.dpkg_force_unsafe_io = not opts.dpkg_noforce_unsafe_io settings.dpkg_force_confdef = opts.dpkg_force_confdef settings.scriptsdirs = opts.scriptsdir settings.bindmounts += opts.bindmount settings.allow_database = opts.allow_database # chroot setup settings.arch = opts.arch settings.basetgz = opts.basetgz settings.savetgz = opts.save settings.lvm_volume = opts.lvm_volume settings.lvm_snapshot_size = opts.lvm_snapshot_size settings.existing_chroot = opts.existing_chroot settings.hard_link = opts.hard_link settings.schroot = opts.schroot settings.end_meta = opts.end_meta settings.save_end_meta = opts.save_end_meta settings.skip_minimize = opts.skip_minimize settings.minimize = opts.minimize if settings.minimize: settings.skip_minimize = False settings.debfoster_options = opts.debfoster_options.split() # tests and checks settings.no_install_purge_test = opts.no_install_purge_test settings.no_upgrade_test = opts.no_upgrade_test settings.distupgrade_to_testdebs = opts.distupgrade_to_testdebs settings.install_purge_install = opts.install_purge_install settings.install_remove_install = opts.install_remove_install settings.list_installed_files = opts.list_installed_files [settings.fake_essential_packages.extend([i.strip() for i in csv.split(",")]) for csv in opts.fake_essential_packages] [settings.extra_old_packages.extend([i.strip() for i in csv.split(",")]) for csv in opts.extra_old_packages] settings.skip_cronfiles_test = opts.skip_cronfiles_test settings.skip_logrotatefiles_test = opts.skip_logrotatefiles_test settings.adequate = not opts.no_adequate settings.check_broken_diversions = not opts.no_diversions settings.check_broken_symlinks = not opts.no_symlinks settings.warn_broken_symlinks = not opts.fail_on_broken_symlinks settings.warn_on_others = opts.warn_on_others settings.warn_on_leftovers_after_purge = opts.warn_on_leftovers_after_purge settings.warn_on_debsums_errors = opts.warn_on_debsums_errors settings.warn_on_install_over_symlink = opts.warn_on_install_over_symlink settings.warn_if_inadequate = not opts.fail_if_inadequate settings.pedantic_purge_test = opts.pedantic_purge_test settings.ignored_files += opts.ignore settings.ignored_patterns += opts.ignore_regex if not settings.pedantic_purge_test: settings.ignored_patterns += settings.non_pedantic_ignore_patterns if opts.adt_virt is None: settings.adt_virt = None else: settings.adt_virt = VirtServ(opts.adt_virt) log_file_name = opts.log_file if opts.log_level == "error": setup_logging(logging.ERROR, log_file_name) elif opts.log_level == "info": setup_logging(logging.INFO, log_file_name) elif opts.log_level == "debug": setup_logging(logging.DEBUG, log_file_name) else: setup_logging(DUMP, log_file_name) exitcode = None if not settings.tmpdir: if "TMPDIR" in os.environ: settings.tmpdir = os.environ["TMPDIR"] else: settings.tmpdir = "/tmp" if not os.path.isdir(settings.tmpdir): logging.error("Temporary directory is not a directory: %s" % settings.tmpdir) panic() for sdir in settings.scriptsdirs: if not os.path.isdir(sdir): logging.error("Scripts directory is not a directory: %s" % sdir) panic() if not settings.debian_distros: settings.debian_distros = defaults.get_distribution() if not settings.debian_mirrors: if opts.defaults: settings.debian_mirrors = defaults.get_mirror() else: settings.debian_mirrors = find_default_debian_mirrors() if not settings.debian_mirrors: settings.debian_mirrors = defaults.get_mirror() settings.distro_config = piupartslib.conf.DistroConfig( DISTRO_CONFIG_FILE, settings.debian_mirrors[0][0]) if settings.keep_sources_list and len(settings.debian_distros) > 1: logging.error("--keep-sources-list only makes sense " "with only one distribution") exitcode = 1 if not args: logging.error("Need command line arguments: " + "names of packages or package files") exitcode = 1 settings.testobjects = args if exitcode is not None: sys.exit(exitcode) return args def get_chroot(): if settings.adt_virt is None: return Chroot() return settings.adt_virt # Process the packages given in a list def process_packages(package_list): # Find the names of packages. if settings.args_are_package_files: packages = get_package_names_from_package_files(package_list) package_files = package_list else: packages = package_list package_files = [] if len(settings.debian_distros) == 1: chroot = get_chroot() chroot.create() if settings.shell_on_error: panic_handler_id = do_on_panic(lambda: chroot.interactive_shell()) chroot_state = {} chroot_state["tree"] = chroot.save_meta_data() chroot_state["selections"] = chroot.get_selections() chroot_state["diversions"] = chroot.get_diversions() testable = True if chroot.run_scripts("is_testable", ignore_errors=True) != 0: logging.info("SKIP: All tests. Package cannot be tested with piuparts: %s.", " ".join(packages)) testable = False if testable and not settings.no_install_purge_test: extra_packages = chroot.get_known_packages(settings.extra_old_packages) if not install_purge_test(chroot, chroot_state, package_files, packages, extra_packages): logging.error("FAIL: Installation and purging test.") panic() logging.info("PASS: Installation and purging test.") if testable and not settings.no_upgrade_test: if not settings.args_are_package_files and not settings.testdebs_repo: logging.info("Can't test upgrades: -a or --apt option used.") else: packages_to_query = unqualify(packages) packages_to_query.extend(settings.extra_old_packages) known_packages = chroot.get_known_packages(packages_to_query) if not known_packages: logging.info("Can't test upgrade: packages not known by apt-get.") elif install_upgrade_test(chroot, chroot_state, package_files, packages, known_packages): logging.info("PASS: Installation, upgrade and purging tests.") else: logging.error("FAIL: Installation, upgrade and purging tests.") panic() if settings.shell_on_error: dont_do_on_panic(panic_handler_id) chroot.remove() else: if install_and_upgrade_between_distros(package_files, packages): logging.info("PASS: Upgrading between Debian distributions.") else: logging.error("FAIL: Upgrading between Debian distributions.") panic() if settings.adt_virt is not None: settings.adt_virt.shutdown() def main(): """Main program. But you knew that.""" args = parse_command_line() # check if user has root privileges if os.getuid(): print 'You need to be root to use piuparts.' sys.exit(1) logging.info("-" * 78) logging.info("To quickly glance what went wrong, scroll down to the bottom of this logfile.") logging.info("FAQ available at https://wiki.debian.org/piuparts/FAQ") logging.info("The FAQ also explains how to contact us in case you think piuparts is wrong.") logging.info("-" * 78) logging.info("piuparts version %s starting up." % VERSION) logging.info("Command line arguments: %s" % " ".join(quote_spaces(sys.argv))) logging.info("Running on: %s %s %s %s %s" % os.uname()) # Make sure debconf does not ask questions and stop everything. # Packages that don't use debconf will lose. os.environ["DEBIAN_FRONTEND"] = "noninteractive" # Reduce the amount of ESC-codes in the logfile. os.environ["DPKG_COLORS"] = "never" if "DISPLAY" in os.environ: del os.environ["DISPLAY"] changes_packages_list = [] regular_packages_list = [] changes_p = re.compile('.*\.changes$') for arg in args: if changes_p.match(arg): package_list = process_changes(arg) if settings.single_changes_list: for package in package_list: regular_packages_list.append(package) else: changes_packages_list.append(package_list) else: regular_packages_list.append(arg) if changes_packages_list: for package_list in changes_packages_list: process_packages(package_list) if regular_packages_list: if settings.single_packages: for package in regular_packages_list: process_packages([package]) else: process_packages(regular_packages_list) logging.info("PASS: All tests.") logging.info("piuparts run ends.") if __name__ == "__main__": try: main() except KeyboardInterrupt: print '' print 'Piuparts interrupted by the user, exiting...' panic(1) sys.exit(1) # vi:set et ts=4 sw=4 : �������piuparts/piuparts_slave_join.8.txt������������������������������������������������������������������0000644�0000000�0000000�00000002015�12762537303�014610� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts_slave_join(8) ====================== :doctype: manpage :revdate: 2013-05-27 NAME ---- piuparts_slave_join - join a running piuparts-slave screen session SYNOPSIS -------- *piuparts_slave_join* DESCRIPTION ----------- *piuparts_slave_join* joins a running piuparts-slave session in screen. OPTIONS ------- There are no options to this command. ENVIRONMENT ----------- Running piuparts in master-slave mode requires configuration in _/etc/piuparts_. NOTES ----- Make sure to also read */usr/share/doc/piuparts-master/README_server.txt*. See *screen*(1) for full instructions on how to navigate between displays. In short: *Ctrl-A <n>*:: Navigates to display <n>. Slaves are found at screens 1 through 'slave-count', defined in the piuparts configuration file. *Ctrl-A d*:: Detach from *screen*. The session can be re-established with *piuparts-slave-join*. SEE ALSO -------- *piuparts*(1), *piuparts_slave_run*(8), *screen*(1) AUTHOR ------ Holger Levsen (holger@layer-acht.org) // vim: set filetype=asciidoc: �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piuparts_slave_run.8.txt�������������������������������������������������������������������0000644�0000000�0000000�00000003170�13061276553�014460� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts_slave_run(8) ===================== :doctype: manpage :revdate: 2017-03-06 NAME ---- piuparts_slave_run - start up piuparts-slave instance(s) in screen SYNOPSIS -------- *piuparts_slave_run* ['SLAVE-NUMBERS'] DESCRIPTION ----------- *piuparts_slave_run* runs piuparts-slave instance(s) in *screen* which run *piuparts* tests which test whether Debian packages handle installation, upgrading, and removal correctly. By default, *piuparts_slave_run* will start the number of slaves specified by the 'slave-count' parameter in the piuparts.conf file. If 'slave-count' is not defined, a single slave will be started. If a single argument is supplied to *piuparts_slave_run*, consisting of a space-separated list of numbers between 1 and 'slave-count', only those slaves will be started. OPTIONS ------- There are no options to this command currently. Note however that *this interface is under development* and may change. ENVIRONMENT ----------- Running piuparts in master-slave mode requires configuration in _/etc/piuparts_. BUGS ---- *piuparts_slave_run* should be able to run slaves in different ways: 1. all, in a new screen (only this is implemented at the moment) 2. a specific one, in a specified screen slot in a (running) screen 3. append one to a (running) screen using a "random" slot 4. start slave in the current directory in the current shell in foreground. NOTES ----- Make sure to also read */usr/share/doc/piuparts-master/README_server.txt*. SEE ALSO -------- *piuparts*(1), *piuparts_slave_join*(8), *piuparts_slave_stop*(8), *screen*(1) AUTHOR ------ Holger Levsen (holger@layer-acht.org) // vim: set filetype=asciidoc: ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piuparts_slave_stop.8.txt������������������������������������������������������������������0000644�0000000�0000000�00000001634�13061276553�014644� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts_slave_stop(8) ====================== :doctype: manpage :revdate: 2017-03-06 NAME ---- piuparts_slave_stop - stop all running piuparts-slaves on a host. SYNOPSIS -------- *piuparts_slave_stop* DESCRIPTION ----------- *piuparts_slave_stop* gracefully stops all running piuparts-slaves sessions on a host. Gracefully means that it asks the slaves to stop running and will patiently wait until all the piuparts-slaves have done so. If more immediate action is required, use *piuparts_slave_join* and manual force instead. OPTIONS ------- There are no options to this command. ENVIRONMENT ----------- Running piuparts in master-slave mode requires configuration in _/etc/piuparts_. NOTES ----- Make sure to also read */usr/share/doc/piuparts-master/README_server.txt*. SEE ALSO -------- *piuparts*(1), *piuparts_slave_join*(8) AUTHOR ------ Holger Levsen (holger@layer-acht.org) // vim: set filetype=asciidoc: ����������������������������������������������������������������������������������������������������piuparts/piupartslib/�������������������������������������������������������������������������������0000755�0000000�0000000�00000000000�13061276553�012161� 5����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piupartslib/__init__.py��������������������������������������������������������������������0000644�0000000�0000000�00000006535�12762537303�014303� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# -*- coding: utf-8 -*- # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright © 2013-2015 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA import bz2 import lzma import zlib import urllib2 import conf import dependencyparser import packagesdb class DecompressedStream(): def __init__(self, fileobj, decompressor=None): self._input = fileobj self._decompressor = decompressor self._buffer = "" self._line_buffer = [] self._i = 0 self._end = 0 def _refill(self): if self._input is None: return False while True: # repeat until decompressor yields some output or input is exhausted chunk = self._input.read(4096) if not chunk: self.close() return False if self._decompressor: chunk = self._decompressor.decompress(chunk) self._buffer = self._buffer + chunk if chunk: return True def readline(self): while not self._i < self._end: self._i = self._end = 0 self._line_buffer = None empty = not self._refill() if not self._buffer: break self._line_buffer = self._buffer.splitlines(True) self._end = len(self._line_buffer) self._buffer = "" if not self._line_buffer[-1].endswith("\n") and not empty: self._buffer = self._line_buffer[-1] self._end = self._end - 1 if self._i < self._end: self._i = self._i + 1 return self._line_buffer[self._i - 1] return "" def close(self): if self._input: self._input.close() self._input = self._decompressor = None def open_packages_url(url): """Open a Packages.bz2 file pointed to by a URL""" socket = None for ext in ['.xz', '.bz2', '.gz', '']: try: socket = urllib2.urlopen(url + ext) except urllib2.HTTPError as httperror: pass else: break if socket is None: raise httperror url = socket.geturl() if ext == '.bz2': decompressor = bz2.BZ2Decompressor() decompressed = DecompressedStream(socket, decompressor) elif ext == '.gz': decompressor = zlib.decompressobj(16 + zlib.MAX_WBITS) decompressed = DecompressedStream(socket, decompressor) elif ext == '.xz': decompressor = lzma.LZMADecompressor() decompressed = DecompressedStream(socket, decompressor) elif ext == '': decompressed = socket else: raise ext return (url, decompressed) # vi:set et ts=4 sw=4 : �������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piupartslib/conf.py������������������������������������������������������������������������0000644�0000000�0000000�00000021405�13061276553�013462� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# -*- coding: utf-8 -*- # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright © 2012-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA # # NOTA BENE: This module MUST NOT use the logging module, since it can and # will be used before logging is set up (because logging can be configured # in a configuration file, that's why). # import ConfigParser import UserDict import subprocess import collections import re import distro_info from functools import reduce class MissingSection(Exception): def __init__(self, filename, section): self.args = "Section %s not defined in configuration file %s" % \ (section, filename), class MissingMandatorySetting(Exception): def __init__(self, filename, key): self.args = "Value for %s not set in configuration file %s" % \ (key, filename), class Config(UserDict.UserDict): def __init__(self, section, defaults, mandatory=[], defaults_section=None): UserDict.UserDict.__init__(self) self._section = section self._defaults_section = defaults_section for key, value in defaults.iteritems(): self[key] = value self._mandatory = mandatory def read(self, filename): cp = ConfigParser.ConfigParser() cp.read(filename) if not cp.has_section(self._section): raise MissingSection(filename, self._section) for key in self.keys(): if cp.has_option(self._section, key): self[key] = cp.get(self._section, key) elif self._defaults_section and cp.has_option(self._defaults_section, key): self[key] = cp.get(self._defaults_section, key) elif key in self._mandatory: raise MissingMandatorySetting(filename, key) def get_mirror(self, distro=None): if self["mirror"] is not None: return self["mirror"] return "http://deb.debian.org/debian" def get_distros(self): if self["upgrade-test-distros"] is not None: return self["upgrade-test-distros"].split() return [] def get_distro(self): if self["distro"]: return self["distro"] distros = self.get_distros() if distros: return distros[-1] return None def get_start_distro(self): distros = self.get_distros() if distros: return distros[0] return self["distro"] def get_final_distro(self): distros = self.get_distros() if distros: return distros[-1] return self["distro"] def _get_distmap(self): debdist = distro_info.DebianDistroInfo() # start with e.g. "sid" -> "unstable" distmap = collections.defaultdict(lambda: "unknown", [ (debdist.old(), "oldstable"), (debdist.devel(), "unstable"), (debdist.stable(), "stable"), (debdist.testing(), "testing"), ("experimental", "experimental"), ("rc", "experimental"), ]) # add mappings for e.g. "oldstable" -> "oldstable" distmap.update(dict([(val, val) for key, val in distmap.iteritems()])) # map e.g. "Debian6" -> "oldstable" where debdist.old(result="fullname") # currently returns 'Debian 6.0 "Squeeze"' dkey = lambda x: "Debian" + re.split('[ \.]', x(result="fullname"))[1] dfuncs = [debdist.old, debdist.stable, debdist.testing] distmap.update(dict([(dkey(x), distmap[x()]) for x in dfuncs])) return distmap def _map_distro(self, distro): distro_root = re.split("[\-\.]", distro)[0] distmap = self._get_distmap() return distmap[distro_root] def get_std_distro(self, distrolist=[]): if not distrolist: distrolist = [self.get_distro()] + self.get_distros() mappedlist = [self._map_distro(x) for x in distrolist] return reduce(lambda x, y: y if y != "unknown" else x, mappedlist) def get_area(self): if self["area"] is not None: return self["area"] return "main" def get_arch(self): if not self["arch"]: # Try to figure it out ourselves, using dpkg p = subprocess.Popen(["dpkg", "--print-architecture"], stdout=subprocess.PIPE) self["arch"] = p.stdout.read().rstrip() return self["arch"] class DistroConfig(UserDict.UserDict): def __init__(self, filename, mirror): UserDict.UserDict.__init__(self) self._mirror = mirror self._defaults = { "uri": None, "distribution": None, "components": None, "target-release": None, "depends": None, "candidates": None, } cp = ConfigParser.SafeConfigParser() cp.read(filename) for section in cp.sections(): self[section] = dict(self._defaults) for key in self._defaults.keys(): if cp.has_option(section, key): self[section][key] = cp.get(section, key) def get(self, section, key=None): if not section in self.keys(): self[section] = dict(self._defaults, distribution=section) if not key is None: return self[section][key] return self[section] def _is_virtual(self, distro): uri = self.get(distro, "uri") return uri is not None and uri == "None" def get_mirror(self, distro): if self._is_virtual(distro): distro = self._expand_depends(distro)[0] return self.get(distro, "uri") or self._mirror def get_distribution(self, distro): if self._is_virtual(distro): distro = self._expand_depends(distro)[0] return self.get(distro, "distribution") or distro def get_candidates(self, distro): return (self.get(distro, "candidates") or "").split() or [distro] def _get_packages_url(self, distro, area, arch): return "%s/dists/%s/%s/binary-%s/Packages" % ( self.get_mirror(distro), self.get_distribution(distro), area, arch) def get_packages_urls(self, distro, area, arch): return [self._get_packages_url(d, area, arch) for d in self.get_candidates(distro)] def _get_sources_url(self, distro, area): return "%s/dists/%s/%s/source/Sources" % ( self.get_mirror(distro), self.get_distribution(distro), area) def get_sources_urls(self, distro, area): return [self._get_sources_url(d, area) for d in self.get_candidates(distro)] def get_target_flags(self, distro): tr = self.get(distro, "target-release") if tr: return ["-t", tr] return [] def _expand_depends(self, distro, include_virtual=False): todo = [distro] done = [] seen = [] while todo: curr = todo[0] todo = todo[1:] if not curr in seen: seen.append(curr) todo = (self.get(curr, "depends") or "").split() + [curr] + todo elif not curr in done: if include_virtual or not self._is_virtual(curr): done.append(curr) assert(len(done) > 0) return done def get_deb_lines(self, distro, components): lines = [] for d in self._expand_depends(distro): for c in components: if self[d]["components"] is None or c in self[d]["components"].split(): lines.append("deb %s %s %s" % ( self.get_mirror(d), self.get_distribution(d), c)) return lines def get_basetgz(self, distro, arch): # look for the first base distribution for d in self._expand_depends(distro): if self.get(d, "depends"): next # skip partial distro return "%s_%s.tar.gz" % (self.get_distribution(d), arch) return None # vi:set et ts=4 sw=4 : �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piupartslib/dependencyparser.py������������������������������������������������������������0000644�0000000�0000000�00000025524�12567171321�016073� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# -*- coding: utf-8 -*- # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright © 2012 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """Parser for Debian package relationship strings This module contains the class DependencyParser, which parses Debian package relationship strings (e.g., the Depends header). The class raises the DependencySyntaxError exception on syntactic errors. The result uses SimpleDependency objects. Lars Wirzenius <liw@iki.fi> """ import re class DependencySyntaxError(Exception): """Syntax error in package dependency declaration""" def __init__(self, msg, cursor): self._msg = "Error: %s: %s (text at error: '%s', full text being parsed: '%s')" % \ (cursor.get_position(), msg, cursor.get_text(10), cursor.get_full_text()) def __str__(self): return self._msg def __repr__(self): return self._msg class _Cursor: """Store an input string and a movable location in it""" def __init__(self, input): self._input = input self._len = len(self._input) self._pos = 0 def skip_whitespace(self): while self._pos < self._len and self._input[self._pos].isspace(): self.next() def at_end(self): """Are we at the end of the input?""" self.skip_whitespace() return self._pos >= self._len def next(self): """Move to the next character""" if self._pos < self._len: self._pos += 1 def get_char(self): """Return current character, None if at end""" if self._pos >= self._len: return None else: return self._input[self._pos] def get_full_text(self): return self._input def get_text(self, length): """Return up to length characters from the current position""" if self._pos >= self._len: return "" else: return self._input[self._pos:self._pos + length] def match(self, regexp): """Match a regular expression against the current position The cursor is advanced by the length of the match, if any. """ m = regexp.match(self._input[self._pos:]) if m: self._pos += len(m.group()) return m def match_literal(self, literal): """Match a literal string against the current position. Return True and move position if there is a match, else return False. """ if self.get_text(len(literal)) == literal: self._pos += len(literal) return True else: return False def get_position(self): """Return current position, as string""" return "pos %d" % self._pos class SimpleDependency: """Express simple dependency towards another package""" def __init__(self, name, operator, version, arch): self.name = name self.operator = operator self.version = version self.arch = arch def __repr__(self): return "<DEP: %s, %s, %s, %s>" % (self.name, self.operator, self.version, self.arch) class DependencyParser: """Parse Debian package relationship strings Debian packages have a rich language for expressing their relationships. See the Debian Policy Manual, chapter 7 ("Declaring relationships between packages"). This Python module implements a parser for strings expressing such relationships. Syntax of dependency fields (Pre-Depends, Depends, Recommends, Suggests, Conflicts, Provides, Replaces, Enhances, Build-Depends, Build-Depends-Indep, Build-Conflicts, Build-Conflicts-Indep), in a BNF-like form: depends-field ::= EMPTY | dependency ("," dependency)* dependency ::= possible-dependency ("|" possible-dependency)* possible-dependency ::= package-name version-dependency? arch-restriction? version-dependency ::= "(" relative-operator version-number ")" relative-operator ::= "<<" | "<=" | "=" | ">=" | ">>" | "<" | ">" version-number ::= epoch? upstream-version debian-revision? arch-restriction ::= "[" arch-name arch-name* "]" | "[" "!" arch-name ("!" arch-name)* "]" package-name ::= alphanumeric name-char name-char* ":any"? epoch ::= integer ":" upstream-version ::= alphanumeric version-char* -- policy says "should start with digit", but not all packages do debian-revision ::= "-" debian-version-char debian-version-char* arch-name ::= alphanumeric alphanumeric* EMPTY ::= "" integer ::= digit digit* alphanumeric ::= "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" | digit digit ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" name-char ::= alphanumeric | "+" | "-" | "." | "_" version-char ::= alphanumeric | "." | "+" | "-" | ":" | "~" debian-version-char ::= alphanumeric | "." | "+" White space can occur between any tokens except inside package-name, version-number, or arch-name. Some of the headers restrict the syntax somewhat, e.g., Provides does not allow version-dependency, but this is not included in the syntax for simplicity. Note: Added "_" to name-char, because some packages (type-handling in particular) use Provides: headers with bogus package names. Note: Added upper case letters to name pattern, since it some of the Mozilla localization packages use or used them. """ def __init__(self, input_string): self._cursor = _Cursor(input_string) self._list = self._parse_dependencies() def get_dependencies(self): """Return parsed dependencies The result is a list of lists of SimpleDependency objects. Let's try that again. The result is a list of dependencies, corresponding to the comma-separated items in the dependency list. Each dependency is also a list, or SimpleDependency objects, representing alternative ways to fulfill the dependency; in other words, items separated by the vertical bar (|). For example, "foo, bar | foobar" would result in the following list: [[foo], [bar, foobar]]. """ return self._list def _parse_dependencies(self): vlist = [] dep = self._parse_dependency() while dep: vlist.append(dep) self._cursor.skip_whitespace() if self._cursor.at_end(): break if not self._cursor.match_literal(","): raise DependencySyntaxError("Expected comma", self._cursor) dep = self._parse_dependency() return vlist def _parse_dependency(self): vlist = [] dep = self._parse_possible_dependency() while dep: vlist.append(dep) self._cursor.skip_whitespace() if not self._cursor.match_literal("|"): break dep = self._parse_possible_dependency() return vlist def _parse_possible_dependency(self): name = self._parse_package_name() if not name: return None (op, version) = self._parse_version_dependency() arch = self._parse_arch_restriction() return SimpleDependency(name, op, version, arch) _name_pat = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9+._-]+") # The MultiArch spec supports an ":any" modifier. Loosen the # accepted arch's, to avoid crashing. _any_suffix_pat = re.compile(r":[a-zA-Z0-9-]+") def _parse_package_name(self): self._cursor.skip_whitespace() if self._cursor.at_end(): return None m = self._cursor.match(self._name_pat) if not m: raise DependencySyntaxError("Expected a package name", self._cursor) if self._cursor.match(self._any_suffix_pat): pass return m.group() _op_pat = re.compile(r"(<<|<=|=|>=|>>|<(?![<=])|>(?![>=]))") _version_pat = re.compile(r"(?P<epoch>\d+:)?" + r"(?P<upstream>[a-zA-Z0-9+][a-zA-Z0-9.+:~-]*)" + r"(?P<debian>-[a-zA-Z0-9.+]+)?") def _parse_version_dependency(self): self._cursor.skip_whitespace() if self._cursor.get_char() == "(": self._cursor.next() self._cursor.skip_whitespace() opm = self._cursor.match(self._op_pat) if not opm: raise DependencySyntaxError("Expected a version relation " + "operator", self._cursor) operator = opm.group() if operator == "<": operator = "<=" elif operator == ">": operator = ">=" self._cursor.skip_whitespace() verm = self._cursor.match(self._version_pat) if not verm: raise DependencySyntaxError("Expected a version number", self._cursor) self._cursor.skip_whitespace() if self._cursor.get_char() != ")": raise DependencySyntaxError("Expected ')'", self._cursor) self._cursor.next() return opm.group(), verm.group() else: return None, None _arch_pat = re.compile(r"!?[a-zA-Z0-9-]+") def _parse_arch_restriction(self): self._cursor.skip_whitespace() if self._cursor.get_char() == "[": self._cursor.next() vlist = [] while True: self._cursor.skip_whitespace() if self._cursor.get_char() == "]": self._cursor.next() break m = self._cursor.match(self._arch_pat) if not m: raise DependencySyntaxError("Expected architecture name", self._cursor) vlist.append(m.group()) return vlist else: return None # vi:set et ts=4 sw=4 : ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piupartslib/dwke.py������������������������������������������������������������������������0000644�0000000�0000000�00000014763�12762537303�013500� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# -*- coding: utf-8 -*- # Copyright 2013 David Steele (dsteele@gmail.com) # Copyright © 2014 Andreas Beckmann (anbe@debian.org) # # This file is part of Piuparts # # Piuparts is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # Piuparts is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, see <http://www.gnu.org/licenses/>. # dwke.py is used by master-bin/detect_well_known_errors.py import os import logging import re from collections import namedtuple KPR_EXT = '.kpr' BUG_EXT = '.bug' LOG_EXT = '.log' class Problem(): """ Encapsulate a particular known problem """ def __init__(self, probpath): """probpath is the path to the problem definition file""" self.probpath = probpath self.name = os.path.basename(probpath) self.short_name = os.path.splitext(self.name)[0] self.tags_are_valid = True self.required_tags = ["PATTERN", "WHERE", "ISSUE", "HEADER", "HELPTEXT"] self.optional_tags = ["EXCLUDE_PATTERN", "EXPLAIN", "PRIORITY"] self.init_problem() for tag in self.required_tags: if not tag in self.__dict__: self.tags_are_valid = False if "PATTERN" in self.__dict__: self.inc_re = re.compile(self.PATTERN) else: self.inc_re = None if "EXCLUDE_PATTERN" in self.__dict__: self.exc_re = re.compile(self.EXCLUDE_PATTERN) else: self.exc_re = None def valid(self): return self.tags_are_valid def init_problem(self): """Load problem file parameters (HELPTEXT="foo" -> self.HELPTEXT)""" pb = open(self.probpath, 'r') probbody = pb.read() pb.close() tagged = re.sub("^([A-Z_]+=)", "<hdr>\g<0>", probbody, 0, re.MULTILINE) for chub in re.split('<hdr>', tagged)[1:]: (name, value) = re.split("=", chub, 1, re.MULTILINE) while value[-1] == '\n': value = value[:-1] if re.search("^\'.+\'$", value, re.MULTILINE | re.DOTALL) \ or re.search('^\".+\"$', value, re.MULTILINE | re.DOTALL): value = value[1:-1] if name in self.required_tags or name in self.optional_tags: self.__dict__[name] = value else: self.tags_are_valid = False self.WHERE = self.WHERE.split(" ") def has_problem(self, logbody, where): """Does the log text 'logbody' contain this known problem?""" if where in self.WHERE: if self.inc_re.search(logbody, re.MULTILINE): for line in logbody.splitlines(): if self.inc_re.search(line): if self.exc_re is None \ or not self.exc_re.search(line): return True return False def get_command(self): cmd = "grep -E \"%s\"" % self.PATTERN if "EXCLUDE_PATTERN" in self.__dict__: cmd += " | grep -v -E \"%s\"" % self.EXCLUDE_PATTERN return cmd class FailureManager(): """Class to track known failures encountered, by package, where (e.g. 'fail'), and known problem type""" def __init__(self, logdict): """logdict is {pkgspec: fulllogpath} across all log files""" self.logdict = logdict self.failures = [] self.load_failures() def load_failures(self): """Collect failures across all kpr files, as named tuples""" for pkgspec in self.logdict: logpath = self.logdict[pkgspec] try: kp = open(get_kpr_path(logpath), 'r') for line in kp.readlines(): (where, problem) = self.parse_kpr_line(line) self.failures.append(make_failure(where, problem, pkgspec)) kp.close() except IOError: logging.error("Error processing %s" % get_kpr_path(logpath)) def parse_kpr_line(self, line): """Parse a line in a kpr file into where (e.g. 'pass') and problem name""" m = re.search("^([a-z]+)/.+ (.+)$", line) return (m.group(1), m.group(2)) def sort_by_path(self): self.failures.sort(key=lambda x: self.logdict[x.pkgspec]) def sort_by_bugged_and_rdeps(self, pkgsdb): self.pkgsdb = pkgsdb def keyfunc(x, pkgsdb=self.pkgsdb, logdict=self.logdict): rdeps = pkgsdb.rrdep_count(get_pkg(x.pkgspec)) is_failed = get_where(logdict[x.pkgspec]) == "fail" return (not is_failed, -rdeps, logdict[x.pkgspec]) self.failures.sort(key=keyfunc) def filtered(self, problem): return [x for x in self.failures if problem == x.problem] def make_failure(where, problem, pkgspec): return (namedtuple('Failure', 'where problem pkgspec')(where, problem, pkgspec)) def get_where(logpath): """Convert a path to a log file to the 'where' component (e.g. 'pass')""" return logpath.split('/')[-2] def replace_ext(fpath, newext): basename = os.path.splitext(os.path.split(fpath)[1])[0] return '/'.join(fpath.split('/')[:-1] + [basename + newext]) def get_pkg(pkgspec): return pkgspec.split('_')[0] def get_kpr_path(logpath): """Return the kpr file path for a particular log path""" return replace_ext(logpath, KPR_EXT) def get_file_dict(workdirs, ext): """For files in [workdirs] with extension 'ext', create a dict of <pkgname>_<version>: <path>""" filedict = {} for dir in workdirs: for fl in os.listdir(dir): if os.path.splitext(fl)[1] == ext: filedict[os.path.splitext(os.path.basename(fl))[0]] \ = os.path.join(dir, fl) return filedict def create_problem_list(pdir): plist = [] for pfile in [x for x in sorted(os.listdir(pdir)) if x.endswith(".conf")]: prob = Problem(os.path.join(pdir, pfile)) if prob.valid(): plist.append(prob) else: logging.error("Keyword error in %s - skipping" % pfile) return plist # vi:set et ts=4 sw=4 : �������������piuparts/piupartslib/packagesdb.py������������������������������������������������������������������0000644�0000000�0000000�00000105151�12762537303�014622� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# -*- coding: utf-8 -*- # Copyright 2005 Lars Wirzenius (liw@iki.fi) # Copyright © 2011-2015 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """Packages database for distributed piuparts processing This module contains tools for keeping track of which packages have been tested, the test results, and for determining what to test next. Lars Wirzenius <liw@iki.fi> """ import logging import os import random import stat import tempfile import time import UserDict import apt_pkg import piupartslib from piupartslib.dependencyparser import DependencyParser apt_pkg.init_system() def rfc822_like_header_parse(input): headers = [] while True: line = input.readline() if not line or line in ["\r\n", "\n"]: break if headers and line and line[0].isspace(): headers[-1] = headers[-1] + line else: headers.append(line) return headers class Package(UserDict.UserDict): def __init__(self, headers): UserDict.UserDict.__init__(self) self.headers = headers for header in headers: name, value = header.split(":", 1) self[name.strip()] = value.strip() self._parsed_deps = {} self._parsed_alt_deps = {} self.rrdep_cnt = None self.block_cnt = None self.waiting_cnt = None self.rdep_chain_len = None def _parse_dependencies(self, header_name): if header_name in self._parsed_deps: depends = self._parsed_deps[header_name] else: parser = DependencyParser(self[header_name]) depends = parser.get_dependencies() depends = [alternatives[0].name for alternatives in depends] self._parsed_deps[header_name] = depends return depends def _parse_alternative_dependencies(self, header_name): if header_name in self._parsed_alt_deps: depends = self._parsed_alt_deps[header_name] else: parser = DependencyParser(self[header_name]) depends = parser.get_dependencies() depends = [[alt.name for alt in alternatives] for alternatives in depends] self._parsed_alt_deps[header_name] = depends return depends # first alternative only - [package_name...] def dependencies(self): vlist = [] for header in ["Depends", "Pre-Depends"]: if header in self: vlist += self._parse_dependencies(header) return vlist # all alternatives - [[package_name...]...] def all_dependencies(self, header_name=None): headers = ["Depends", "Pre-Depends"] if header_name is not None: headers = [header_name] vlist = [] for header in headers: if header in self: vlist += self._parse_alternative_dependencies(header) return vlist def prefer_alt_depends(self, header_name, dep_idx, dep): if header_name in self: if header_name not in self._parsed_deps: self._parse_dependencies(header_name) if self._parsed_deps[header_name][dep_idx]: self._parsed_deps[header_name][dep_idx] = dep def provides(self): vlist = [] for header in ["Provides"]: if header in self: vlist += self._parse_dependencies(header) return vlist def dump(self, output_file): output_file.write("".join(self.headers)) class PackagesFile(UserDict.UserDict): def __init__(self): UserDict.UserDict.__init__(self) self._urllist = [] def load_packages_urls(self, urls, restrict_packages=None): for url in urls: logging.debug("Opening %s.*" % url) (url, stream) = piupartslib.open_packages_url(url) logging.debug("Fetching %s" % url) self._read_file(stream, restrict_packages=restrict_packages) stream.close() self._urllist.append(url) def _read_file(self, input, restrict_packages=None): """Parse a Packages file and add its packages to us-the-dict""" while True: headers = rfc822_like_header_parse(input) if not headers: break p = Package(headers) if p["Package"] in self: q = self[p["Package"]] if apt_pkg.version_compare(p["Version"], q["Version"]) <= 0: # there is already a newer version continue if restrict_packages is not None: if p["Package"] not in restrict_packages: # unwanted package continue self[p["Package"]] = p def get_urls(self): return self._urllist class LogDB: def exists(self, pathname): try: cache = self.exists_cache except AttributeError: self.exists_cache = {} cache = self.exists_cache if pathname not in cache: cache[pathname] = os.path.exists(pathname) return cache[pathname] def _evict(self, pathname): try: cache = self.exists_cache if pathname in cache: del cache[pathname] except AttributeError: pass def bulk_load_dir(self, dirname): try: cache = self.exists_cache except AttributeError: self.exists_cache = {} cache = self.exists_cache for basename in os.listdir(dirname): if basename.endswith(".log"): cache[os.path.join(dirname, basename)] = True def open_file(self, pathname, mode): return file(pathname, mode) def remove_file(self, pathname): os.remove(pathname) def _log_name(self, package, version): return "%s_%s.log" % (package, version) def log_exists(self, package, subdirs): log_name = self._log_name(package["Package"], package["Version"]) for subdir in subdirs: if self.exists(os.path.join(subdir, log_name)): return True return False def create(self, subdir, package, version, contents): (fd, temp_name) = tempfile.mkstemp(dir=subdir) if os.write(fd, contents) != len(contents): raise Exception("Partial write?") os.close(fd) # tempfile.mkstemp sets the file mode to be readable only by owner. # Let's make it follow the umask. umask = os.umask(0) os.umask(umask) os.chmod(temp_name, 0o666 & ~umask) full_name = os.path.join(subdir, self._log_name(package, version)) try: os.link(temp_name, full_name) except OSError: return False finally: os.remove(temp_name) self._evict(full_name) return True def remove(self, subdir, package, version): full_name = os.path.join(subdir, self._log_name(package, version)) if self.exists(full_name): self.remove_file(full_name) self._evict(full_name) def stat(self, subdir, package, version): full_name = os.path.join(subdir, self._log_name(package, version)) return os.stat(full_name) class LogfileExists(Exception): def __init__(self, path, package, version): self.args = (path, package, version) class PackagesDB: # these packages are uses as dependencies but are only available # from foreign architectures # HACK: this hardcoded list should be moved to some data file _foreign_packages = { "ia32-libs-i386": "i386", "ia32-libs-gtk-i386": "i386", "libnss-mdns-i386": "i386", } # keep in sync with piuparts-report.py: emphasize_reason() # FIXME: can we reorder this list or remove entries without breaking the counts.txt for the plot? _states = [ "successfully-tested", "failed-testing", "cannot-be-tested", "essential-required", # obsolete "waiting-to-be-tested", "waiting-for-dependency-to-be-tested", "dependency-failed-testing", "dependency-cannot-be-tested", "dependency-does-not-exist", "circular-dependency", # obsolete "unknown", "unknown-preferred-alternative", # obsolete "no-dependency-from-alternatives-exists", # obsolete # "foreign:*", # can only happen as query result for a dependency # "does-not-exist", # can only happen as query result for a dependency ] _good_states = [ "successfully-tested", "essential-required", ] + ["foreign:%s" % arch for arch in set(_foreign_packages.values())] _obsolete_states = [ "essential-required", "circular-dependency", "unknown-preferred-alternative", "no-dependency-from-alternatives-exists", ] _propagate_error_state = { "failed-testing": "dependency-failed-testing", "cannot-be-tested": "dependency-cannot-be-tested", "dependency-failed-testing": "dependency-failed-testing", "dependency-cannot-be-tested": "dependency-cannot-be-tested", "dependency-does-not-exist": "dependency-cannot-be-tested", "does-not-exist": "dependency-does-not-exist", } _propagate_waiting_state = { "waiting-to-be-tested": "waiting-for-dependency-to-be-tested", "waiting-for-dependency-to-be-tested": "waiting-for-dependency-to-be-tested", } def __init__(self, logdb=None, prefix=None): self.prefix = prefix self._packages_files = [] self._ready_for_testing = None self._logdb = logdb or LogDB() self._packages = None self._in_state = None self._package_state = {} self._dependency_databases = [] self._recycle_mode = False self._candidates_for_testing = None self._rdeps = None self.set_subdirs(ok="pass", fail="fail", evil="untestable", reserved="reserved", morefail=["bugged", "affected"], recycle="recycle") self.create_subdirs() def set_subdirs(self, ok=None, fail=None, evil=None, reserved=None, morefail=None, recycle=None): # Prefix all the subdirs with the prefix if self.prefix: pformat = self.prefix + "/%s" else: pformat = "%s" self._submissions = pformat % "submissions.txt" self._all = [] if ok: self._ok = pformat % ok self._all.append(self._ok) if fail: self._fail = pformat % fail self._all.append(self._fail) if evil: self._evil = pformat % evil self._all.append(self._evil) if reserved: self._reserved = pformat % reserved self._all.append(self._reserved) if morefail: self._morefail = [pformat % s for s in morefail] self._all.extend(self._morefail) if recycle: self._recycle = pformat % recycle self._all.append(self._recycle) def create_subdirs(self): for sdir in self._all: if not os.path.exists(sdir): os.makedirs(sdir) def enable_recycling(self): if self._recycle_mode: return True if self._packages is not None: logging.info("too late for recycling") return False for basename in os.listdir(self._recycle): if basename.endswith(".log"): self._recycle_mode = True return True logging.info("nothing to recycle") return False def get_mtime(self): return max([os.path.getmtime(sdir) for sdir in self._all]) def load_packages_urls(self, urls): pf = PackagesFile() pf.load_packages_urls(urls) self._packages_files.append(pf) self._packages = None def load_alternate_versions_from_packages_urls(self, urls): # take version numbers (or None) from alternate URLs pf2 = PackagesFile() pf2.load_packages_urls(urls) for package in self.get_all_packages(): if package["Package"] in pf2: package["Version"] = pf2[package["Package"]]["Version"] else: package["Version"] = "None" def get_urls(self): urls = [] for pf in self._packages_files: urls.extend(pf.get_urls()) return urls def set_dependency_databases(self, dependency_databases=[]): self._dependency_databases = list(dependency_databases) def _find_all_packages(self): if self._packages is None: self._packages = {} self._virtual_packages = {} for pf in self._packages_files: for p in pf.values(): self._packages[p["Package"]] = p for p in self._packages.values(): for provided in p.provides(): if provided != p["Package"]: if provided not in self._virtual_packages: self._virtual_packages[provided] = [] self._virtual_packages[provided].append(p["Package"]) def _get_recursive_dependencies(self, package): assert self._packages is not None deps = [] more = package.dependencies() while more: dep = more[0] more = more[1:] if dep not in deps: deps.append(dep) dep_pkg = self.get_package(dep, recurse=True, resolve_virtual=True) if dep_pkg is not None: more += dep_pkg.dependencies() return deps def _get_dependency_cycle(self, package_name): deps = [] circular = [] more = [package_name] while more: dep = more[0] more = more[1:] if dep not in deps: deps.append(dep) dep_pkg = self.get_package(dep, recurse=True, resolve_virtual=True) if dep_pkg is not None and package_name in self._get_recursive_dependencies(dep_pkg): circular.append(dep) more += dep_pkg.dependencies() return circular def _is_successfully_tested(self, package): # a pass/ log exists but no corresponding recycle/ log exists if self._logdb.log_exists(package, [self._ok]): if not (self._recycle_mode and self._logdb.log_exists(package, [self._recycle])): return True return False def _lookup_package_state(self, package, use_cached_success): if self._recycle_mode and self._logdb.log_exists(package, [self._recycle]): return "unknown" if self._logdb.log_exists(package, [self._ok]): success = True if not use_cached_success: # if a pass/ log exists but any dependency may be not # trivially satisfiable do not skip dependency resolution for dep in package.dependencies(): if not self.get_package(dep, resolve_virtual=True): success = False break if success: return "successfully-tested" if self._logdb.log_exists(package, [self._fail] + self._morefail): return "failed-testing" if self._logdb.log_exists(package, [self._evil]): return "cannot-be-tested" return "unknown" def _compute_package_state(self, package): # First attempt to resolve (still-unresolved) multiple alternative depends # Definitely sub-optimal, but improvement over blindly selecting first one # Select the first alternative in the highest of the following states: # 1) "essential-required" # 2) "successfully-tested" # 3) "waiting-to-be-tested" / "waiting-for-dependency-to-be-tested" # 4) "unknown" (retry later) # and update the preferred alternative of that dependency. # If no alternative is in any of these states we retry later ("unknown") # or set "dependency-does-not-exist". # # Problems: # a) We will test and fail when >=1 "successfully-tested" but another # that failed is selected by apt during test run # b) We may report a status of "waiting-for-dependency-to-be-tested" # instead of "waiting-to-be-tested" depending on the order the # package states get resolved. for header in ["Depends", "Pre-Depends"]: alt_deps = package.all_dependencies(header) for d in range(len(alt_deps)): if len(alt_deps[d]) > 1: alt_found = 0 prefer_alt_score = -1 prefer_alt = None for alternative in alt_deps[d]: altdep_state = self.get_package_state(alternative) if altdep_state != "does-not-exist": alt_found += 1 if prefer_alt_score < 3 and altdep_state == "essential-required": prefer_alt = alternative prefer_alt_score = 3 elif prefer_alt_score < 2 and altdep_state == "successfully-tested": prefer_alt = alternative prefer_alt_score = 2 elif prefer_alt_score < 1 and \ altdep_state in ["waiting-to-be-tested", "waiting-for-dependency-to-be-tested"]: prefer_alt = alternative prefer_alt_score = 1 elif prefer_alt_score < 0 and altdep_state == "unknown": prefer_alt = alternative prefer_alt_score = 0 if alt_found == 0: return "dependency-does-not-exist" if prefer_alt_score >= 0: package.prefer_alt_depends(header, d, prefer_alt) dep_states = [(dep, self.get_best_package_state(dep)) for dep in package.dependencies()] for dep, dep_state in dep_states: if dep_state in self._propagate_error_state: return self._propagate_error_state[dep_state] testable = True for dep, dep_state in dep_states: if dep_state not in self._good_states: testable = False break if testable: if self._is_successfully_tested(package): return "successfully-tested" return "waiting-to-be-tested" # treat circular-dependencies as testable (for the part of the circle) circular_deps = self._get_dependency_cycle(package["Package"]) if package["Package"] in circular_deps: testable = True for dep, dep_state in dep_states: if dep in circular_deps: # allow any non-error dep_state on the cycle for testing # (error states are handled by the error propagation above) pass elif dep_state not in self._good_states: # non-circular deps must have passed before testing circular deps testable = False break if testable: if self._is_successfully_tested(package): return "successfully-tested" return "waiting-to-be-tested" for dep, dep_state in dep_states: if dep_state in self._propagate_waiting_state: return self._propagate_waiting_state[dep_state] return "unknown" def _compute_package_states(self, use_cached_success=False): if self._in_state is not None: return self._stamp = time.time() for subdir in self._all: self._logdb.bulk_load_dir(subdir) self._find_all_packages() self._package_state = {} self._in_state = {} for state in self._states: self._in_state[state] = [] todo = [] for package_name, package in self._packages.iteritems(): state = self._lookup_package_state(package, use_cached_success) assert state in self._states self._package_state[package_name] = state if state == "unknown": todo.append(package_name) else: self._in_state[state].append(package_name) for db in self._dependency_databases: db._compute_package_states(use_cached_success=True) while todo: package_names = todo todo = [] done = [] for package_name in package_names: if self._package_state[package_name] == "unknown": state = self._compute_package_state(self._packages[package_name]) assert state in self._states if state == "unknown": todo.append(package_name) else: self._in_state[state].append(package_name) self._package_state[package_name] = state done.append(package) if not done: # If we didn't do anything this time, we sure aren't going # to do anything the next time either. break self._in_state["unknown"] = todo for state in self._states: self._in_state[state].sort() def get_states(self): return self._states def get_active_states(self): return [x for x in self._states if not x in self._obsolete_states] def get_error_states(self): return [x for x in self._propagate_error_state.keys() if x in self._states] def get_waiting_states(self): return [x for x in self._propagate_waiting_state.keys() if x in self._states] def get_pkg_names_in_state(self, state): self._compute_package_states() return set(self._in_state[state]) def has_package(self, name): self._find_all_packages() return name in self._packages def get_package(self, name, recurse=False, resolve_virtual=False): self._find_all_packages() if name in self._packages: return self._packages[name] if recurse: for db in self._dependency_databases: if db.has_package(name): return db.get_package(name) if resolve_virtual: providers = self.get_providers(name, recurse=recurse) if providers: return self.get_package(providers[0], recurse=recurse, resolve_virtual=False) return None def get_version(self, name): self._find_all_packages() if name in self._packages: return self._packages[name]["Version"] return None def get_source(self, name): self._find_all_packages() if name in self._packages: return self.get_control_header(name, "Source") return None def get_providers(self, name, recurse=True): self._find_all_packages() providers = [] if name in self._virtual_packages: providers.extend(self._virtual_packages[name]) if recurse: for db in self._dependency_databases: providers.extend(db.get_providers(name, recurse=False)) return providers def get_all_packages(self): self._find_all_packages() return self._packages.values() def get_all_package_names(self): self._find_all_packages() return self._packages.keys() def get_control_header(self, package_name, header): self._find_all_packages() if header == "Source": # binary packages build from the source package with the same name # don't have a Source header, so let's try: try: _source = self._packages[package_name][header] # for binNMU the Source header in Packages files holds the version # too, so we need to chop it of: if " " in _source: source, version = _source.split(" ") else: source = _source except: source = self._packages[package_name]["Package"] return source elif header == "Uploaders": # not all (source) packages have an Uploaders header uploaders = "" try: uploaders = self._packages[package_name][header] except: pass return uploaders else: return self._packages[package_name][header] def get_package_state(self, package_name, resolve_virtual=True, recurse=True): self._compute_package_states() if package_name in self._package_state: return self._package_state[package_name] if package_name in self._virtual_packages: if resolve_virtual: provider = self._virtual_packages[package_name][0] return self._package_state[provider] else: return "virtual" if recurse: for db in self._dependency_databases: state = db.get_package_state(package_name, resolve_virtual=resolve_virtual, recurse=False) if state != "does-not-exist": return state if package_name in self._foreign_packages: return "foreign:%s" % self._foreign_packages[package_name] return "does-not-exist" def get_best_package_state(self, package_name, resolve_virtual=True, recurse=True): package_state = self.get_package_state(package_name, resolve_virtual=resolve_virtual, recurse=recurse) if package_state in self._good_states: return package_state providers = [] if resolve_virtual: providers = self.get_providers(package_name, recurse=recurse) if not providers: return package_state states = [self.get_package_state(name, resolve_virtual=False, recurse=recurse) for name in [package_name] + providers] for state in self._good_states + self._propagate_waiting_state.keys() + self._propagate_error_state.keys(): if state in states: return state return package_state def _get_package_weight(self, p): # compute the priority of a package that needs testing # result will be used as a reverse sorting key, so higher is earlier waiting_count = self.waiting_count(p["Package"]) rdep_chain_len = self.rdep_chain_len(p["Package"]) if not self._recycle_mode: return ( min(rdep_chain_len, waiting_count), waiting_count, ) try: statobj = self._logdb.stat(self._recycle, p["Package"], p["Version"]) ctime = statobj[stat.ST_CTIME] # last inode modification = time of linking into recycle/ mtime = statobj[stat.ST_MTIME] except OSError: ctime = 0 mtime = 0 return ( min(rdep_chain_len, waiting_count), waiting_count, not self._logdb.log_exists(p, [self._ok]), # prefer problematic logs -ctime / 3600, # prefer older, at 1 hour granularity to allow randomization -mtime / 3600, # prefer older, at 1 hour granularity to allow randomization ) def _find_packages_ready_for_testing(self): if self._candidates_for_testing is None: self._candidates_for_testing = [self.get_package(pn) for pn in self.get_pkg_names_in_state("waiting-to-be-tested")] self._candidates_for_testing = [p for p in self._candidates_for_testing if not self._logdb.log_exists(p, [self._reserved]) or self._logdb.log_exists(p, [self._recycle])] if len(self._candidates_for_testing) > 1: tuples = [(self._get_package_weight(p), random.random(), p) for p in self._candidates_for_testing] self._candidates_for_testing = [x[-1] for x in sorted(tuples, reverse=True)] return self._candidates_for_testing[:] def _remove_unavailable_candidate(self, p): self._candidates_for_testing.remove(p) def reserve_package(self): all_but_recycle = [x for x in self._all if x != self._recycle] for p in self._find_packages_ready_for_testing(): if self._recycle_mode and self._logdb.log_exists(p, [self._recycle]): for vdir in all_but_recycle: if self._logdb.log_exists(p, [vdir]): self._logdb.remove(vdir, p["Package"], p["Version"]) logging.info("Recycled %s %s %s" % (vdir, p["Package"], p["Version"])) if self._logdb.log_exists(p, all_but_recycle): self._remove_unavailable_candidate(p) continue if self._logdb.log_exists(p, [self._recycle]): self._logdb.remove(self._recycle, p["Package"], p["Version"]) if self._logdb.create(self._reserved, p["Package"], p["Version"], ""): return p return None def _check_for_acceptability_as_filename(self, str): if "/" in str: raise Exception("'/' in (partial) filename: %s" % str) def _record_submission(self, category, package, version): with open(self._submissions, "a") as submissions: submissions.write("%d %s %s %s\n" % (time.time(), category, package, version)) def unreserve_package(self, package, version): self._check_for_acceptability_as_filename(package) self._check_for_acceptability_as_filename(version) self._logdb.remove(self._reserved, package, version) def pass_package(self, package, version, log): self._check_for_acceptability_as_filename(package) self._check_for_acceptability_as_filename(version) if self._logdb.create(self._ok, package, version, log): self._logdb.remove(self._reserved, package, version) self._record_submission("pass", package, version) else: raise LogfileExists(self._ok, package, version) def fail_package(self, package, version, log): self._check_for_acceptability_as_filename(package) self._check_for_acceptability_as_filename(version) if self._logdb.create(self._fail, package, version, log): self._logdb.remove(self._reserved, package, version) self._record_submission("fail", package, version) else: raise LogfileExists(self._fail, package, version) def make_package_untestable(self, package, version, log): self._check_for_acceptability_as_filename(package) self._check_for_acceptability_as_filename(version) if self._logdb.create(self._evil, package, version, log): self._logdb.remove(self._reserved, package, version) self._record_submission("untestable", package, version) else: raise LogfileExists(self._evil, package, version) def _get_rdep_dict(self): """Return dict of one-level reverse dependencies by package""" if self._rdeps is None: self._rdeps = {} for pkg_name in self.get_all_package_names(): for dep in self.get_package(pkg_name).dependencies(): dep_pkg = self.get_package(dep, recurse=True, resolve_virtual=True) if dep_pkg is not None: dep = dep_pkg["Package"] if not dep in self._rdeps: self._rdeps[dep] = set() self._rdeps[dep].add(pkg_name) return self._rdeps def _calc_rrdep_pkg_counts(self, pkg): pkg_name = pkg['Package'] self._compute_package_states() # populate _package_state # calc full recursive reverse dependency package set rrdep_set = set() rdeps = self._get_rdep_dict() next_level = set([pkg_name]) chain_len = 0 while next_level: chain_len += 1 rrdep_set |= next_level new_pkgs = next_level next_level = set([y for x in new_pkgs if x in rdeps for y in rdeps[x]]) next_level -= rrdep_set rrdep_set.remove(pkg_name) # calculate and set the metrics pkg.rrdep_cnt = len(rrdep_set) error_states = self.get_error_states() if self._package_state[pkg_name] in error_states: block_list = [x for x in rrdep_set if self._package_state[x] in error_states] pkg.block_cnt = len(block_list) else: pkg.block_cnt = 0 waiting_states = self.get_waiting_states() if self._package_state[pkg_name] in waiting_states: waiting_list = [x for x in rrdep_set if self._package_state[x] in waiting_states] pkg.waiting_cnt = len(waiting_list) else: pkg.waiting_cnt = 0 pkg.rdep_chain_len = chain_len def block_count(self, name): pkg = self.get_package(name) if pkg is None: return -1 if pkg.block_cnt is None: self._calc_rrdep_pkg_counts(pkg) return pkg.block_cnt def rrdep_count(self, name): pkg = self.get_package(name) if pkg is None: return -1 if pkg.rrdep_cnt is None: self._calc_rrdep_pkg_counts(pkg) return pkg.rrdep_cnt def waiting_count(self, name): pkg = self.get_package(name) if pkg is None: return -1 if pkg.waiting_cnt is None: self._calc_rrdep_pkg_counts(pkg) return pkg.waiting_cnt def rdep_chain_len(self, name): pkg = self.get_package(name) if pkg is None: return -1 if pkg.rdep_chain_len is None: self._calc_rrdep_pkg_counts(pkg) return pkg.rdep_chain_len # vi:set et ts=4 sw=4 : �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/piupartslib/pkgsummary.py������������������������������������������������������������������0000644�0000000�0000000�00000016063�12762537303�014740� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # Copyright 2014 David Steele (dsteele@gmail.com) # # This file is part of Piuparts # # Piuparts is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # Piuparts is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, see <http://www.gnu.org/licenses/>. # Piuparts summary generation module # # This module is used to create exportable section and global package testing # result summaries. # # The canonical location for section summaries is at # # https://piuparts.debian.org/<section>/summary.json # # The global summary is at # # https://piuparts.debian.org/summary.json # # Example output: # # summary.json # { # "_comment": "Debian Piuparts Package Results - https://anonscm.debian.org/... # "_date": "Wed Feb 26 01:48:43 UTC 2014", # "_id": "Piuparts Package Test Results Summary", # "_type": "source", # "_version": "1.0", # "packages": { # "0ad": { # "overall": [ # "X", # 0, # "http://localhost/piuparts/sid-fail-broken-symlinks/source/0/0ad.html" # ], # "stable": [ # "P", # 0, # "http://localhost/piuparts/wheezy/source/0/0ad.html" # ], # "unstable": [ # "X", # 0, # "http://localhost/piuparts/sid-fail-broken-symlinks/source/0/0ad.html" # ] # }, # "0ad-data": { # ... # } # # # The packages are listed by source package. E.g. "unstable" here is a # json-section (see README_server.txt). The single character flags are # defined below. The number is the number of packages which # are blocked from testing due to a failed package. The URL is a human # friendly page for inspecting the results for that package/distribution. # # Binary package results are combined into source package results. The 'worst' # flag in the group is reported ("F" is worst overall). # # For the global summary, the packages 'worst' result across json-sections # is used. In the case of a tie, the more-important-precedence # section/json-section result is used. # # The global file also includes an 'overall' json-section, which contains # the 'worst' result across the other json-sections. import json import datetime from collections import namedtuple, defaultdict class SummaryException(Exception): pass SUMMID = "Piuparts Package Test Results Summary" SUMMVER = "1.0" DEFSEC = 'overall' FlagInfo = namedtuple('FlagInfo', ['word', 'priority', 'states']) flaginfo = { 'F': FlagInfo('Failed', 0, ["failed-testing"]), 'X': FlagInfo('Blocked', 1, [ "cannot-be-tested", "dependency-failed-testing", "dependency-cannot-be-tested", "dependency-does-not-exist", ]), 'W': FlagInfo('Waiting', 2, [ "waiting-to-be-tested", "waiting-for-dependency-to-be-tested", ]), 'P': FlagInfo('Passed', 3, [ "essential-required", "successfully-tested", ]), '-': FlagInfo('Unknown', 4, [ "does-not-exist", "unknown", ]), } state2flg = dict([(y, x[0]) for x in flaginfo.iteritems() for y in x[1].states]) def worst_flag(*flags): try: flag = min(*flags, key=lambda x: flaginfo[x].priority) except KeyError: raise SummaryException("Unknown flag in " + flags.__repr__()) return(flag) def get_flag(state): try: flag = state2flg[state] except KeyError: raise SummaryException("Unknown state - " + state) return(flag) def new_summary(): cdate_array = datetime.datetime.utcnow().ctime().split() utcdate = " ".join(cdate_array[:-1] + ["UTC"] + [cdate_array[-1]]) # define the packages struct. The default should never be the one added dfltentry = ['-', 0, 'invalid url'] pkgstruct = defaultdict(lambda: defaultdict(lambda: dfltentry)) return({ "_id": SUMMID, "_version": SUMMVER, "_date": utcdate, "_comment": "Debian Piuparts Package Results - " "https://anonscm.debian.org/cgit/piuparts/" "piuparts.git/tree/piupartslib/pkgsummary.py", "_type": "source", "packages": pkgstruct, }) def add_summary(summary, rep_sec, pkg, flag, block_cnt, url): if not flag in flaginfo or not isinstance(block_cnt, int) \ or not url.startswith('http'): raise SummaryException("Invalid summary argument") pdict = summary["packages"] [old_flag, old_cnt, old_url] = pdict[pkg][rep_sec] block_cnt = max(block_cnt, old_cnt) if old_flag != worst_flag(old_flag, flag): pdict[pkg][rep_sec] = [flag, block_cnt, url] else: pdict[pkg][rep_sec] = [old_flag, block_cnt, old_url] return summary def merge_summary(gbl_summ, sec_summ): spdict = sec_summ["packages"] for pkg in spdict: for rep_sec in spdict[pkg]: flag, block_cnt, url = spdict[pkg][rep_sec] add_summary(gbl_summ, rep_sec, pkg, flag, block_cnt, url) add_summary(gbl_summ, DEFSEC, pkg, flag, block_cnt, url) return gbl_summ def tooltip(summary, pkg): """Returns e.g. "Failed in testing and stable, blocking 5 packages".""" tip = '' pkgdict = summary['packages'] if pkg in pkgdict: flag, block_cnt, url = pkgdict[pkg][DEFSEC] sections = [x for x in pkgdict[pkg] if x != DEFSEC] applicable = [x for x in sections if pkgdict[pkg][x][0] == flag] tip = flaginfo[flag].word if len(applicable) > 2: tip += ' in ' + ', '.join(applicable[:-1]) + ' and ' + applicable[-1] elif len(applicable) == 2: tip += ' in ' + ' and '.join(applicable) elif len(applicable) == 1: tip += ' in ' + applicable[0] if block_cnt: tip += ", blocking %d packages" % block_cnt tip += '.' return tip def write_summary(summary, fname): with open(fname, 'w') as fl: json.dump(summary, fl, sort_keys=True, indent=1) def read_summary(fname): with open(fname, 'r') as fl: result = json.load(fl) if result["_id"] != SUMMID or result["_version"] != SUMMVER: raise SummaryException('Summary JSON header mismatch') return result if __name__ == '__main__': import sys # read a global summary file and return DDPO info by package summary = read_summary(sys.argv[1]) for pkg in summary['packages']: flag, blocked, url = summary['packages'][pkg][DEFSEC] print pkg, flag, url, tooltip(summary, pkg) �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/slave-bin/���������������������������������������������������������������������������������0000755�0000000�0000000�00000000000�13061276553�011503� 5����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/slave-bin/detect_leftover_processes.in�����������������������������������������������������0000755�0000000�0000000�00000003742�12567171321�017305� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # Copyright 2009 Holger Levsen (holger@layer-acht.org) # Copyright © 2012-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # find processes running in deleted chroots # . @sharedir@/piuparts/lib/read_config.sh # outputs file age in seconds (or 0 if the file does not exist) file_age() { if [ -e "$1" ]; then local ctime now ctime=$(stat -c %Z "$1" 2>/dev/null || echo 0) now=$(date +%s) echo $(($now - $ctime)) else echo "0" fi } get_config_value SLAVEROOT global slave-directory get_config_value PIUPARTS_TMPDIR global tmpdir STATEFILE=$SLAVEROOT/leftover_processes # clear the statefile daily and whine again test $(file_age $STATEFILE) -lt 86000 || rm -f $STATEFILE OUTPUT="$(sudo ls --color=never -lad /proc/*/root 2>/dev/null | grep "$PIUPARTS_TMPDIR" | grep "(deleted)")" if [ -z "$OUTPUT" ]; then rm -f $STATEFILE elif [ "$(cat $STATEFILE 2>/dev/null)" != "$OUTPUT" ]; then echo "Found processes running with a deleted chroot in $PIUPARTS_TMPDIR" echo "This is usually because of 'FAIL: Processes are running inside chroot' which" echo "usually means the package violates 'must use invoke-rc.d (policy 9.3.3.2)'." echo echo "$OUTPUT" echo echo "Please cleanup manually." echo "Since #522918 has been fixed this should no longer happen." echo "$OUTPUT" > $STATEFILE fi ������������������������������piuparts/slave-bin/detect_slave_problems.in���������������������������������������������������������0000755�0000000�0000000�00000004217�12567171321�016404� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # Copyright 2009-2010 Holger Levsen (holger@layer-acht.org) # Copyright © 2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # this scripts monitors the output of piuparts-slave # when running in screen started by ~piupartss/bin/slave_run # . @sharedir@/piuparts/lib/read_config.sh # outputs file age in seconds (or 0 if the file does not exist) file_age() { if [ -e "$1" ]; then local ctime now ctime=$(stat -c %Z "$1" 2>/dev/null || echo 0) now=$(date +%s) echo $(($now - $ctime)) else echo "0" fi } get_config_value SLAVEROOT global slave-directory get_config_value IDLE_SLEEP global idle-sleep 1800 SCREENLOG=$SLAVEROOT/screenlog.0 STATEFILE=$SLAVEROOT/slave-problems # clear the statefile daily and whine again test $(file_age $STATEFILE) -lt 86000 || rm -f $STATEFILE # Only complain if screenlog is older than $IDLE_SPEEP + 1 minute (the slave # likes to sleep that long) and the problem is new or was not reported within # the previous 24 hours. if [ $(file_age $SCREENLOG) -le $(($IDLE_SLEEP + 60)) ]; then rm -f $STATEFILE elif [ ! -f $STATEFILE ]; then { echo "Either a test is running for a very long time (but no test" echo "should run longer than an hour), piuparts-slave hangs or is" echo "not running at all or wasn't started with" echo "~piupartss/bin/slave_run - please investigate and take" echo "appropriate measures!" echo tail $SCREENLOG } | mail -s "problem with piuparts-slave detected" piupartss touch $STATEFILE fi ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/slave-bin/detect_tmp_cruft.in��������������������������������������������������������������0000755�0000000�0000000�00000003267�12567171321�015376� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # Copyright 2009 Holger Levsen (holger@layer-acht.org) # Copyright © 2012-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # check for stale mounts and chroots in $PIUPARTS_TMPDIR # . @sharedir@/piuparts/lib/read_config.sh get_config_value PIUPARTS_TMPDIR global tmpdir MOUNTS="$(mktemp)" cp /proc/mounts "$MOUNTS" if [ "$(grep -v trash "$MOUNTS" | grep -c "$PIUPARTS_TMPDIR")" -gt 1 ] ; then echo "More than one mountpoint below $PIUPARTS_TMPDIR detected!" echo grep "$PIUPARTS_TMPDIR" "$MOUNTS" echo echo "Zero or one mountpoint is normal for piuparts operation, more is not." echo "Please investigate and cleanup." echo fi rm "$MOUNTS" LS_TMP=$(ls --color=never -l "$PIUPARTS_TMPDIR") if [ "$(echo "$LS_TMP" | wc -l)" -gt 12 ] ; then echo "More than ten directories in $PIUPARTS_TMPDIR detected!" echo echo "$LS_TMP" echo echo "$(du -shx "$PIUPARTS_TMPDIR" 2>/dev/null)" echo echo "One is normal for piuparts operation, more is not." echo "Please investigate and cleanup." echo fi �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/slave-bin/slave_cleanup.in�����������������������������������������������������������������0000755�0000000�0000000�00000003132�12567171321�014653� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # Copyright 2012 Holger Levsen (holger@layer-acht.org) # Copyright © 2012-2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # cleanup old chroots etc. in $PIUPARTS_TMPDIR # # - this should only be run (automatically) on boot # . @sharedir@/piuparts/lib/read_config.sh get_config_value PIUPARTS_TMPDIR global tmpdir # do nothing if piuparts-slave is running if pgrep -f share/piuparts/piuparts-slave >/dev/null ; then exit 0 fi # umount all mount points (should be none on boot, but this script can also be called at other times) for MOUNTPOINT in $(tac /proc/mounts | cut -d " " -f 2 | grep "$PIUPARTS_TMPDIR/") do sudo umount "$MOUNTPOINT" done # cleanup tmp mkdir -p "$PIUPARTS_TMPDIR" for dir in $(ls -d1 "$PIUPARTS_TMPDIR"/*/ 2>/dev/null || true) do if sudo test -f "$dir/.piuparts.tmpdir"; then sudo rm -rf --one-file-system "$dir" test ! -d "$dir" || sudo touch "$dir/.piuparts.tmpdir" fi done ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/slave-bin/slave_join�����������������������������������������������������������������������0000755�0000000�0000000�00000002407�12567171321�013562� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # Copyright 2009-2010 Holger Levsen (holger@layer-acht.org) # Copyright © 2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # reattach to an existing slave session # SESSIONNAME=piuparts_slave_screen if screen -ls $SESSIONNAME | grep -q "No Sockets found" ; then echo "piuparts-slave not running!" echo echo "ps faxu | grep piuparts" ps faxu | grep piuparts echo echo "screen -ls" screen -ls exit 1 fi if [ -w $(tty) ]; then screen -x -S $SESSIONNAME else # use script hack to get a new writable tty script -q -c "screen -x -S $SESSIONNAME" /dev/null fi ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/slave-bin/slave_run.in���������������������������������������������������������������������0000755�0000000�0000000�00000004461�13061276553�014041� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # Copyright 2009-2013 Holger Levsen (holger@layer-acht.org) # Copyright © 2013 Andreas Beckmann (anbe@debian.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # Run (several) piuparts-slave instance(s) in screen to allow interactive # control later on. # . @sharedir@/piuparts/lib/read_config.sh get_config_value PYTHONPATH global PYTHONPATH '' get_config_value SLAVEROOT global slave-directory get_config_value PIUPARTS_TMPDIR global tmpdir get_config_value SLAVECOUNT global slave-count 1 export PYTHONPATH SESSIONNAME=piuparts_slave_screen SCREENLOG=$SLAVEROOT/screenlog.0 if ! screen -ls $SESSIONNAME | grep -q "No Sockets found" ; then echo "piuparts-slave is already running!" echo screen -ls exit 1 fi # cleanup cruft from previous runs @sharedir@/piuparts/slave/slave_cleanup rm -f $SCREENLOG # ensure the temporary directory exists mkdir -p $PIUPARTS_TMPDIR mkdir -p $SLAVEROOT cd $SLAVEROOT # Ensure the screen session exists, run normal shell in screen 0 screen -S $SESSIONNAME -d -m echo "Started screen session '$SESSIONNAME'." # run this on a single slave or a sequence of slaves # (use value given, if given and an integer and <= $SLAVECOUNT) if [ "$1" != "" ] && [ "$1" -eq "$1" ] && [ "$1" -le "$SLAVECOUNT" ] ; then SLAVES="$(seq $1)" else SLAVES="$(seq $SLAVECOUNT)" fi for SLAVENUM in $SLAVES do mkdir -p $SLAVENUM SLAVEDIR=$(readlink -f $SLAVENUM) # Ensure there is a window for this slave. screen -S $SESSIONNAME -X screen -t slave$SLAVENUM -L $SLAVENUM # Launch the slave. screen -S $SESSIONNAME -p $SLAVENUM -X stuff " cd $SLAVEDIR @sharedir@/piuparts/piuparts-slave " echo "piuparts-slave $SLAVENUM has been started." done ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/slave-bin/slave_stop.in��������������������������������������������������������������������0000755�0000000�0000000�00000003635�13061276553�014224� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # Copyright 2017 Holger Levsen (holger@layer-acht.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # Stop all running piuparts-slaves gracefully and watch… # . @sharedir@/piuparts/lib/read_config.sh get_config_value SLAVEROOT global slave-directory get_config_value SLAVECOUNT global slave-count 1 cleanup_tails() { pkill --full 'tail -f screenlog.' } cd $SLAVEROOT trap cleanup_tails INT TERM EXIT for i in $(seq 1 $SLAVECOUNT) ; do tail -f screenlog.$i & done sleep 0.5 pkill --signal INT --full '/usr/bin/python /srv/piuparts.debian.org/share/piuparts/piuparts-slave' i=1 while pgrep --full '/usr/bin/python /srv/piuparts.debian.org/share/piuparts/piuparts-slave' > /dev/null ; do echo -n "$(date -u +%T) - " pgrep --full '/usr/bin/python /srv/piuparts.debian.org/share/piuparts/piuparts-slave' | xargs -r echo -n "slaves running: " SLEEP=$(( $i * $i )) echo -n " sleeping ${SLEEP}sec now..." for j in $(seq 1 $SLEEP) ; do sleep 1 echo -n "." done echo i=$(( $i + 1 )) done echo -n "$(date -u +%T) - " pgrep --full '/usr/bin/python /srv/piuparts.debian.org/share/piuparts/piuparts-slave' | xargs echo "slaves running: (should list none) " trap - INT TERM EXIT pkill --full 'tail -f screenlog.' pkill --full 'SCREEN -S piuparts_slave_screen -d -m' ���������������������������������������������������������������������������������������������������piuparts/tests/�������������������������������������������������������������������������������������0000755�0000000�0000000�00000000000�13061276553�010765� 5����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/tests/README�������������������������������������������������������������������������������0000644�0000000�0000000�00000001221�12567171321�011636� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������To get discovered by nosetests the python files in this directory must be named test_<sourcefile>.py. The test case classes should be named 'NameOfTheClassUnderTestTests' and the test functions should be named 'test_functionundertest_expected_bahavior' e.g.: # foobar.py ----- class FooBar(object): def baz(self): return 'baz ----- # tests/test_foobar.py ----- class FooBarTests(unittest.Testcase): def setUp(self): self.fb = FooBar() def test_baz_should_return_baz(self): ret_val = self.fb.baz() self.assertEqual(ret_val, 'baz') ----- Use pymox as mocking framework if needed. apt-get install python-mox3 �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/tests/test_config.py�����������������������������������������������������������������������0000644�0000000�0000000�00000006350�12762537303�013647� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������import unittest import piupartslib.conf as conf import distro_info class ConfStdDistroTests(unittest.TestCase): def setUp(self): self.cobj = conf.Config('notimportant', {}) debdist = distro_info.DebianDistroInfo() self.stable = debdist.stable() self.unstable = debdist.devel() self.oldstable = debdist.old() self.testing = debdist.testing() self.experimental = 'experimental' def testConfStdDistroNames(self): self.assertEqual(self.unstable, 'sid') self.assertEqual(self.experimental, 'experimental') def testConfMapDistro(self): self.assertEqual(self.cobj._map_distro('bogus'), 'unknown') self.assertEqual(self.cobj._map_distro(self.oldstable), 'oldstable') self.assertEqual(self.cobj._map_distro(self.stable), 'stable') self.assertEqual(self.cobj._map_distro(self.testing), 'testing') self.assertEqual(self.cobj._map_distro(self.unstable), 'unstable') self.assertEqual(self.cobj._map_distro(self.experimental), 'experimental') self.assertEqual(self.cobj._map_distro('oldstable'), 'oldstable') self.assertEqual(self.cobj._map_distro('stable'), 'stable') self.assertEqual(self.cobj._map_distro('testing'), 'testing') self.assertEqual(self.cobj._map_distro('unstable'), 'unstable') self.assertEqual(self.cobj._map_distro('experimental'), 'experimental') def testConfMapProposedDistro(self): self.assertEqual( self.cobj._map_distro('stable-proposed'), 'stable') self.assertEqual( self.cobj._map_distro(self.stable + '-proposed'), 'stable') def testConfMapRemainingDistros(self): self.assertEqual(self.cobj._map_distro('rc-buggy'), 'experimental') self.assertEqual( self.cobj._map_distro('Debian6.0.9'), self.cobj._map_distro('squeeze')) self.assertEqual( self.cobj._map_distro('Debian7.4'), self.cobj._map_distro('wheezy')) self.assertEqual( self.cobj._map_distro('Debian8'), self.cobj._map_distro('jessie')) self.assertEqual( self.cobj._map_distro('Debian8.1'), self.cobj._map_distro('jessie')) self.assertEqual( self.cobj._map_distro('Debian9'), self.cobj._map_distro('stretch')) def testConfGetStdDistro(self): for std in [ 'oldstable', 'stable', 'testing', 'unstable', 'experimental']: self.assertEqual( self.cobj.get_std_distro([self.__dict__[std]]), std) self.assertEqual( self.cobj.get_std_distro([self.__dict__[std], 'unknown']), std) self.assertEqual( self.cobj.get_std_distro(['unknown', self.__dict__[std]]), std) self.assertEqual( self.cobj.get_std_distro( ['unknown', 'unknown', self.__dict__[std]]), std) self.assertEqual( self.cobj.get_std_distro( [self.__dict__[std], 'unknown', 'unknown']), std) self.assertEqual(self.cobj.get_std_distro(['unknown']), 'unknown') self.assertEqual( self.cobj.get_std_distro(['unknown', 'unknown']), 'unknown') ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/tests/test_dependencyparser.py�������������������������������������������������������������0000644�0000000�0000000�00000001701�12567171321�015725� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������import unittest import piupartslib.dependencyparser class DependencyParserTests(unittest.TestCase): """Tests for module dependencyparser.""" def parse(self, str): parser = piupartslib.dependencyparser.DependencyParser(str) deps = parser.get_dependencies() names = [] for dep in deps: names.append([]) for simpledep in dep: names[-1].append(simpledep.name) return deps, names def testEmpty(self): deps, names = self.parse("") self.failUnlessEqual(deps, []) def testSingle(self): deps, names = self.parse("foo") self.failUnlessEqual(names, [["foo"]]) def testTwo(self): deps, names = self.parse("foo, bar") self.failUnlessEqual(names, [["foo"], ["bar"]]) def testAlternatives(self): deps, names = self.parse("foo, bar | foobar") self.failUnlessEqual(names, [["foo"], ["bar", "foobar"]]) ���������������������������������������������������������������piuparts/tests/test_piuparts.py���������������������������������������������������������������������0000644�0000000�0000000�00000017770�13061276553�014261� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������import unittest from mox3 import mox import os import shutil import piuparts from piuparts import is_broken_symlink class DefaultsFactoryTests(unittest.TestCase): def setUp(self): self.mox = mox.Mox() self.df = piuparts.DefaultsFactory() piuparts.settings = piuparts.Settings() def tearDown(self): self.mox.UnsetStubs() def test_new_defaults_return_debian_defaults(self): # mock the guess_flavor function as it runs lsb_release in a subprocess self.mox.StubOutWithMock(self.df, 'guess_flavor') self.df.guess_flavor().AndReturn('debian') self.mox.ReplayAll() defaults = self.df.new_defaults() self.mox.VerifyAll() self.assertEqual(defaults.get_keyring(), '/usr/share/keyrings/debian-archive-keyring.gpg') self.assertEqual(defaults.get_components(), ["main", "contrib", "non-free"]) self.assertEqual(defaults.get_mirror(), [("http://deb.debian.org/debian", ["main", "contrib", "non-free"])]) self.assertEqual(defaults.get_distribution(), ['sid']) def test_new_defaults_return_ubuntu_defaults(self): # mock the guess_flavor function as it runs lsb_release in a subprocess self.mox.StubOutWithMock(self.df, 'guess_flavor') self.df.guess_flavor().AndReturn('ubuntu') self.mox.ReplayAll() defaults = self.df.new_defaults() self.mox.VerifyAll() self.assertEqual(defaults.get_keyring(), '/usr/share/keyrings/ubuntu-archive-keyring.gpg') self.assertEqual(defaults.get_components(), ["main", "universe", "restricted", "multiverse"]) self.assertEqual(defaults.get_mirror(), [("http://archive.ubuntu.com/ubuntu", ["main", "universe", "restricted", "multiverse"])]) def test_new_defaults_panics_with_unknown_flavor(self): # mock the guess_flavor function as it runs lsb_release in a subprocess # and the panic function as it would use sys.exit() self.mox.StubOutWithMock(self.df, 'guess_flavor') self.df.guess_flavor().AndReturn('centos') self.mox.StubOutWithMock(piuparts, 'panic') piuparts.panic().AndReturn('Oh dear! Its CentOS!') self.mox.ReplayAll() defaults = self.df.new_defaults() self.mox.VerifyAll() # panic() would cause sys.exit() so no Defaults object would # ever be returned self.assertEqual(defaults, None) class IsBrokenSymlinkTests(unittest.TestCase): testdir = "is-broken-symlink-testdir" def symlink(self, target, name): pathname = os.path.join(self.testdir, name) os.symlink(target, pathname) self.symlinks.append(pathname) def setUp(self): self.symlinks = [] os.mkdir(self.testdir) self.symlink("notexist", "relative-broken") self.symlink("relative-broken", "relative-broken-to-symlink") self.symlink(".", "relative-works") self.symlink("relative-works", "relative-works-to-symlink") self.symlink("/etc", "absolute-broken") self.symlink("absolute-broken", "absolute-broken-to-symlink") self.symlink("/", "absolute-works") self.symlink("/absolute-works", "absolute-works-to-symlink") os.mkdir(os.path.join(self.testdir, "dir")) self.symlink("dir", "dir-link") os.mkdir(os.path.join(self.testdir, "dir/subdir")) self.symlink("subdir", "dir/subdir-link") self.symlink("notexist/", "trailing-slash-broken") self.symlink("dir/", "trailing-slash-works") self.symlink("selfloop", "selfloop") self.symlink("/absolute-selfloop", "absolute-selfloop") self.symlink("../dir/selfloop", "dir/selfloop") self.symlink("../dir-link/selfloop", "dir/selfloop1") self.symlink("../../dir/subdir/selfloop", "dir/subdir/selfloop") self.symlink("../../dir-link/subdir/selfloop", "dir/subdir/selfloop1") self.symlink("../../link/subdir-link/selfloop", "dir/subdir/selfloop2") self.symlink("../../dir-link/subdir-link/selfloop", "dir/subdir/selfloop3") self.symlink("explode/bomb", "explode") def tearDown(self): shutil.rmtree(self.testdir) def testRelativeBroken(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "relative-broken")) def testRelativeBrokenToSymlink(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "relative-broken-to-symlink")) def testAbsoluteBroken(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "absolute-broken")) def testAbsoluteBrokenToSymlink(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "absolute-broken-to-symlink")) def testTrailingSlashBroken(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "trailing-slash-broken")) def testSelfLoopBroken(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "selfloop")) def testExpandingSelfLoopBroken(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "explode")) def testAbsoluteSelfLoopBroken(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "absolute-selfloop")) def testSubdirSelfLoopBroken(self): self.failUnless(is_broken_symlink(self.testdir, self.testdir, "dir/selfloop")) self.failUnless(is_broken_symlink(self.testdir, self.testdir, "dir/selfloop1")) self.failUnless(is_broken_symlink(self.testdir, self.testdir, "dir/subdir/selfloop")) self.failUnless(is_broken_symlink(self.testdir, self.testdir, "dir/subdir/selfloop1")) self.failUnless(is_broken_symlink(self.testdir, self.testdir, "dir/subdir/selfloop2")) self.failUnless(is_broken_symlink(self.testdir, self.testdir, "dir/subdir/selfloop3")) def testRelativeWorks(self): self.failIf(is_broken_symlink(self.testdir, self.testdir, "relative-works")) def testRelativeWorksToSymlink(self): self.failIf(is_broken_symlink(self.testdir, self.testdir, "relative-works-to-symlink")) def testAbsoluteWorks(self): self.failIf(is_broken_symlink(self.testdir, self.testdir, "absolute-works")) def testAbsoluteWorksToSymlink(self): self.failIf(is_broken_symlink(self.testdir, self.testdir, "absolute-works-to-symlink")) def testTrailingSlashWorks(self): self.failIf(is_broken_symlink(self.testdir, self.testdir, "trailing-slash-works")) def testMultiLevelNestedSymlinks(self): # target/first-link -> ../target/second-link -> ../target os.mkdir(os.path.join(self.testdir, "target")) self.symlink("../target", "target/second-link") self.symlink("../target/second-link", "target/first-link") self.failIf(is_broken_symlink(self.testdir, self.testdir, "target/first-link")) def testMultiLevelNestedAbsoluteSymlinks(self): # first-link -> /second-link/final-target # second-link -> /target-dir os.mkdir(os.path.join(self.testdir, "final-dir")) os.mkdir(os.path.join(self.testdir, "final-dir/final-target")) self.symlink("/second-link/final-target", "first-link") self.symlink("/final-dir", "second-link") self.failIf(is_broken_symlink(self.testdir, self.testdir, "first-link")) ��������piuparts/tests/test_pkgsummary.py�������������������������������������������������������������������0000644�0000000�0000000�00000012571�12762537303�014603� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������import unittest import datetime import shutil import tempfile import os import json import piupartslib.pkgsummary as pkgsummary class PkgSummaryTests(unittest.TestCase): def testSummFlaginfoStateDups(self): finfo = pkgsummary.flaginfo states = sorted([y for x in finfo for y in finfo[x].states]) nodups = sorted(list(set(states))) self.assertTrue('successfully-tested' in states) self.assertEqual(states, nodups) def testSummGetFlag(self): self.assertEqual('F', pkgsummary.get_flag('failed-testing')) self.assertEqual('X', pkgsummary.get_flag('dependency-does-not-exist')) self.assertEqual('P', pkgsummary.get_flag('successfully-tested')) self.assertEqual('W', pkgsummary.get_flag('waiting-to-be-tested')) with self.assertRaises(pkgsummary.SummaryException): pkgsummary.get_flag('bogus-state') def testSummWorstFlag(self): self.assertEqual('F', pkgsummary.worst_flag('F')) self.assertEqual('P', pkgsummary.worst_flag('P')) self.assertEqual('F', pkgsummary.worst_flag('P', 'F')) self.assertEqual('F', pkgsummary.worst_flag('F', 'F')) self.assertEqual('W', pkgsummary.worst_flag('W', 'P')) self.assertEqual('F', pkgsummary.worst_flag('W', 'P', 'F', 'X', '-')) with self.assertRaises(pkgsummary.SummaryException): pkgsummary.worst_flag('Z') class PkgSummaryAddTests(unittest.TestCase): def setUp(self): self.summ = pkgsummary.new_summary() def testSummNewSumm(self): # Verify any parameters which are depended on downstream self.assertEqual("Piuparts Package Test Results Summary", self.summ['_id']) self.assertEqual("1.0", self.summ['_version']) self.assertEqual({}, self.summ['packages']) thedate = datetime.datetime.strptime(self.summ['_date'], "%a %b %d %H:%M:%S UTC %Y") def testSummAddArgValidation(self): with self.assertRaises(pkgsummary.SummaryException): pkgsummary.add_summary( self.summ, 'foodist', 'foopkg', 'Z', 0, 'http://foo') with self.assertRaises(pkgsummary.SummaryException): pkgsummary.add_summary( self.summ, 'foodist', 'foopkg', 'X', 'bogus', 'http://foo') with self.assertRaises(pkgsummary.SummaryException): pkgsummary.add_summary( self.summ, 'foodist', 'foopkg', 'X', 1, 'ittp://foo') pkgsummary.add_summary( self.summ, 'foodist', 'foopkg', 'X', 1, 'http://foo') def testSummAddArgStorageFormat(self): # store non-overlapping entries pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'X', 0, 'http://foo') pkgsummary.add_summary( self.summ, 'dist', 'pkg2', 'W', 1, 'http://foo2') pkgsummary.add_summary( self.summ, 'dist2', 'pkg3', 'P', 2, 'http://foo3') self.assertEqual( ['X', 0, 'http://foo'], self.summ['packages']['pkg']['dist']) self.assertEqual( ['W', 1, 'http://foo2'], self.summ['packages']['pkg2']['dist']) self.assertEqual( ['P', 2, 'http://foo3'], self.summ['packages']['pkg3']['dist2']) def testSummAddOverwriteFlag(self): pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'X', 0, 'http://foo') pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'P', 0, 'http://foo2') self.assertEqual('X', self.summ['packages']['pkg']['dist'][0]) self.assertEqual('http://foo', self.summ['packages']['pkg']['dist'][2]) pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'F', 0, 'http://foo3') self.assertEqual('F', self.summ['packages']['pkg']['dist'][0]) self.assertEqual('http://foo3', self.summ['packages']['pkg']['dist'][2]) def testSummAddBlockCount(self): pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'X', 0, 'http://foo') pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'P', 1, 'http://foo') self.assertEqual(1, self.summ['packages']['pkg']['dist'][1]) pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'F', 2, 'http://foo') self.assertEqual(2, self.summ['packages']['pkg']['dist'][1]) def testSummMerge(self): pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'X', 0, 'http://foo') mergesumm = pkgsummary.new_summary() pkgsummary.merge_summary(mergesumm, self.summ) self.assertEqual(mergesumm['packages']['pkg']['dist'], self.summ['packages']['pkg']['dist']) self.assertEqual(mergesumm['packages']['pkg']['dist'], mergesumm['packages']['pkg']['overall']) class PkgSummaryStorageTests(unittest.TestCase): def setUp(self): self.summ = pkgsummary.new_summary() pkgsummary.add_summary(self.summ, 'dist', 'pkg', 'X', 0, 'http://foo') self.tmpdir = tempfile.mkdtemp() self.tmpfilename = os.path.join(self.tmpdir, "foo.json") pkgsummary.write_summary(self.summ, self.tmpfilename) def tearDown(self): shutil.rmtree(self.tmpdir) def testSummFileRead(self): summ2 = pkgsummary.read_summary(self.tmpfilename) self.assertEqual(self.summ, summ2) def testSummFileStorage(self): with open(self.tmpfilename, 'r') as fl: summ2 = json.load(fl) self.assertEqual(self.summ, summ2) ���������������������������������������������������������������������������������������������������������������������������������������piuparts/tests/unittests.py�������������������������������������������������������������������������0000644�0000000�0000000�00000003642�12567171321�013403� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# -*- coding: utf-8 -*- import os import StringIO import unittest import piupartslib.packagesdb class FakeLogDB(piupartslib.packagesdb.LogDB): """A fake version of the LogDB class, for testing This version simulates filesystem actions so that there is no need to do actual I/O. Cleaner, although not quite as thorough. """ def __init__(self): self.dict = { "pass": [], "fail": [], "untestable": [], "reserved": [], "bugged": [], "affected": [], } def listdir(self, dirname): return self.dict[dirname] def _parse(self, pathname): return os.path.dirname(pathname), os.path.basename(pathname) def exists(self, pathname): vdir, base = self._parse(pathname) return base in self.dict[vdir] def open_file(self, pathname, mode): vdir, base = self._parse(pathname) self.dict[vdir].append(base) return StringIO.StringIO() def remove_file(self, pathname): vdir, base = self._parse(pathname) if base in self.dict[vdir]: del self.dict[vdir] def create(self, subdir, package, version, contents): return True class PackagesDbTests(unittest.TestCase): def new_db(self, packages_file_contents): db = piupartslib.packagesdb.PackagesDB(FakeLogDB()) db.read_packages_file(StringIO.StringIO(packages_file_contents)) return db def reserve(self, packages_file_contents): db = self.new_db(packages_file_contents) return db.reserve_package() def testNoPackages(self): p = self.reserve("") self.failUnlessEqual(p, None) def testNoDeps(self): p = self.reserve("""\ Package: foo Version: 1.0-1 """) self.failIfEqual(p, None) self.failUnlessEqual(p["Package"], "foo") if __name__ == "__main__": unittest.main() # vi:set et ts=4 sw=4 : ����������������������������������������������������������������������������������������������piuparts/update-piuparts-master-setup���������������������������������������������������������������0000755�0000000�0000000�00000005350�13061276553�015332� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # # update piuparts master setup from git (eg. used on pejacevic.debian.org) # # Copyright 2009-2013 Holger Levsen (holger@layer-acht.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA if [ "`id -n -u`" != "piupartsm" ] ; then echo please run this script as piupartsm user exit 1 fi PIUPARTS_PREFIX=/srv/piuparts.debian.org PIUPARTS_HTDOCS=$PIUPARTS_PREFIX/htdocs PIUPARTS_TMPDIR=$PIUPARTS_PREFIX/tmp # # create $PIUPARTS_PREFIX # if [ ! -d $PIUPARTS_PREFIX ] ; then sudo mkdir -p $PIUPARTS_PREFIX sudo chown piupartsm:piuparts $PIUPARTS_PREFIX sudo chmod 0775 $PIUPARTS_PREFIX fi # # update source # if [ ! -d $PIUPARTS_PREFIX/src ] ; then mkdir -p $PIUPARTS_PREFIX/src chmod 0755 $PIUPARTS_PREFIX/src cd $PIUPARTS_PREFIX/src git clone git://git.debian.org/git/piuparts/piuparts.git cd piuparts git checkout develop fi cd $PIUPARTS_PREFIX/src/piuparts pwd # git checkout branch if $1 is given if [ ! -z "$1" ] ; then git checkout $1 fi # git fetch+pull if $2 is given if [ ! -z "$2" ] ; then git fetch $2 git pull $2 $1 fi # # install everything from GIT into PIUPARTS_PREFIX # make clean make prefix=$PIUPARTS_PREFIX \ build build-doc make prefix=$PIUPARTS_PREFIX \ docdir=$PIUPARTS_HTDOCS/doc \ htdocsdir=$PIUPARTS_HTDOCS \ install install-doc install-conf-4-running-from-git make clean # # install copies of the weather icons # to avoid needing FollowSymlinks in the apache config # for icon in weather-severe-alert.png sunny.png do if [ -e /usr/share/icons/Tango/24x24/status/$icon ] ; then cp -f /usr/share/icons/Tango/24x24/status/$icon $PIUPARTS_HTDOCS/images/$icon fi done # # update $PIUPARTS_PREFIX # cd $PIUPARTS_PREFIX pwd mkdir -p master backup # to support multiple hosts with this setup cd etc/piuparts HOSTNAME=$(hostname) for f in piuparts.conf do ln -sf $f.$HOSTNAME $f done # # create working dir # mkdir -p $PIUPARTS_TMPDIR # tell DSA not to back it up touch $PIUPARTS_TMPDIR/.nobackup # # update master home # cd pwd ln -sf $PIUPARTS_PREFIX/share/piuparts/master bin crontab $PIUPARTS_PREFIX/etc/piuparts/crontab-master echo "Update finished." ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������piuparts/update-piuparts-slave-setup����������������������������������������������������������������0000755�0000000�0000000�00000005345�13061276553�015155� 0����������������������������������������������������������������������������������������������������ustar ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/sh set -e # # update piuparts slave setup from git (eg. used on piu-slave-*.debian.org) # # Copyright 2009-2013, 2017 Holger Levsen (holger@layer-acht.org) # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA if [ "`id -n -u`" != "piupartss" ] ; then echo please run this script as piupartss user exit 1 fi PIUPARTS_PREFIX=/srv/piuparts.debian.org PIUPARTS_HTDOCS=$PIUPARTS_PREFIX/htdocs PIUPARTS_TMPDIR=$PIUPARTS_PREFIX/tmp # # create $PIUPARTS_PREFIX # if [ ! -d $PIUPARTS_PREFIX ] ; then sudo mkdir -p $PIUPARTS_PREFIX sudo chown piupartss:piuparts $PIUPARTS_PREFIX sudo chmod 0775 $PIUPARTS_PREFIX fi # # update source # if [ ! -d $PIUPARTS_PREFIX/src ] ; then mkdir -p $PIUPARTS_PREFIX/src chmod 0755 $PIUPARTS_PREFIX/src cd $PIUPARTS_PREFIX/src git clone git://git.debian.org/git/piuparts/piuparts.git cd piuparts git checkout develop fi cd $PIUPARTS_PREFIX/src/piuparts pwd # git checkout branch if $1 is given if [ ! -z "$1" ] ; then git checkout $1 fi # git fetch+pull if $2 is given if [ ! -z "$2" ] ; then git fetch $2 git pull $2 $1 fi # # install everything from GIT into PIUPARTS_PREFIX # make clean make prefix=$PIUPARTS_PREFIX \ build build-doc make prefix=$PIUPARTS_PREFIX \ docdir=$PIUPARTS_HTDOCS/doc \ htdocsdir=$PIUPARTS_HTDOCS \ install install-doc install-conf-4-running-from-git make clean # # update $PIUPARTS_PREFIX # cd $PIUPARTS_PREFIX pwd mkdir -p slave slave/basetgz # to support multiple hosts with this setup cd etc/piuparts HOSTNAME=$(hostname) for f in piuparts.conf do ln -sf $f.$HOSTNAME $f done # # create working dir # if [ ! -d $PIUPARTS_TMPDIR ] ; then if ! [ -L $PIUPARTS_TMPDIR ] && [ -d /srv/piuparts-workspace ] ; then ln -sf /srv/piuparts-workspace $PIUPARTS_TMPDIR fi fi if [ ! -d $PIUPARTS_TMPDIR ] && [ ! -L $PIUPARTS_TMPDIR ] ; then echo "$PIUPARTS_TMPDIR is neither a directory nor link, please fix." exit 1 fi # tell DSA not to back it up touch $PIUPARTS_TMPDIR/.nobackup # # update slave home # cd pwd ln -sf $PIUPARTS_PREFIX/share/piuparts/slave bin crontab $PIUPARTS_PREFIX/etc/piuparts/crontab-slave echo "Update finished." �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������