[Piuparts-commits] [SCM] piuparts git repository branch, master, updated. eda668423fa87898c59d1075118693714aa5a053
Holger Levsen
holger at layer-acht.org
Fri Dec 23 10:26:55 UTC 2011
The following commit has been merged in the master branch:
commit e51d1708c9c1055c8242d301d0b53bf879a59f20
Merge: 8a1e3b79a6bd7fa07b628b9b75089a5f8b24d22f 2ca8affee64f9279015b721e55e14c272ed89d9e
Author: Holger Levsen <holger at layer-acht.org>
Date: Sun Nov 27 11:55:16 2011 +0100
Merge branch 'feature/640648' into develop
* piupartslib/packagesdb.py: Modify several functions in PackagesDB class
to use python 'set' type, instead of a list. This permitted replacing
inefficient combination of unique() function and random.shuffle() with
python set.pop() method. Since python prohibits storing non-hashable
object in a set, minor modifications to piuparts-report and to
piuparts-master required. (Closes: #640648)
diff --combined debian/changelog
index e7ecb9c,46a5d17..fa4f3e1
--- a/debian/changelog
+++ b/debian/changelog
@@@ -3,90 -3,23 +3,96 @@@ piuparts (0.42) UNRELEASED; urgency=lo
[ Holger Levsen ]
* piuparts.py:
- add to self.ignored_files: /etc/blkid.tab (Closes: #638831)
+ - add to self.ignored_patterns: /var/lib/apt/lists/.*
+ - apply patch by Stefano Rivera to properly install and remove
+ logrotate. Thanks Stefano! (Closes: #638832)
+ - apply patch by Gregor Herrmann to fix --minimize. (Closes: #648423)
+ * Remove Debian.NEWS entry about source in git. (Closes: #640121)
+ * piuparts.py, piuparts-report.py, ChangeLog: Expand tabs to spaces.
+ * Remove whitespaces from whitespace-only lines.
+ * piuparts-slave.py: Replace deprecated os.popen2 with subprocess.Popen,
+ thanks to Scott Schaefer and Evgeni Golov for their patches.
+ (Closes: #640646)
[ Mika Pflüger ]
- * piuparts-analyze.py: Rewrite to use python-debianbts to analyze if bugs
- are filed already.
+ * piuparts-analyze.py:
+ - Rewrite to use python-debianbts to analyze if bugs are filed already.
+ - The BTS only tracks source versions, so remove binNMU-part from
+ package versions when comparing with versions from the BTS.
+ - Reduce noise in the output by only printing one action/advise per
+ package.
+ - Fix extraction of package versions from bug reports. Thanks to
+ Andreas Beckmann for catching and solving the error.
* debian/control: Add python-apt and python-debianbts to piuparts depends.
- [ Holger Levsen ]
- * piuparts.py: apply patch by Stefano Rivera to properly install and remove
- logrotate. Thanks Stefano! (Closes: #638832)
-
[ Scott Schaefer ]
+ * debian/copyright: Make it compliant with DEP-5.
+ * piuparts-slave.py:
+ - Replace deprecated os.popen2 with subprocess.Popen. (Closes: #640646)
+ - Add some more logging.
+ * piuparty.py: kill (via SIGTERM, then if that fails, via SIGKILL), leftover
+ processes. (Closes: #522918)
+ * Test for incorrect diversions (dpkg-divert):
+ a) Existing diversions removed/modified, and/or
+ b) Installed diversions not removed by purge:
+ (Closes: #588313)
+ * Kill children (hard-coded value, 45 minutes) to insure test doesn't
+ run "forever" (Closes: #640647)
+ * piupartslib/packagesdb.py: Modify several functions in PackagesDB class
+ to use python 'set' type, instead of a list. This permitted replacing
+ inefficient combination of unique() function and random.shuffle() with
+ python set.pop() method. Since python prohibits storing non-hashable
+ object in a set, minor modifications to piuparts-report and to
- piuparts-master required. (Closes: 640648)
++ piuparts-master required. (Closes: #640648)
+
+ [ Andreas Beckmann ]
+ * *.py: Add vim modeline.
+ * piuparts.py:
+ - Add unittests for misbehaving symlinks.
+ - Fix resolving absolute symlinks of intermediate directory components,
+ i.e. /var/run -> /run while checking /etc/motd -> /var/run/motd.
+ Solves about 30000 false positives of
+ 'Broken symlinks: /etc/motd -> /var/run/motd'. (Closes: #648784)
+ - When running commands in the chroot, redirect stdin from /dev/null.
+ - Stop using Popen.communicate() as it may run out of memory.
+ - Terminate commands producing more than 2 MB of output.
+ - Add missing post_install_* hook to install_packages_by_name().
+ (Closes: #628077)
+ - Create /etc/dpkg/dpkg.cfg.d/ if missing inside the chroot (e.g. on
+ lenny). (Closes: #647752)
+ - Remove logrotate and its dependencies only once.
+ - Only run 'apt-get update' after updating the sources.list.
+ - Only run 'apt-get clean' before creating tarballs or saving meta data.
+ - Do the same checks for running processes and broken symlinks in all
+ tests.
+ - Use consistent variable names for package lists.
+ - Compute the changes in restore_selections().
+ - Check for settings.scriptsdir inside run_scripts().
+ - Consistently use chroot.relative() to build filenames inside the chroot.
+ - Create piupart's apt config in the chroot as /etc/apt.conf.d/piuparts
+ instead of /etc/apt.conf in order to allow overriding the settings from
+ custom scripts by just dropping new config bits in e.g.
+ /etc/apt/apt.conf.d/piuparts-foobar.
+ apt.conf.d is supported in lenny, possibly earlier releases, too.
+ * piuparts-slave.py:
+ - Fix triggering tarball recreation.
+ - Check tarball age regularily.
+ - Log tarball creation in *.tgz.log.
+ * piuparts-report.py:
+ - state-*.html: Sort package lists by name.
+ - source/?/*.html: Sort binary packages by name.
+ - Dependency lists: resolve virtual packages and link to a real package.
+ - Update list of error states to be highlighted.
+
+ [ Dave Steele ]
+ * piuparts-slave.py: make Section.run() report the number of packages
+ processed and use this to decide whether a slave should sleep.
+ (Closes: #649967)
+
+ [ Stefano Rivera ]
+ * piuparts.py: use eatmydata by default, add option --no-eatmydata. (This
+ was discussed in #633033.)
+ * debian/control: Add eatmydata to recommends.
-- Holger Levsen <holger at debian.org> Sun, 28 Aug 2011 09:50:12 +0200
@@@ -120,7 -53,7 +126,7 @@@ piuparts (0.41) unstable; urgency=lo
environment variable, the latter overwriting the former (if present)
- Thanks to Scott Schaefer for the patch. (Closes: #632046)
- new option "--no-install-purge-test" to only do upgrade tests
- - Thanks to Andreas Bergmann for the patch (Closes: #588482)
+ - Thanks to Andreas Beckmann for the patch (Closes: #588482)
- run dpkg with --force-unsafe-io by default and introduce new option
"--dpkg-noforce-unsafe-io" to disable this feature. (Closes: #633033)
Thanks to Scott once more!
diff --combined piuparts-master.py
index 203a9c2,f90119e..ab5f843
--- a/piuparts-master.py
+++ b/piuparts-master.py
@@@ -79,7 -79,7 +79,7 @@@ class Protocol
line = self._input.readline()
logging.debug(">> " + line.rstrip())
return line
-
+
def _writeline(self, line):
logging.debug("<< " + line)
self._output.write(line + "\n")
@@@ -152,8 -152,8 +152,8 @@@ class Master(Protocol)
(count, command, " ".join(args)))
def dump_pkgs(self):
for st in self._binary_db.get_states():
- for pkg in self._binary_db.get_pkg_names_in_state(st):
- logging.debug("%s : %s\n" % (st,pkg))
+ for name in self._binary_db.get_pkg_names_in_state(st):
+ logging.debug("%s : %s\n" % (st,name))
def _reserve(self, command, args):
self._check_args(0, command, args)
@@@ -198,12 -198,12 +198,12 @@@ def main()
section = sys.argv[1]
config = Config(section=section)
config.read(CONFIG_FILE)
-
+
setup_logging(logging.DEBUG, config["log-file"])
if not os.path.exists(os.path.join(master_directory, section)):
os.makedirs(os.path.join(master_directory, section))
-
+
logging.info("Fetching %s" % config["packages-url"])
packages_file = piupartslib.open_packages_url(config["packages-url"])
known_circular_depends = config["known_circular_depends"]
@@@ -218,5 -218,3 +218,5 @@@
if __name__ == "__main__":
main()
+
+# vi:set et ts=4 sw=4 :
diff --combined piuparts-report.py
index 99e923b,e3028a7..f5a8ef2
--- a/piuparts-report.py
+++ b/piuparts-report.py
@@@ -185,11 -185,8 +185,11 @@@ HTML_FOOTER = ""
<div id="footer">
<div>
piuparts was written by <a href="mailto:liw at iki.fi">Lars Wirzenius</a> and is now maintained by
- <a href="mailto:holger at debian.org">Holger Levsen</a>,
- <a href="mailto:luk at debian.org">Luk Claes</a> and <a href="http://qa.debian.org/" target="_blank">others</a>.
+ <a href="mailto:holger at debian.org">Holger Levsen</a> and
+ <a href="mailto:piuparts-devel at lists.alioth.debian.org">others</a> using
+ <a href="http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git" target="_blank">piuparts.git</a> and
+ <a href="http://anonscm.debian.org/gitweb/?p=piuparts/piatti.git" target="_blank">piatti.git</a> and the
+ <a href="http://packages.qa.debian.org/piuparts" target="_blank">PTS</a>.
GPL2 <a href="http://packages.debian.org/changelogs/pool/main/p/piuparts/current/copyright" target="_blank">licenced</a>.
Weather icons are from the <a href="http://tango.freedesktop.org/Tango_Icon_Library" target="_blank">Tango Icon Library</a>.
<a href="http://validator.w3.org/check?uri=referer">
@@@ -418,7 -415,7 +418,7 @@@ def setup_logging(log_level, log_file_n
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(formatter)
logger.addHandler(handler)
-
+
if log_file_name:
handler = logging.FileHandler(log_file_name)
logger.addHandler(handler)
@@@ -434,22 -431,7 +434,22 @@@ def html_protect(vstr)
def emphasize_reason(reason):
- if reason in ("unknown", "failed-testing", "circular-dependency", "dependency-failed-testing", "dependency-does-not-exist", "cannot-be-tested"):
+ bad_states = [
+ #"successfully-tested",
+ "failed-testing",
+ "cannot-be-tested",
+ #"essential-required",
+ #"waiting-to-be-tested",
+ #"waiting-for-dependency-to-be-tested",
+ "dependency-failed-testing",
+ "dependency-cannot-be-tested",
+ "dependency-does-not-exist",
+ "circular-dependency",
+ "unknown",
+ "unknown-preferred-alternative",
+ "no-dependency-from-alternatives-exists",
+ ]
+ if reason in bad_states:
reason = "<em>"+reason+"</em>"
return reason
@@@ -643,7 -625,7 +643,7 @@@ class Section
return links
def link_to_maintainer_summary(self, maintainer):
- email = get_email_address(maintainer)
+ email = get_email_address(maintainer)
return "<a href=\"/%s/maintainer/%s/%s.html\">%s</a>" % (self._config.section,maintainer_subdir(email),email,html_protect(maintainer))
def link_to_uploaders(self, uploaders):
@@@ -665,7 -647,7 +665,7 @@@
if state != "unknown":
link = "<a href=\"/%s/%s\">%s</a>" % (
section,
- "state-"+state+".html"+"#"+package_name,
+ "state-"+state+".html"+"#"+self._binary_db._packages[package_name]["Package"],
link_target)
else:
if link_target == package_name:
@@@ -702,7 -684,7 +702,7 @@@
counts = current_day
total = 0
for state in self._binary_db.get_states():
- count = len(self._binary_db.get_packages_in_state(state))
+ count = len(self._binary_db.get_pkg_names_in_state(state))
header += ", %s" % state
counts += ", %s" % count
logging.debug("%s: %s" % (state, count))
@@@ -779,7 -761,7 +779,7 @@@
success = True
failed = False
binaryrows = ""
- for binary in binaries.split(", "):
+ for binary in sorted(binaries.split(", ")):
state = self._binary_db.state_by_name(binary)
if state == "unknown":
# Don't track udebs and binary packages on other archs.
@@@ -808,7 -790,7 +808,7 @@@
sourcerows += "<tr class=\"normalrow\"><td class=\"labelcell\">Maintainer:</td><td class=\"contentcell2\" colspan=\"5\">%s</td></tr>" % (self.link_to_maintainer_summary(maintainer))
if uploaders:
sourcerows += "<tr class=\"normalrow\"><td class=\"labelcell\">Uploaders:</td><td class=\"contentcell2\" colspan=\"5\">%s</td></tr>" % (self.link_to_uploaders(uploaders))
-
+
source_summary_page_path = os.path.join(self._output_directory, "source", source_subdir(source))
if not os.path.exists(source_summary_page_path):
os.makedirs(source_summary_page_path)
@@@ -884,7 -866,7 +884,7 @@@
if source_state != "udeb":
sources += "%s: %s\n" % (source, source_state)
self.create_maintainer_templates_for_source(source, source_state, sourcerows, binaryrows, maintainer, uploaders)
-
+
write_file(os.path.join(self._output_directory, "sources.txt"), sources)
@@@ -970,7 -952,7 +970,7 @@@
analysis = self.create_and_link_to_analysises(state)
tablerows += ("<tr class=\"normalrow\"><td class=\"contentcell2\"><a href='state-%s.html'>%s</a>%s</td>" +
"<td class=\"contentcell2\">%d</td><td class=\"contentcell2\">%s</td></tr>\n") % \
- (html_protect(state), html_protect(state), analysis, len(self._binary_db.get_packages_in_state(state)),
+ (html_protect(state), html_protect(state), analysis, len(self._binary_db.get_pkg_names_in_state(state)),
dir_link)
try:
tablerows += self.make_stats_graph();
@@@ -993,8 -975,8 +993,9 @@@
for state in self._binary_db.get_states():
logging.debug("Writing page for %s" % state)
vlist = ""
- for package in sorted(self._binary_db.get_packages_in_state(state),
- for name in self._binary_db.get_pkg_names_in_state(state):
++ for name in sorted(self._binary_db.get_pkg_names_in_state(state):
+ key=lambda pkg: pkg["Package"]):
+ package = self._binary_db.get_package(name)
vlist += "<li id=\"%s\">%s (%s)" % (
package["Package"],
self.link_to_source_summary(package["Package"]),
@@@ -1115,5 -1097,3 +1116,5 @@@ def main()
if __name__ == "__main__":
main()
+
+# vi:set et ts=4 sw=4 :
diff --combined piupartslib/packagesdb.py
index 1c9d27c,667485c..28337dc
--- a/piupartslib/packagesdb.py
+++ b/piupartslib/packagesdb.py
@@@ -26,7 -26,6 +26,6 @@@ Lars Wirzenius <liw at iki.fi
import dircache
import os
- import random
import tempfile
import UserDict
@@@ -45,45 -44,8 +44,7 @@@ def rfc822_like_header_parse(input)
else:
headers.append(line)
return headers
-
- def unique (s):
- # taken from http://code.activestate.com/recipes/52560/ - thanks to Tim Peters
- n = len(s)
- if n == 0:
- return []
-
- u = {}
- try:
- for x in s:
- u[x] = 1
- except TypeError:
- del u # move on to the next method
- else:
- return u.keys()
-
- try:
- t = list(s)
- t.sort()
- except TypeError:
- del t # move on to the next method
- else:
- assert n > 0
- last = t[0]
- lasti = i = 1
- while i < n:
- if t[i] != last:
- t[lasti] = last = t[i]
- lasti += 1
- i += 1
- return t[:lasti]
-
- # Brute force is all that's left.
- u = []
- for x in s:
- if x not in u:
- u.append(x)
- return u
-
class Package(UserDict.UserDict):
def __init__(self, headers):
@@@ -93,7 -55,7 +54,7 @@@
name, value = header.split(":", 1)
self[name.strip()] = value.strip()
self._parsed_deps = {}
-
+
def _parse_dependencies(self, header_name):
if header_name in self._parsed_deps:
depends = self._parsed_deps[header_name]
@@@ -160,7 -122,7 +121,7 @@@ class LogDB
def listdir(self, dirname):
return dircache.listdir(dirname)
-
+
def exists(self, pathname):
try:
cache = self.exists_cache
@@@ -170,10 -132,10 +131,10 @@@
if pathname not in cache:
cache[pathname] = os.path.exists(pathname)
return cache[pathname]
-
+
def open_file(self, pathname, mode):
return file(pathname, mode)
-
+
def remove_file(self, pathname):
os.remove(pathname)
@@@ -186,7 -148,7 +147,7 @@@
if self.exists(os.path.join(subdir, log_name)):
return True
return False
-
+
def any_log_exists(self, package, subdirs):
try:
cache = self.basename_cache
@@@ -202,11 -164,11 +163,11 @@@
if len(parts) == 2 and parts[0] == package_name:
return True
return False
-
+
def create(self, subdir, package, version, contents):
(fd, temp_name) = tempfile.mkstemp(dir=subdir)
os.close(fd)
-
+
# tempfile.mkstemp sets the file mode to be readable only by owner.
# Let's make it follow the umask.
umask = os.umask(0)
@@@ -229,7 -191,7 +190,7 @@@
full_name = os.path.join(subdir, self._log_name(package, version))
if self.exists(full_name):
self.remove_file(full_name)
-
+
class PackagesDB:
@@@ -248,7 -210,7 +209,7 @@@
"unknown-preferred-alternative",
"no-dependency-from-alternatives-exists",
]
-
+
_dep_state_to_state = {
"failed-testing": "dependency-failed-testing",
"cannot-be-tested": "dependency-cannot-be-tested",
@@@ -272,7 -234,7 +233,7 @@@
self._package_state = {}
self.set_subdirs(ok="pass", fail="fail", evil="untestable",
reserved="reserved", morefail=["bugged"])
-
+
def set_subdirs(self, ok=None, fail=None, evil=None, reserved=None, morefail=None):
# Prefix all the subdirs with the prefix
if self.prefix:
@@@ -290,12 -252,12 +251,12 @@@
if morefail:
self._morefail = [pformat % s for s in morefail]
self._all = [self._ok, self._fail, self._evil, self._reserved] + self._morefail
-
+
def create_subdirs(self):
for sdir in self._all:
if not os.path.exists(sdir):
os.makedirs(sdir)
-
+
def read_packages_file(self, input):
self._packages_files.append(PackagesFile(input))
self._packages = None
@@@ -326,13 -288,13 +287,13 @@@
deps.append(dep)
if dep in self._packages:
more += self._packages[dep].dependencies()
-
+
# Break circular dependencies
if break_circles and package["Package"] in deps:
deps.remove(package["Package"])
return deps
-
+
def _compute_package_state(self, package):
if self._logdb.log_exists(package, [self._ok]):
return "successfully-tested"
@@@ -433,7 -395,7 +394,7 @@@
deps.remove(pkg)
if package["Package"] in deps:
return "circular-dependency" # actually, it's a unknown circular-dependency
-
+
# treat circular-dependencies as testable (for the part of the circle)
state = "unknown"
if package["Package"] in self._known_circular_depends:
@@@ -451,7 -413,7 +412,7 @@@
def _compute_package_states(self):
if self._in_state is not None:
return
-
+
todo = []
unpreferred_alt = []
@@@ -493,7 -455,7 +454,7 @@@
self._in_state["unknown"] = todo
self._in_state["unknown-preferred-alternative"] = unpreferred_alt
-
+
for state in self._states:
self._in_state[state].sort()
@@@ -502,12 -464,11 +463,11 @@@
def get_pkg_names_in_state(self, state):
self._compute_package_states()
- return self._in_state[state]
-
- def get_packages_in_state(self, state):
- self._compute_package_states()
- return unique([self._packages[name] for name in self._in_state[state]])
-
+ return set(self._in_state[state])
+
+ def get_package(self, name):
+ return self._packages[name]
+
def get_all_packages(self):
self._find_all_packages()
return self._packages
@@@ -548,14 -509,13 +508,13 @@@
return "unknown"
def _find_packages_ready_for_testing(self):
- return self.get_packages_in_state("waiting-to-be-tested")
+ return self.get_pkg_names_in_state("waiting-to-be-tested")
def reserve_package(self):
- plist = self._find_packages_ready_for_testing()
- random.shuffle(plist)
- for p in plist:
- if self._logdb.create(self._reserved, p["Package"],
- p["Version"], ""):
+ pset = self._find_packages_ready_for_testing()
+ while (len(pset)):
+ p = self.get_package(pset.pop())
+ if self._logdb.create(self._reserved, p["Package"], p["Version"], ""):
return p
return None
@@@ -594,5 -554,3 +553,5 @@@
else:
raise Exception("Log file exists already: %s (%s)" %
(package, version))
+
+# vi:set et ts=4 sw=4 :
--
piuparts git repository
More information about the Piuparts-commits
mailing list