[Piuparts-commits] rev 322 - piatti trunk trunk/debian trunk/piupartslib
Holger Levsen
holger at alioth.debian.org
Sun Apr 5 23:24:38 UTC 2009
Author: holger
Date: 2009-04-05 23:24:37 +0000 (Sun, 05 Apr 2009)
New Revision: 322
Modified:
piatti/README.txt
trunk/TODO
trunk/debian/changelog
trunk/piuparts-report.py
trunk/piupartslib/packagesdb.py
Log:
- generate sources.txt (per section) with state summary per source package
(for the PTS to display a source packages piuparts state)
- generate html status pages for all source packages (to link from the PTS)
Modified: piatti/README.txt
===================================================================
--- piatti/README.txt 2009-04-05 20:07:18 UTC (rev 321)
+++ piatti/README.txt 2009-04-05 23:24:37 UTC (rev 322)
@@ -3,6 +3,8 @@
== ToDo
+- improve slave-watcher cronjob entry, so that i checks every hour whether tail still looks the same, if so mail it, else mail the full day diff at midnight only -> less frequent mails :)
+- review sudoers and come up with (a bit) more restrictive one
- use local mirror
- there should be a 2nd group of piuparts-people. those who can sudo into piupartsm to process logfiles. maybe make that the qa-group
@@ -28,7 +30,7 @@
=== piuparts installation from svn source
-* sudo apt-get install apt python debootstrap lsof lsb-release python-debian make dpkg-dev docbook2x python-support docbook-xml asciidoc dblatex
+* sudo apt-get install apt python debootstrap lsof lsb-release python-debian make dpkg-dev docbook2x python-support docbook-xml dblatex docbook-xsl asciidoc
* you need a webserver too, if you run the master
* Copy 'svn://svn.debian.org/svn/piuparts/piatti/home/piupartss/bin/update-piuparts-setup' on the host and run it. It assumes you want to set it up in '/org/piuparts.debian.org' and does all further svn checkouts as well as source code installation. It needs the piupartss and piupartsm user set up as described below, though.
* sudo ln -s /org/piuparts.debian.org/etc/ /etc/piuparts
Modified: trunk/TODO
===================================================================
--- trunk/TODO 2009-04-05 20:07:18 UTC (rev 321)
+++ trunk/TODO 2009-04-05 23:24:37 UTC (rev 322)
@@ -4,16 +4,26 @@
must for 0.36
- report:
--- create $section_packages.txt with into about all packages
--- write_packages_summary() is not used atm
--- write stats about the reasons for failures
+--- dont do source stats if source-url not defined, document that in readme.txt
+--- find_log() is not used atm
+--- include links to logfiles in source summary pages
+--- include links to http://qa.debian.org/developer.php?login=$maintainer_email
- then R
- dont write counts.txt if it has already been written that day
- take care of old conf files on upgrades
- take care of old pyc files from python-central
+- rewrite manpage in asciidoc and get rid of some build-depends. also dont build readme.pdf
+- test pipuparts with piuparts before uploading
for 0.37 and on
+- report: write stats about the reasons for failures
+
+- create emacspeak-broken-dpkg-preconfigure package for broken repo. (then later
+ put more broken packages in there and use that for testing piuparts)
+
+- write slave-watcher to monitor hanging processes, eg. looping dpkg-preconfigure
+
- Check for and kill extraneous processes afterwards. Perhaps by checking
whether their working directory is in the chroot.
Introduce a whitelist of processes to wait for and assume it's an error
Modified: trunk/debian/changelog
===================================================================
--- trunk/debian/changelog 2009-04-05 20:07:18 UTC (rev 321)
+++ trunk/debian/changelog 2009-04-05 23:24:37 UTC (rev 322)
@@ -40,6 +40,9 @@
- generate counts.txt (per section) with raw numbers of packages per state,
include these stats in the debug output (so it gets included in the mails
send by cron.)
+ - generate sources.txt (per section) with state summary per source package
+ (for the PTS to display a source packages piuparts state)
+ - generate html status pages for all source packages (to link from the PTS)
* Break backwards compatibility of the configuration files for master-slave-
mode. Merge those three into one: /etc/piuparts/piuparts/piuparts.conf.
Introduce a new global section in piuparts.conf which is shared among the
Modified: trunk/piuparts-report.py
===================================================================
--- trunk/piuparts-report.py 2009-04-05 20:07:18 UTC (rev 321)
+++ trunk/piuparts-report.py 2009-04-05 23:24:37 UTC (rev 322)
@@ -106,6 +106,11 @@
</td>
</tr>
$section_navigation
+ <tr>
+ <td class="contentcell">
+ <a href="http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=piuparts;users=debian-qa@lists.debian.org&archive=both" target="_blank">Bugs filed</a>
+ </td>
+ </tr>
<tr class="titlerow">
<td class="titlecell">
Other Debian QA efforts
@@ -143,10 +148,16 @@
</tr>
<tr class="normalrow">
<td class="contentcell">
- <a href="/doc" target="_blank">piuparts README</a>
+ <a href="/doc/README.html" target="_blank">piuparts README</a>
</td>
</tr>
<tr class="titlerow">
+ <tr class="normalrow">
+ <td class="contentcell">
+ <a href="/doc/piuparts.html" target="_blank">piuparts manpage</a>
+ </td>
+ </tr>
+ <tr class="titlerow">
<td class="titlecell">
Last update
</td>
@@ -265,6 +276,26 @@
</table>
"""
+SOURCE_PACKAGE_BODY_TEMPLATE = """
+ <table class="righttable">
+ <tr class="titlerow">
+ <td class="titlecell" colspan="2">
+ Source package
+ </td>
+ </tr>
+ <tr class="normalrow">
+ <td class="contentcell2" colspan="2">
+ $source
+ </td>
+ </tr>
+ <tr class="titlerow">
+ <td class="titlecell" colspan="2">
+ Binary package(s) in $section
+ </td>
+ </tr>
+ $binaryrows
+ </table>
+"""
INDEX_BODY_TEMPLATE = """
<table class="righttable">
@@ -375,6 +406,7 @@
"sections": "report",
"output-directory": "html",
"packages-url": None,
+ "sources-url": None,
"master-directory": ".",
"description": "",
}, "")
@@ -411,6 +443,12 @@
return str
+def path_to_source_summary_page(source):
+ if source[:3] == "lib":
+ return source[:4]
+ else:
+ return source[:1]
+
def find_log_files(dir):
return [name for name in os.listdir(dir) if name.endswith(".log")]
@@ -472,12 +510,19 @@
logging.debug("-------------------------------------------")
logging.debug("Running section " + self._config.section)
logging.debug("Loading and parsing Packages file")
+
logging.info("Fetching %s" % self._config["packages-url"])
packages_file = piupartslib.open_packages_url(self._config["packages-url"])
- self._st = piupartslib.packagesdb.PackagesDB()
- self._st.read_packages_file(packages_file)
+ self._binary_db = piupartslib.packagesdb.PackagesDB()
+ self._binary_db.read_packages_file(packages_file)
packages_file.close()
+ logging.info("Fetching %s" % self._config["sources-url"])
+ sources_file = piupartslib.open_packages_url(self._config["sources-url"])
+ self._source_db = piupartslib.packagesdb.PackagesDB()
+ self._source_db.read_packages_file(sources_file)
+ sources_file.close()
+
def write_log_list_page(self, filename, title, preface, logs):
packages = {}
for pathname, package, version in logs:
@@ -544,7 +589,7 @@
logging.debug("Writing section statistics page")
tablerows = ""
- for state in self._st.get_states():
+ for state in self._binary_db.get_states():
dir_link = ""
for dir in dirs:
if state_by_dir[dir] == state:
@@ -552,10 +597,10 @@
tablerows += ("<tr class=\"normalrow\"><td class=\"contentcell2\"><a href='state-%s.html'>%s</a></td>" +
"<td class=\"contentcell2\">%d</td><td class=\"contentcell2\">%s</td></tr>\n") % \
(html_protect(state), html_protect(state),
- len(self._st.get_packages_in_state(state)),
+ len(self._binary_db.get_packages_in_state(state)),
dir_link)
tablerows += "<tr class=\"normalrow\"> <td class=\"labelcell\">Total</td> <td class=\"labelcell\" colspan=\"2\">%d</td></tr>\n" % \
- self._st.get_total_packages()
+ self._binary_db.get_total_packages()
htmlpage = string.Template(HTML_HEADER + SECTION_STATS_BODY_TEMPLATE + HTML_FOOTER)
write_file(os.path.join(self._output_directory, "index.html"), htmlpage.safe_substitute( {
"section_navigation": create_section_navigation(self._section_names),
@@ -566,10 +611,10 @@
"packagesurl": html_protect(self._config["packages-url"]),
}))
- for state in self._st.get_states():
+ for state in self._binary_db.get_states():
logging.debug("Writing page for %s" % state)
list = "<ul>\n"
- for package in self._st.get_packages_in_state(state):
+ for package in self._binary_db.get_packages_in_state(state):
list += "<li>%s (%s)" % (html_protect(package["Package"]),
html_protect(package["Maintainer"]))
if package.dependencies():
@@ -577,7 +622,7 @@
for dep in package.dependencies():
list += "<li>dependency %s is %s</li>\n" % \
(html_protect(dep),
- emphasize_reason(html_protect(self._st.state_by_name(dep))))
+ emphasize_reason(html_protect(self._binary_db.state_by_name(dep))))
list += "</ul>\n"
list += "</li>\n"
list += "</ul>\n"
@@ -595,8 +640,8 @@
logging.debug("Writing counts.txt")
header = "date"
counts = "%s" % time.strftime("%Y%m%d")
- for state in self._st.get_states():
- count = len(self._st.get_packages_in_state(state))
+ for state in self._binary_db.get_states():
+ count = len(self._binary_db.get_packages_in_state(state))
header += ", %s" % state
counts += ", %s" % count
logging.debug("%s: %s" % (state, count))
@@ -608,29 +653,51 @@
append_file("counts.txt", counts)
def find_log(self, package):
- n = self._db._logdb._log_name(package["Package"], package["Version"])
+ n = self._binary_db._logdb._log_name(package["Package"], package["Version"])
for dirname in self._db._all:
nn = os.path.join(dirname, n)
if os.path.exists(nn):
return nn
return None
- def write_packages_summary(self):
- fd, name = tempfile.mkstemp(prefix="packages.txt.", dir=".")
- os.close(fd)
- os.chmod(name, 0644)
+ def write_sources_summaries(self):
+ logging.debug("Writing source package summaries")
+ sources = ""
+ for source in self._source_db.get_all_packages():
+ binaries = self._source_db.get_binary_package_names(source)
+ success = True
+ failed = False
+ binaryrows = ""
+ for binary in binaries.split(", "):
+ state = self._binary_db.state_by_name(binary)
+ binaryrows += "<tr class=\"normalrow\"><td class=\"contentcell2\">%s</td><td class=\"contentcell2\">%s</td></tr>" % (binary, state)
+ if state != "successfully-tested":
+ success = False
+ if state == "failed-testing":
+ failed = True
+ source_state="unknown"
+ if success: source_state="success"
+ if failed: source_state="failed"
+ sources += "%s: %s\n" % (source, source_state)
+ summary_page_path = os.path.join(self._output_directory, "source", path_to_source_summary_page(source))
+ if not os.path.exists(summary_page_path):
+ os.makedirs(summary_page_path)
+ filename = os.path.join(summary_page_path, (source + ".html"))
+ htmlpage = string.Template(HTML_HEADER + SOURCE_PACKAGE_BODY_TEMPLATE + HTML_FOOTER)
+ f = file(filename, "w")
+ f.write(htmlpage.safe_substitute( {
+ "section_navigation": create_section_navigation(self._section_names),
+ "time": time.strftime("%Y-%m-%d %H:%M %Z"),
+ "source": html_protect(source),
+ "section": html_protect(self._config.section),
+ "binaryrows": binaryrows,
+ }))
+ f.close()
+ write_file("sources.txt", sources)
- f = file(name, "w")
- for pkgname in self._db._packages:
- state = self._db.state_by_name(pkgname)
- logname = self.find_log(self._db._packages[pkgname]) or ""
- f.write("%s %s %s\n" % (pkgname, state, logname))
- f.close()
-
-
def generate_file_output(self):
self.write_counts_summary()
- #self.write_packages_summary()
+ self.write_sources_summaries()
def generate_output(self, master_directory, output_directory, section_names):
self._section_names = section_names
Modified: trunk/piupartslib/packagesdb.py
===================================================================
--- trunk/piupartslib/packagesdb.py 2009-04-05 20:07:18 UTC (rev 321)
+++ trunk/piupartslib/packagesdb.py 2009-04-05 23:24:37 UTC (rev 322)
@@ -266,7 +266,7 @@
for provided in p.provides():
if provided not in self._packages:
self._packages[provided] = p
-
+
def _get_recursive_dependencies(self, package, break_circles=True):
assert self._packages is not None
deps = []
@@ -372,6 +372,16 @@
self._compute_package_states()
return [self._packages[name] for name in self._in_state[state]]
+ def get_all_packages(self):
+ self._find_all_packages()
+ return self._packages
+
+ def get_binary_package_names(self, packagename):
+ return self._packages[packagename]["Binary"]
+
+ def get_package_state(self, package_name):
+ return self._package_state[packagename]
+
def state_by_name(self, package_name):
if package_name in self._package_state:
return self._package_state[package_name]
More information about the Piuparts-commits
mailing list