[Piuparts-commits] rev 301 - piatti/home/piupartsm trunk trunk/debian
Holger Levsen
holger at alioth.debian.org
Mon Mar 23 15:08:48 UTC 2009
Author: holger
Date: 2009-03-23 15:08:48 +0000 (Mon, 23 Mar 2009)
New Revision: 301
Modified:
piatti/home/piupartsm/crontab
trunk/TODO
trunk/debian/changelog
trunk/debian/copyright
trunk/piuparts-master.py
trunk/piuparts-report.py
Log:
* piuparts-master.py: remove code to write statistics, that is done only in
piuparts-report.py now.
* piuparts-slave.py:
- generate counts.txt (per section) with raw numbers of packages per state.
+ * Update debian/copyright to reflect that piuparts is licenced under GPL2+.
Modified: piatti/home/piupartsm/crontab
===================================================================
--- piatti/home/piupartsm/crontab 2009-03-22 15:22:54 UTC (rev 300)
+++ piatti/home/piupartsm/crontab 2009-03-23 15:08:48 UTC (rev 301)
@@ -8,7 +8,7 @@
# create reports once a day
# (dinstall runs 1|7|13|19:52, so this is long after mirror pushes...)
#
-42 6 * * * PYTHONPATH=/org/piuparts.debian.org/lib/python2.4/site-packages:/org/piuparts.debian.org/lib/python2.5/site-packages nice python /org/piuparts.debian.org/share/piuparts/piuparts-report
+5 0 * * * PYTHONPATH=/org/piuparts.debian.org/lib/python2.4/site-packages:/org/piuparts.debian.org/lib/python2.5/site-packages nice python /org/piuparts.debian.org/share/piuparts/piuparts-report
42 7 * * * /home/piupartsm/bin/detect_stale_mounts
43 7 * * * /home/piupartsm/bin/detect_network_issues
Modified: trunk/TODO
===================================================================
--- trunk/TODO 2009-03-22 15:22:54 UTC (rev 300)
+++ trunk/TODO 2009-03-23 15:08:48 UTC (rev 301)
@@ -4,15 +4,19 @@
must for 0.36
- document globalised config variables this in README
-- rename packages.txt into $section_done_today.txt and include sections
+- better counts.txt:
+-- factorize st used in write_counts_summary() and in generate_html_output()
- create $section_packages.txt with into about all packages
+-- write_packages_summary() is not used atm
+- then R
- better header (link to FAQ, point out to read the bottom first..) for piuparts-logs
-- debug is on, even though its off in the config
- master.log is hardcoded in p-master.py?!
- take care of old conf files on upgrades
- take care of old pyc files from python-central
- scripts in /usr/share/piuparts/ probably a FHS violation
- make sure README.* gets included in the package
+- document the feature that you cannot turn off debug for piuparts.py ;-) or fix it (its possible to workaround it..)
+- same for piuparts-reports.py
- rm piuparts.css?
- the templates used by update-reports.py should be taken from /etc/piuparts/templates/
and not be included in the python source
Modified: trunk/debian/changelog
===================================================================
--- trunk/debian/changelog 2009-03-22 15:22:54 UTC (rev 300)
+++ trunk/debian/changelog 2009-03-23 15:08:48 UTC (rev 301)
@@ -22,6 +22,8 @@
are not testable without the circular dependent package, so in case of
failure analysing will be harder.
* Install scripts for master-slave mode without .py extension.
+ * piuparts-master.py: remove code to write statistics, that is done only in
+ piuparts-report.py now.
* piuparts-slave.py:
- support looping trough multiple sections in slave.conf via new config
value "sections". Thanks to Frank Ploss for writing this patch with me
@@ -35,6 +37,7 @@
value "sections".
- provide links to logfiles in statistics page.
- improve layout, generate navigation for the pages.
+ - generate counts.txt (per section) with raw numbers of packages per state.
* Break backwards compatibility of the configuration files for master-slave-
mode. Merge those three into one: /etc/piuparts/piuparts/piuparts.conf.
Introduce a new global section in piuparts.conf which is shared among the
@@ -53,6 +56,7 @@
restructured and updated to reflect the configuration changes in
master-slave mode.
* Add NEWS file.
+ * Update debian/copyright to reflect that piuparts is licenced under GPL2+.
-- Holger Levsen <holger at debian.org> Tue, 10 Mar 2009 15:23:59 +0100
Modified: trunk/debian/copyright
===================================================================
--- trunk/debian/copyright 2009-03-22 15:22:54 UTC (rev 300)
+++ trunk/debian/copyright 2009-03-23 15:08:48 UTC (rev 301)
@@ -8,9 +8,10 @@
Copyright (C) 2005-2008 Lars Wirzenius and others
piuparts is free software. You may copy it according to the
-GNU General Public License, version 2. A copy of the license
-is not included, but you can get one from most FTP sites that
-have GNU software, for example, prep.ai.mit.edu.
+GNU General Public License, version 2, or at your option, any
+later version. A copy of the license is not included, but you
+can get one from most FTP sites that have GNU software, for
+example, prep.ai.mit.edu.
On a Debian GNU/Linux system, the GPL version 2 can be found in
/usr/share/common-licenses/GPL-2.
Modified: trunk/piuparts-master.py
===================================================================
--- trunk/piuparts-master.py 2009-03-22 15:22:54 UTC (rev 300)
+++ trunk/piuparts-master.py 2009-03-23 15:08:48 UTC (rev 301)
@@ -178,64 +178,6 @@
self._db.make_package_untestable(args[0], args[1], log)
self._short_response("ok")
- def count_packages_in_states(self, states):
- count = 0
- for state in states:
- count += len(self._db.get_packages_in_state(state))
- return count
-
- def rename_if_newer_else_delete(self, orig, new):
- ok = True
- if os.path.exists(orig):
- st_orig = os.stat(orig)
- st_new = os.stat(new)
- ok = st_orig.st_mtime < st_new.st_mtime
- if ok:
- os.rename(new, orig)
- else:
- os.remove(new)
-
- def write_counts_summary(self):
- fd, name = tempfile.mkstemp(prefix="counts.txt.", dir=".")
- os.close(fd)
- os.chmod(name, 0644)
-
- failed = self.count_packages_in_states(self._failed_states)
- passed = self.count_packages_in_states(self._passed_states)
-
- f = file(name, "w")
- f.write("Fail: %d\n" % failed)
- f.write("Pass: %d\n" % passed)
- f.close()
- self.rename_if_newer_else_delete("counts.txt", name)
-
- def find_log(self, package):
- n = self._db._logdb._log_name(package["Package"], package["Version"])
- for dirname in self._db._all:
- nn = os.path.join(dirname, n)
- if os.path.exists(nn):
- return nn
- return None
-
- def write_packages_summary(self):
- fd, name = tempfile.mkstemp(prefix="packages.txt.", dir=".")
- os.close(fd)
- os.chmod(name, 0644)
-
- f = file(name, "w")
- for pkgname in self._db._packages:
- state = self._db.state_by_name(pkgname)
- logname = self.find_log(self._db._packages[pkgname]) or ""
- f.write("%s %s %s\n" % (pkgname, state, logname))
- f.close()
-
- self.rename_if_newer_else_delete("packages.txt", name)
-
- def write_summaries(self):
- self.write_counts_summary()
- #self.write_packages_summary()
-
-
def main():
# piuparts-master is always called by the slave with a section as argument
if len(sys.argv) == 2:
@@ -251,7 +193,6 @@
packages_file.close()
while m.do_transaction():
pass
- m.write_summaries()
if __name__ == "__main__":
Modified: trunk/piuparts-report.py
===================================================================
--- trunk/piuparts-report.py 2009-03-22 15:22:54 UTC (rev 300)
+++ trunk/piuparts-report.py 2009-03-23 15:08:48 UTC (rev 301)
@@ -445,6 +445,12 @@
f.close()
+def append_file(filename, contents):
+ f = file(filename, "a")
+ f.write(contents)
+ f.close()
+
+
def create_section_navigation(section_names):
tablerows = ""
for section in section_names:
@@ -458,6 +464,8 @@
def __init__(self, section):
self._config = Config(section=section)
self._config.read(CONFIG_FILE)
+ logging.debug("-------------------------------------------")
+ logging.debug("Running section " + self._config.section)
def write_log_list_page(self, filename, title, preface, logs):
packages = {}
@@ -507,9 +515,125 @@
desc_by_dir[dir], list)
- def output(self, master_directory, output_directory, section_names):
- logging.debug("-------------------------------------------")
- logging.debug("Running section " + self._config.section)
+ def generate_html_output(self):
+ logging.debug("Finding log files")
+ dirs = ["pass", "fail", "bugged", "fixed", "reserved", "untestable"]
+ logs_by_dir = {}
+ for dir in dirs:
+ logs_by_dir[dir] = find_log_files(dir)
+
+ logging.debug("Copying log files")
+ copy_logs(logs_by_dir, self._output_directory)
+
+ logging.debug("Removing old log files")
+ remove_old_logs(logs_by_dir, self._output_directory)
+
+ logging.debug("Writing per-dir HTML pages")
+ self.print_by_dir(self._output_directory, logs_by_dir)
+
+ logging.debug("Loading and parsing Packages file")
+ if 1:
+ logging.info("Fetching %s" % self._config["packages-url"])
+ packages_file = piupartslib.open_packages_url(self._config["packages-url"])
+ else:
+ packages_file = file("Packages")
+ st = piupartslib.packagesdb.PackagesDB()
+ st.read_packages_file(packages_file)
+ packages_file.close()
+
+ logging.debug("Writing section statistics page")
+ tablerows = ""
+ for state in st.get_states():
+ dir_link = ""
+ for dir in dirs:
+ if state_by_dir[dir] == state:
+ dir_link += "<a href='%s.html'>%s</a> logs<br>" % (dir, html_protect(dir))
+ tablerows += ("<tr class=\"normalrow\"><td class=\"contentcell2\"><a href='state-%s.html'>%s</a></td>" +
+ "<td class=\"contentcell2\">%d</td><td class=\"contentcell2\">%s</td></tr>\n") % \
+ (html_protect(state), html_protect(state),
+ len(st.get_packages_in_state(state)),
+ dir_link)
+ tablerows += "<tr class=\"normalrow\"> <td class=\"labelcell\">Total</td> <td class=\"labelcell\" colspan=\"2\">%d</td></tr>\n" % \
+ st.get_total_packages()
+ htmlpage = string.Template(HTML_HEADER + SECTION_STATS_BODY_TEMPLATE + HTML_FOOTER)
+ write_file(os.path.join(self._output_directory, "index.html"), htmlpage.safe_substitute( {
+ "section_navigation": create_section_navigation(self._section_names),
+ "time": time.strftime("%Y-%m-%d %H:%M %Z"),
+ "section": html_protect(self._config.section),
+ "description": html_protect(self._config["description"]),
+ "tablerows": tablerows,
+ "packagesurl": html_protect(self._config["packages-url"]),
+ }))
+
+ for state in st.get_states():
+ logging.debug("Writing page for %s" % state)
+ list = "<ul>\n"
+ for package in st.get_packages_in_state(state):
+ list += "<li>%s (%s)" % (html_protect(package["Package"]),
+ html_protect(package["Maintainer"]))
+ if package.dependencies():
+ list += "\n<ul>\n"
+ for dep in package.dependencies():
+ list += "<li>dependency %s is %s</li>\n" % \
+ (html_protect(dep),
+ emphasize_reason(html_protect(st.state_by_name(dep))))
+ list += "</ul>\n"
+ list += "</li>\n"
+ list += "</ul>\n"
+ htmlpage = string.Template(HTML_HEADER + STATE_BODY_TEMPLATE + HTML_FOOTER)
+ write_file(os.path.join(self._output_directory, "state-%s.html" % state), htmlpage.safe_substitute( {
+ "section_navigation": create_section_navigation(self._section_names),
+ "time": time.strftime("%Y-%m-%d %H:%M %Z"),
+ "state": html_protect(state),
+ "section": html_protect(self._config.section),
+ "list": list
+ }))
+
+ def write_counts_summary(self):
+ st = piupartslib.packagesdb.PackagesDB()
+ packages_file = piupartslib.open_packages_url(self._config["packages-url"])
+ st.read_packages_file(packages_file)
+ packages_file.close()
+
+ logging.debug("Writing counts.txt")
+ header = "date"
+ counts = "%s" % time.strftime("%Y%m%d")
+ for state in st.get_states():
+ header += ", %s" % state
+ counts += ", %s" % len(st.get_packages_in_state(state))
+ header += "\n"
+ counts += "\n"
+
+ if not os.path.isfile("counts.txt"):
+ write_file("counts.txt", header)
+ append_file("counts.txt", counts)
+
+ def find_log(self, package):
+ n = self._db._logdb._log_name(package["Package"], package["Version"])
+ for dirname in self._db._all:
+ nn = os.path.join(dirname, n)
+ if os.path.exists(nn):
+ return nn
+ return None
+
+ def write_packages_summary(self):
+ fd, name = tempfile.mkstemp(prefix="packages.txt.", dir=".")
+ os.close(fd)
+ os.chmod(name, 0644)
+
+ f = file(name, "w")
+ for pkgname in self._db._packages:
+ state = self._db.state_by_name(pkgname)
+ logname = self.find_log(self._db._packages[pkgname]) or ""
+ f.write("%s %s %s\n" % (pkgname, state, logname))
+ f.close()
+
+
+ def generate_file_output(self):
+ self.write_counts_summary()
+ #self.write_packages_summary()
+
+ def generate_output(self, master_directory, output_directory, section_names):
self._section_names = section_names
self._master_directory = os.path.abspath(os.path.join(master_directory, self._config.section))
if os.path.exists(self._master_directory):
@@ -521,81 +645,12 @@
oldcwd = os.getcwd()
os.chdir(self._master_directory)
- logging.debug("Finding log files")
- dirs = ["pass", "fail", "bugged", "fixed", "reserved", "untestable"]
- logs_by_dir = {}
- for dir in dirs:
- logs_by_dir[dir] = find_log_files(dir)
+ self.generate_html_output()
+ self.generate_file_output()
- logging.debug("Copying log files")
- copy_logs(logs_by_dir, self._output_directory)
+ os.chdir(oldcwd)
- logging.debug("Removing old log files")
- remove_old_logs(logs_by_dir, self._output_directory)
- logging.debug("Writing per-dir HTML pages")
- self.print_by_dir(self._output_directory, logs_by_dir)
-
- logging.debug("Loading and parsing Packages file")
- if 1:
- logging.info("Fetching %s" % self._config["packages-url"])
- packages_file = piupartslib.open_packages_url(self._config["packages-url"])
- else:
- packages_file = file("Packages")
- st = piupartslib.packagesdb.PackagesDB()
- st.read_packages_file(packages_file)
- packages_file.close()
-
- logging.debug("Writing section statistics page")
- tablerows = ""
- for state in st.get_states():
- dir_link = ""
- for dir in dirs:
- if state_by_dir[dir] == state:
- dir_link += "<a href='%s.html'>%s</a> logs<br>" % (dir, html_protect(dir))
- tablerows += ("<tr class=\"normalrow\"><td class=\"contentcell2\"><a href='state-%s.html'>%s</a></td>" +
- "<td class=\"contentcell2\">%d</td><td class=\"contentcell2\">%s</td></tr>\n") % \
- (html_protect(state), html_protect(state),
- len(st.get_packages_in_state(state)),
- dir_link)
- tablerows += "<tr class=\"normalrow\"> <td class=\"labelcell\">Total</td> <td class=\"labelcell\" colspan=\"2\">%d</td></tr>\n" % \
- st.get_total_packages()
- htmlpage = string.Template(HTML_HEADER + SECTION_STATS_BODY_TEMPLATE + HTML_FOOTER)
- write_file(os.path.join(self._output_directory, "index.html"), htmlpage.safe_substitute( {
- "section_navigation": create_section_navigation(self._section_names),
- "time": time.strftime("%Y-%m-%d %H:%M %Z"),
- "section": html_protect(self._config.section),
- "description": html_protect(self._config["description"]),
- "tablerows": tablerows,
- "packagesurl": html_protect(self._config["packages-url"]),
- }))
-
- for state in st.get_states():
- logging.debug("Writing page for %s" % state)
- list = "<ul>\n"
- for package in st.get_packages_in_state(state):
- list += "<li>%s (%s)" % (html_protect(package["Package"]),
- html_protect(package["Maintainer"]))
- if package.dependencies():
- list += "\n<ul>\n"
- for dep in package.dependencies():
- list += "<li>dependency %s is %s</li>\n" % \
- (html_protect(dep),
- emphasize_reason(html_protect(st.state_by_name(dep))))
- list += "</ul>\n"
- list += "</li>\n"
- list += "</ul>\n"
- htmlpage = string.Template(HTML_HEADER + STATE_BODY_TEMPLATE + HTML_FOOTER)
- write_file(os.path.join(self._output_directory, "state-%s.html" % state), htmlpage.safe_substitute( {
- "section_navigation": create_section_navigation(self._section_names),
- "time": time.strftime("%Y-%m-%d %H:%M %Z"),
- "state": html_protect(state),
- "section": html_protect(self._config.section),
- "list": list
- }))
-
- os.chdir(oldcwd)
-
def main():
setup_logging(logging.DEBUG, None)
@@ -613,7 +668,7 @@
sections = []
for section_name in section_names:
section = Section(section_name)
- section.output(master_directory=global_config["master-directory"],output_directory=global_config["output-directory"],section_names=section_names)
+ section.generate_output(master_directory=global_config["master-directory"],output_directory=global_config["output-directory"],section_names=section_names)
sections.append(section)
logging.debug("Writing index page")
More information about the Piuparts-commits
mailing list