[Piuparts-commits] [piuparts] 01/05: Update README_server.txt and README_pejacevic.txt and instances/* and conf/*.

Holger Levsen holger at moszumanska.debian.org
Sun Oct 26 19:19:22 UTC 2014 

This is an automated email from the git hooks/post-receive script.

holger pushed a commit to branch master
in repository piuparts.

commit 94703f3135411198798f53275fcafae05ebce7b3
Author: Holger Levsen <holger at layer-acht.org>
Date:   Sun Oct 26 19:44:26 2014 +0100

    Update README_server.txt and README_pejacevic.txt and instances/* and conf/*.
    
    * Update README_server.txt and README_pejacevic.txt after setting up a
      piuparts master/slave system from packages myself.
      and include all commands piupartss needs to run as root in
      conf/piuparts-slave.sudoers.
    * Update instances/piuparts.conf* to match what is in use for
      piuparts.debian.org today. Add two new systems, goldwasser and lamarr, as
      examples taken from piuparts development.
---
 README_pejacevic.txt               |  45 +++++++---------
 README_server.txt                  | 106 ++++++++++++++++++++++---------------
 conf/piuparts-slave.sudoers        |   5 +-
 debian/changelog                   |   9 +++-
 instances/piuparts.conf.goldwasser |   1 -
 instances/piuparts.conf.pejacevic  |   1 +
 6 files changed, 95 insertions(+), 72 deletions(-)

diff --git a/README_pejacevic.txt b/README_pejacevic.txt
index 4ba2a04..4c59f74 100644
--- a/README_pejacevic.txt
+++ b/README_pejacevic.txt
@@ -1,5 +1,5 @@
-Notes about the piuparts installation on pejacevic.debian.org and it's slave
-============================================================================
+Notes about the piuparts installation on pejacevic.debian.org and it's slave(s)
+===============================================================================
 
 This document describes the setup for https://piuparts.debian.org - it's used
 for reference for the Debian System Administrators (DSA) as well as a guide
@@ -10,9 +10,11 @@ piuparts-master and piuparts-slaves packages as described in
 
 == Installation
 
-piuparts.debian.org is a setup running on two systems: pejacevic.debian.org,
-running the piuparts-master instance and an apache webserver to display the
-results and piu-slave-bm-a.debian.org, running four piuparts-slave nodes.
+piuparts.debian.org is a setup running on several systems:
+pejacevic.debian.org, running the piuparts-master instance and an apache
+webserver to display the results and piu-slave-bm-a.debian.org, running
+four piuparts-slave nodes. Not yet in operation there is another,
+piu-slave-1und1-01.debian.org, which soon shall go into operation...
 
 === piuparts installation from source
 
@@ -25,7 +27,7 @@ results and piu-slave-bm-a.debian.org, running four piuparts-slave nodes.
   done for a long time to run them on the same host.(
   Run the scripts as the piupartsm and piupartss users and clone that git
   repository into '/srv/piuparts.debian.org/src' in the first place. Then
-  checkout the bikeshed branch.
+  checkout the develop branch.
 * Ideally provide '/srv/piuparts.debian.org/tmp' on (a sufficiently large)
   tmpfs.
 * `sudo ln -s /srv/piuparts.debian.org/etc/piuparts /etc/piuparts`
@@ -40,7 +42,7 @@ be chmod 2775 and chown piuparts(sm):piuparts.
 
 ==== '~/bashrc' for piupartsm and piupartss
 
-Do this for the piupartsm user on pejacevic and piupartss on the slave:
+Do this for the piupartsm user on pejacevic and piupartss on the slave(s):
 
 ----
 piupartsm at pejacevic$ cat >> ~/.bashrc <<-EOF
@@ -62,25 +64,18 @@ $ cat /etc/ssh/userkeys/piupartsm
 command="/srv/piuparts.debian.org/share/piuparts/piuparts-master",from="2001:41c8:1000:21::21:7,5.153.231.7",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa ...
 ----
 
-=== Setup sudo
+=== Setup sudo for the slave(s)
 
 This is actually done by DSA:
 
-==== '/etc/sudoers' for pejacevic
-
-----
-#piuparts admins
-%piuparts       ALL=(piupartsm) ALL
-----
-
-==== '/etc/sudoers' for piu-slave-bm-a
+==== '/etc/sudoers' for piu-slave-bm-a and piu-slave-1und1-01
 
 ----
 # The piuparts slave needs to handle chroots.
-piupartss       ALL = NOPASSWD: ALL
-
-#piuparts admins
-%piuparts       ALL=(piupartss) ALL
+piupartss       ALL = NOPASSWD: /usr/sbin/piuparts *, \
+                                /bin/umount /srv/piuparts.debian.org/tmp/tmp*, \
+                                /usr/bin/test -f /srv/piuparts.debian.org/tmp/tmp*, \
+                                /usr/bin/rm -rf --one-file-system /srv/piuparts.debian.org/tmp/tmp*
 ----
 
 === Apache configuration
@@ -116,13 +111,13 @@ Any other webserver will do but apache is used on pejacevic (and maintained by D
 Updating the master, pejacevic.debian.org:
 
 ----
-holger at pejacevic$ sudo su - piupartsm update-piuparts-master-setup bikeshed origin
+holger at pejacevic~$ sudo su - piupartsm update-piuparts-master-setup develop origin
 ----
 
-Updating the slave, piu-slave-bm-a.debian.org:
+Updating the slave(s), for example on piu-slave-bm-a.debian.org:
 
 ----
-holger at piu-slave-bm-a$ sudo su - piupartss update-piuparts-slave-setup bikeshed origin
+holger at piu-slave-bm-a~$ sudo su - piupartss update-piuparts-slave-setup develop origin
 ----
 
 === Running piuparts
@@ -137,7 +132,7 @@ piuparts-slave on pejacevic, piuparts-master will be started automatically by
 the slaves.
 
 ----
-holger at pejacevic:~$ sudo -u piupartss -i slave_run
+holger at piu-slave-bm-a:~$ sudo -u piupartss -i slave_run
 ----
 
 There are several cronjobs installed via '~piupartsm/crontab' and
@@ -182,7 +177,7 @@ More checks should be added as we become aware of them.
 
 == Authors
 
-Last updated: May 2014
+Last updated: October 2014
 
 Holger Levsen <holger at layer-acht.org>
 
diff --git a/README_server.txt b/README_server.txt
index 5bef355..c4ca02b 100644
--- a/README_server.txt
+++ b/README_server.txt
@@ -4,6 +4,16 @@ piuparts README_server
 Author: Lars Wirzenius, Holger Levsen and Andreas Beckmann
 Email: <debian-qa at lists.debian.org>
 
+=== piuparts runs itself and other stuff as root
+
+WARNING: Please note that running piuparts on unknown packages is somewhat
+risky, to say the least. There are security implications that you want to
+consider. It's best to do it on machines that you don't mind wiping clean
+at a moment's notice, and preferably so that they don't have direct network
+access.
+
+You have been warned.
+
 == piuparts in master/slave mode
 
 As part of the quality assurance efforts of Debian, piuparts is
@@ -18,54 +28,66 @@ of packages it has tested already, and to get more work.
 
 To set this up for yourself, the following steps should suffice:
 
+=== Setting up the master
+
 . Pick a machine for running the piuparts master. It cannot be a chroot, but
  basically any real (or properly virtualized) Debian system is good enough.
-. Install the package piuparts-master on it.
+. Install the package 'piuparts-master' on it.
 . Create an account for the master, if you install the piuparts-master package
- it will automatically create a piupartsm user for you.
+ it will automatically create a 'piupartsm' user for you.
 . Configure '/etc/piuparts/piuparts.conf' appropriately.
+. Create the master and backup directories as defined in that 'piuparts.conf'
+ and make sure master owns them.
+. To generate the web reports, configure your webserver as needed. If you
+ want to use the supplied 'conf-available/piuparts-master.conf' for apache2,
+ you will need to do two things: a.) enable it and b.) link the htdocs
+ directory defined in 'piuparts.conf' to '/var/lib/piuparts/htdocs'
+ (thats the DocumentRoot as defined in 'conf-available/piuparts-master.conf').
+
+=== Setting up the slave(s)
 
 . Pick one or more machines for running one or several piuparts slaves. You
  can use the machine which is running the master also for running a slave.
  It's also perfectly ok to run several slaves on a multi-core machine which
  has lots of IO available.
-. Install the package piuparts-slave on it.
+. Install the package 'piuparts-slave' on it.
 . Configure '/etc/piuparts/piuparts.conf' appropriately - if master
  and slave share the machine, they also share the config file.
+ If you want to run more than one slave on a machine, set the slave-count
+ parameter as desired. By default one slave will be run.
+. Create the slave and tmp directories as defined in that 'piuparts.conf' and
+ make sure the slave can read and write there.
 . Create an account for the slave. This must be different from the master
- account. The piuparts-slave package will create a piupartss user on
- installation.
-. Create an ssh keypair for the slave. No passphrase.
-. Add the slave's public key to the master's '.ssh/authorized_keys'
- The key should be restricted to only allow running
- 'piuparts-master' by prefixing it with
- 'command="/usr/share/piuparts/piuparts-master",no-pty,no-port-forwarding'
-. Configure sudo to allow the slave account to run '/usr/sbin/piuparts'
- as root without password. There are examples provided in
- /usr/share/doc/piuparts-(master|slave)/examples/.
-. Run '/usr/bin/piuparts-slave-run' and 'piuparts-slave-join' to actually
+ account. The piuparts-slave package will create a 'piupartss' user on
+ installation. Whether you run one or many slaves, they run with the same
+ user.
+. Create an ssh keypair for the slave. No passphrase. If you installed the
+ piuparts-slave package this was done automatically and the public key can
+ be found in '/var/lib/piuparts/piupartss/.ssh/id_rsa.pub'
+. Copy the slave's public key to the master's '.ssh/authorized_keys', for
+ an installation from packages this will be
+ '/var/lib/piuparts/piupartsm/.ssh/authorized_keys'.
+ The key should be restricted to only allow running 'piuparts-master'
+ by prefixing it with
+ 'command="/usr/share/piuparts/piuparts-master",no-port-forwarding,no-X11-forwarding,no-agent-forwarding '
+. Configure sudo to allow the slave account to run several commands as root
+ as root without password. See the example provided in
+ '/usr/share/doc/piuparts-slave/examples/' to learn which.
+. Run '/usr/bin/piuparts_slave_run' and 'piuparts_slave_join' to actually
  let the slave(s) run and to join their sessions.
 . The logs go into the master account, into subdirectories.
 
-=== Setup from piuparts-master and piuparts-slaves packages
+=== Tuning the setup
 
 The piuparts-server package installs a piuparts server along the lines of
 https://piuparts.debian.org/.
 
-Before running the server, edit /etc/piuparts.conf appropriately (install
-piuparts-slave (which ships that file), too, or use the template
-/usr/share/doc/piuparts-master/piuparts.conf.sample), to define
-'sections' to be tested (e.g. 'sid') and define references to the Debian
-mirror. Note that the server can place a significant load on the
-repository. Consider setting up a local mirror, or a caching proxy for http
-and apt-get, to reduce the load. Running multiple slaves on a fast host can
-easily saturate a 100 MBit link.
-
-Edit '/etc/sudoers.d/piuparts' to grant permissions to the piupartss user.
-Start the server using /usr/bin/piuparts_slave_run, which will launch a
-'screen' session. The slave will launch a master process via ssh, as needed,
-to retrieve work and return results. Use /usr/bin/piuparts_slave_join to
-join the screen session.
+Custome '/etc/piuparts/piuparts.conf' according to your needs, most probably
+you will want to re-define the 'sections' to be tested (e.g. 'sid') and also
+maybe use a different Debian mirror. Note that the server can place a
+significant load on the repository. Consider setting up a local mirror,
+or a caching proxy for http and apt-get, to reduce the load. Running multiple
+slaves on a fast host can easily saturate a 100 MBit link.
 
 Logs are stored under '/var/lib/piuparts' by default. They are stored there
 because they are basically the result of piuparts running.
@@ -80,15 +102,8 @@ http://localhost/piuparts to be served by any webserver.
 https://piuparts.debian.org has been set up directly from GIT, this is
 described in '/usr/share/doc/piuparts-master/README_pejacevic.txt'.
 
-== Distributed testing
 
-WARNING: Please note that running piuparts this way is somewhat risky, to
-say the least. There are security implications that you want to
-consider. It's best to do it on machines that you don't mind
-wiping clean at a moment's notice, and preferably so that they
-don't have direct network access.
-
-=== Distributed piuparts testing protocol
+== Distributed piuparts testing protocol
 
 The slave machine and the piuparts-master program communicate
 using a simplistic line based protocol. SSH takes care of
@@ -245,7 +260,7 @@ The master may likewise abort, without an error message, if the
 slave sends garbage, or sends too much data.
 
 
-=== piuparts.conf configuration file
+== piuparts.conf configuration file
 
 piuparts-master, piuparts-slave and piuparts-report share the
 configuration file '/etc/piuparts/piuparts.conf'. The syntax is
@@ -257,7 +272,7 @@ this:
     foo = bar
 ----
 
-==== global configuration
+=== global configuration
 
 These settings have to be placed in the [global] section and are
 used for all further sections.
@@ -317,7 +332,7 @@ used for all further sections.
  "http://localhost:3128") due to the high bandwidth consumption of
  piuparts and repeated downloading of the same files.
 
-==== section specific configuration
+=== section specific configuration
 
 The section specific settings will be reloaded each time a section
 is being run. All these keys can be specified in the [global]
@@ -524,7 +539,7 @@ section, too, and will serve as defaults for all other sections
 Some of the configuration items are not required, but it is best
 to set them all to be sure what the configuration actually is.
 
-==== piuparts.debian.org specific configuration
+=== piuparts.debian.org specific configuration
 
 In addition to some of the above settings the following
 configuration settings are used by the scripts in '~piuparts?/bin/'
@@ -535,8 +550,7 @@ values are set in the scripts.
  piuparts instance. Used to provide links to logfiles in email
  reports. It defaults to "https://piuparts.debian.org".
 
-
-=== Running piuparts-report as it is done for piuparts.debian.org
+== Running piuparts-report as it is done for piuparts.debian.org
 
 If you want to run piuparts-report (which is only+very useful if
 you run piuparts in master-slave mode), you need to 'apt-get
@@ -544,4 +558,10 @@ install python-rpy r-recommended r-base-dev'. For more
 information see
 link:https://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_pejacevic.txt[https://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_pejacevic.txt].
 
+To generate the report on the master host run:
+
+----
+piupartsm at goldwasser:~$ /usr/share/piuparts/master/generate_daily_report
+----
+
 // vim: set filetype=asciidoc:
diff --git a/conf/piuparts-slave.sudoers b/conf/piuparts-slave.sudoers
index fba2441..c19ef9f 100644
--- a/conf/piuparts-slave.sudoers
+++ b/conf/piuparts-slave.sudoers
@@ -3,5 +3,8 @@
 #
 
 # The piuparts slave needs to handle chroots.
-piupartss	ALL = NOPASSWD: /usr/sbin/piuparts
+piupartss	ALL = NOPASSWD: /usr/sbin/piuparts *, \
+				/bin/umount /srv/piuparts.debian.org/tmp/tmp*, \
+				/usr/bin/test -f /srv/piuparts.debian.org/tmp/tmp*, \
+				/usr/bin/rm -rf --one-file-system /srv/piuparts.debian.org/tmp/tmp*
 
diff --git a/debian/changelog b/debian/changelog
index 0ad6737..8dd4436 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -10,9 +10,14 @@ piuparts (0.60) UNRELEASED; urgency=medium
   * tests: Remove the Ubuntu devel release test.
 
   [ Holger Levsen ]
+  * Update README_server.txt and README_pejacevic.txt after setting up a
+    piuparts master/slave system from packages myself.
   * Drop conf/piuparts-master.sudoers from debian/piuparts-master.examples
-    and modify conf/piuparts-slave.sudoers to only allow the piupartss user
-    to run /usr/sbin/piuparts as root - that's "all that's needed" anyway.
+    and include all commands piupartss needs to run as root in
+    conf/piuparts-slave.sudoers.
+  * Update instances/piuparts.conf* to match what is in use for
+    piuparts.debian.org today. Add two new systems, goldwasser and lamarr, as
+    examples taken from piuparts development.
   * Add "AddType text/plain .log" to the apache2 configuration snipplet
     example, so that logfiles are displayed in the browser.
   * Bump standards version to 3.9.6, no changes needed.
diff --git a/instances/piuparts.conf.goldwasser b/instances/piuparts.conf.goldwasser
index ce4e6b2..fdd9029 100644
--- a/instances/piuparts.conf.goldwasser
+++ b/instances/piuparts.conf.goldwasser
@@ -83,7 +83,6 @@ backup-directory = /srv/piuparts.debian.org/backup
 tmpdir = /srv/piuparts.debian.org/tmp
 doc-root = /piuparts
 components = main
-slave-count = 2
 # this value is too high for production (it will cause piuparts-slave to sleep
 # for a whole day) but useful for testing master-slave mode, for running on a
 # test system 24/7 without causing load for 24/7
diff --git a/instances/piuparts.conf.pejacevic b/instances/piuparts.conf.pejacevic
index 24bef9d..9520adf 100644
--- a/instances/piuparts.conf.pejacevic
+++ b/instances/piuparts.conf.pejacevic
@@ -100,6 +100,7 @@ backup-directory = /srv/piuparts.debian.org/backup
 tmpdir = /srv/piuparts.debian.org/tmp
 doc-root = /
 components = main
+# the slave-count setting is for the slave(s)
 slave-count = 4
 # 30*60
 idle-sleep = 1800

-- 
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/piuparts/piuparts.git

More information about the Piuparts-commits mailing list