[Soc-coordination] Backend Tools and Infrastructure for DEX - Report 2

[Nathan Handler]

Hello everyone,

A lot of work has taken place since my last report. First, since my
original mentor, Matt Zimmerman, is going to be moving, he helped
arrange for Stefano Zacchiroli to fill in for him temporarily. I have
had a call with Stefano, and I am looking forward to getting to work
with him more over the next couple of weeks.

Shortly after my first report, we realized that having the dashboard
only display bugs severely limited its usefulness. In our original
design, we had it pulling a list of bugs with specific usertags from
the BTS. This forced all information to be added to bugs, which we
viewed as a good thing since it made the information available to the
Debian Maintainers without requiring them to familiarize themselves
with DEX. The issue was that when looking back at the ancient-patches
project or forward to the big merges project, a large portion of the
work will be triaging and determining whether or not the changes are
even applicable in Debian. If they aren't, there is no reason for a
bug to be filed on the BTS. As a result, I spent a fair amount of time
re-writing most of the dashboard. It still supports finding bugs on
the BTS that have been tagged with a specific DEX usertag, but it is
now task-based rather than bug-based.

A task can be a bug number, a package name, a patch name, or anything.
It is merely something that needs to be done for a project. Each task
will have a name, an assignee, a status, a note, and a bug associated
with it. There are two ways to define a list of tasks for a project.
The first is to tag bugs on the BTS with a user of
'debian-derivatives at lists.debian.org' and a tag with a format of
'dex-<distribution>-<project>' (i.e. dex-ubuntu-ancient-patches). The
script will convert the bugs to tasks and automatically set all of the
fields. The second method is to specify the list of tasks in a plain
text file on the server the scripts are being run on. Eventually, this
list will be able to be specified via a web interface. Currently, you
are only able to specify the task name in the file. This means that
all of the other fields for a task created in this manner are
initially blank. I am still deciding on the best way to specify
additional information about each task in the file. Once I decide on a
method, it should not be too difficult to modify the script to parse
and store that additional information.

For the ancient patches DEX project, everyone involved had to update a
status file stored in a VCS on alioth. This was rather annoying, so we
decided early on that we want to be prevent this from being necessary
for future projects. This means that the dashboard needs a way for
users to modify the information on it. For tasks that are based on
bugs, this is rather easy; you simply use the BTS email interface to
adjust the bug. However, for other tasks, we needed the ability to
modify them on the dashboard. I have managed to get this partially
implemented. Currently, all cells on the table are text input fields
(with css being used to hide the boxes). This means that you can click
on any cell and edit its contents. Hitting the submit button will
execute a script to save these changes. This is the tricky part. The
dashboard is a collection of html files generated by a python script.
The script is meant to be run by cron every n minutes. When you update
something via a web interface, you usually expect to see the changes
instantly. This means that we can't wait for the python script to run
again and re-generate the html files. As a result, I currently have it
setup so that when you submit the form, it will go and modify the html
files to reflect the changes. The next time cron runs the python
script, the html files will be re-created, and the changes made by the
user will be lost. Once I come up with a method for storing additional
information about a task, it shouldn't be too hard to make changes
made by a user via the web interface permanent.

I also did some work on a script to generate graphs. There are two
types of graphs I hope to be able to display on the dashboard. One
graph will help show which individuals have done the most work on a
project. I have not started working on this graph yet. The second type
of graph is meant to show the number of open tasks versus time. This
will allow people to quickly see the progress of a project, estimate
how long the project will take to complete, and notice if things start
to slow down. I have some initial code for this graph using
matplotlib. However, due to the many changes that were made to the
dashboard, I had to hold off a bit on this graph, as it relies on
certain information provided by the dashboard. I hope to sort out some
of the last few issues with this graph, such as getting the axes to
display properly, and then merge it with the dashboard sometime in the
next few weeks.

Allison Randal approached me a while ago to see if my Summer of Code
project might be in a state where it could be used to assist with a
Python 2.7 DEX project that she was working on. For a while, I did not
think that it would be ready. However, last week, I helped her get the
related bugs usertagged appropriately, and they now show up in the DEX
dashboard. While the dashboard is still not complete, it does allow
her and everyone else working on the project to clearly see the tasks
(bugs) that they need to focus on and the current status of each of
them. I plan to try and incorporate any feedback I receive from people
working on this project into my designs.

All of my code is available in a git repository on alioth [1]. Keep in
mind that it should not be considered stable at this point. Further
status updates will be posted on my blog [2], and I will try to keep
the wiki page [3] updated. I am still trying work out where we will
host the actual dashboard. In the meantime, you can try out my testing
version of it [4].

During the next few weeks, I will be focusing my efforts on coming up
with a method for storing additional task information. That will allow
me to support permanently storing changes made by users via the web
interface. I also hope to finish the first graph script and merge it
with the dashboard. Finally, I hope to find a permanent home for the
dashboard, as having it on my personal server is not a viable
long-term solution.

Finally, if anyone is interested in getting involved with DEX, feel
free to join the debian-derivatives mailing list, join
#debian-derivatives on OFTC, or contact me via email or on IRC
(nhandler).

Thanks,
Nathan Handler

[1] http://anonscm.debian.org/gitweb/?p=dex/gsoc2011.git
[2] http://blog.nhandler.com/
[3] http://wiki.debian.org/dextools
[4] http://nhandler.com/gsoc/projects/dex.html

Blog Posts Since Last Report:

Google Summer of Code Update #3:
http://blog.nhandler.com/2011/06/10/google-summer-of-code-update-3/
Last week was a rather slow week. I did some work on allowing the
dashboard to store and process information that does not come from the
BTS. This exposed a few issues with the way we were storing and
presenting information. After talking to Matt, we decided that the
dashboard should deal with task objects. These tasks would contain
basic information such as a name, person assigned to it, status, and
possibly a link to a bug. This means that we would be able to start up
a new DEX project by simply choosing a project name and creating a
basic text file containing a list of tasks (package names, patches,
bugs, etc).

I also have been trying to get a hold of some people who participated
in the last DEX project. I am interested in getting some more feedback
about what went well, problems they encountered, and tools that might
have made things easier. I am also talking to some people who are
currently working on starting up new DEX projects to try and make
dextools align with as many workflows as possible. If you fall into
either of these categories, I would really appreciate it if you
contact me.

There is also a new wiki page that I created so that it is easier for
people who are not interested in me or the Google Summer of Code to
quickly see what dextools is about and monitor its progress. I will be
adding more information over the next few days.

Currently, I am continuing to work on the dashboard and talking to
people about their DEX experiences. I have also begun working on some
scripts to generate graphs that will go on the dashboard. These graphs
will allow us to quickly check our progress on different projects.
They will also allow us to see and acknowledge individuals working on
the tasks. I will add more information about these graphs to this blog
and to the dextools wiki page.


Google Summer of Code Update #4:
http://blog.nhandler.com/2011/06/16/google-summer-of-code-update-4/
Since Matt is going to be busy the next few weeks moving and
traveling, he arranged for Stefano Zacchiroli to fill in for him
temporarily as my mentor. I must say, I am quite lucky to have the
chance to have both Matt and Stefano as mentors this summer.
Yesterday, I had my first call with Stefano. I started by filling him
in on the current status of the project. We discussed how the
dashboard is going to be based on 'tasks' rather than 'bug'. He
reminded me that we want to make sure all useful information makes its
way back to the BTS (ideally in an automated or) in a fast and easy
way. Most of the information that will be stored in the dashboard
probably won't be that useful to the package maintainers. However,
there will be a column for DEX members to add notes about tasks. This
column could potentially contain some useful information. That is one
reason that the dashboard will parse this column for bug numbers and
then display extended information about the bug. Hopefully, this will
encourage DEX members to send information to the BTS when relevant.
Having the bug numbers will also allow us to automate sending
information from the DEX dashboard to the BTS if we choose to.

I also spent some time this week working on a script to generate some
graphs. These graphs will allow us to track the number of remaining
tasks in a project vs. time. Stefano mentioned possibly borrowing some
code from the Ubuntu burndown charts, but right now, I think the
Ubuntu charts might be a bit more complicated than what we need. We
can always decide to borrow some features/code down the road if
necessary.

It is also just about time for me to prepare my second formal report
for the Summer of Code. I will use that to showcase the current status
of the code and make it available for people to test and review.