[axel-devel] RFC: Roadmap for Axel 3

[Philipp Hagemeister]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi All,

I'd like to share some thoughts about the next version/future
development of Axel (not necessarily version 3). Most of the points are
already reported in the bug tracking system, but I'd like to give and
discuss an overview. Any specific numbers may be specific to my system
(x86, Linux), but should be about the same on other platforms.

Mission statement
=================

Axel is a lightweight Unix download accelerator. Lightweight means the
minimal axel binary size should not exceed 50 Kilobyte and should have
no dependencies but libc.

For comparison, on my system:

2642404 aria2c
 103988 curl (plus 248716 for libcurl)
 202588 wget
 38596 axel (1.1)
 47976 axel (2.1, r43)
 35644 axel (working copy, -Os)

Features
========

* User agent (#311073)
 Currently, Axel sends the HTTP User Agent header "Axel (2.1)
(Platform)" no matter what headers are present. Since HTTP forbids
double headers and some servers serve different responses for different
UAs, setting the UA would be a nice option. Possible solutions:

 [1] wget and curl offer -U and -A options for this purpose, respectively.
  Adding such an option costs about 16 Byte in binary size, 4 Byte in
memory and makes the code more complex. Furthermore, we break the ABI in
conf.h. On the plus side, the option is intuitive, wget-compatible and
likely to be used in many -H options anyway.
 [2] Scan the user-defined headers for UA specification and suppress the
UA in this case. Due to the magics of strip, this actually does not
change the binary size. However, it might lead to unexpected behavior:
Adding an option to send additional data could make the request smaller.
Additionally, not everyone knows HTTP in-depth.
 [3] Just don't send the UA. However, that could break lead some
websites to reject requests and hinder the reporting of bugs by server
admins.

 Due to the limited impact of [1], I prefer this one. Giridhar proposed
[2], but agreed that arguments in favor of [1] "sound reasonable". I
guess that means yes  ;)

* Cookies (#310835)
 Can be implemented via -H. However, the arguments of [1] above (in
particular ease of use) still apply. Furthermore, the bug called for
reading the Netscape-style cookies.txt (Wget's --load--cookies option).
Domain-specific cookies could be rather complex to implement. If the
implementation of this feature costs more than 100B, it should be
excludable. Due to the complexity and -H workaround, I'd like to
POSTPONE this feature.

* HTTP authentication (#310785)
 This is actually already implemented and should be documented. Using -H
is possible, too.

* Metalink (#310625)
 Basic Metalink support should not be that difficult. However, it will
only be compiled if METALINK is defined. Metalink support will require
libmetalink (http://code.google.com/p/libmetalink/) which is not yet
packaged for Debian. However, there is hearsay of mortals actually
getting all whitespace and legal issues right to be granted a new Debian
package.

* .netrc (#310635)
 There are lots of GPLed implementations flying around. To minimize code
size, it shouldn't be compiled in by default if the code exceeds a
couple of bytes. Anyway, it's just one call from Axel's point of view.

* Prevent connection to same server (#310638)
 See tcp.c below for the implementation (aside from a flag in the
configuration and a cli flag).

 * Force overriding state file (#311022)
Shouldn't be difficult and take more than a couple of bytes.

Code structure
==============

* conn.c
 needs cleanup, possibly even elimination. Most functions look like
 if (ftp && !proxy) {
 // ... do FTP stuff (15 lines)
 } else {
 // ... do HTTP stuff (20 lines)
 }
 We should at least abstract the switch between HTTP and FTP and look
what can be done about simplifying and documenting the functions here.

 Furthermore, redirecting should be cached somehow/done only once lest
we reach the redirect limit because it's less than -n.
* tcp.c
 should be checked. The functions look a little bit obscure to me. But
maybe, that's just me. Before we implement #310638, we should include
some round-robin trickery in here.
* Removing MAX_STRING(#311085) and MAX_ADD_HEADERS. These are arbitrary
restrictions and may hide a number of strange bugs. Furthermore,
statically sized fields are a waste of memory.
* Add die messages: Axel must not exit with != 0 without telling why.
* Add debugging messages: When compiled with DEBUG, Axel could be more
verbose. This won't harm anything and may serve as documentation.
* Some functions could use a little bit of documentation.

Bugs
====

We're gonna fix them all!
#310979 seems pretty vague.

(User) Documentation
====================

* The man page should include documentation to the -H custom header option.
* The man page should be cleaned up. Especially the copyright part which
actually states Alex wouldn't be Free Software.
* The vertical spacing in the man page  is weird.
* As previously mentioned, authentication should be documented.

As I wrote above, this is an RFC. So please feel free to comment and
discuss. Comments regarding a specific implementation issue rather than
this bird's-eye view should be asked in the relevant bug entries.

Regards,

Philipp Hagemeister

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEAREKAAYFAkjNewkACgkQ9eq1gvr7CFzFqgCfURH/ZWFThE59MZh4hB18QhqZ
kb0Anj/urmvNKOaaXiITirFl3/TE7DUr
=U2rH
-----END PGP SIGNATURE-----