<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix"><br>
Hi Robie,<br>
<br>
On Tuesday 08 July 2014 10:34 PM, Robie Basak wrote:<br>
</div>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">Hi Akhil,
On Mon, Jul 07, 2014 at 07:30:59PM +0530, Akhil Mohan wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Thanks for your response which makes it easier to sort out the priority
list at this moment. As I get it, I am placing the tasks in order of
priority below:
1. Upgrade Paths:
1. Separation of data directory.
2. mysql-defaults package should contain common functionality and
own data directory as well.
3. Handling upgrade scenarios from old format to new format that
depends on mysql-defaults.
2. Get remaining delta from 5.5 into 5.6 packaging source.
3. Import latest released version of 5.6 into experimental and trusty
universe.
4. Get 5.6 as default in Debian testing and Ubuntu 14.10.
</pre>
</blockquote>
<pre wrap="">
Agreed, and thank you for the summary.
</pre>
<blockquote type="cite">
<pre wrap="">For now, as you said already, lets discuss on upgrade paths first.
You proposal looks promising but I suspect that users might find it
difficult to handle change of data directory location who wish continue
with MySQL. Moreover, MySQL 5.5 data dir can be directly taken over by
MySQL 5.6 using simple mysql_upgrade utility as it is being done at this
moment. There will no need for having separate /var/lib/mysql-5.6 data
dir for reason of binary incompatibility.
</pre>
</blockquote>
<pre wrap="">
Does this include all variants which uptil now have written to
/var/lib/mysql/ as they wish?</pre>
</blockquote>
MySQL 5.x should not be a problem and for Mariadb 5.5 is said to be
a drop-in replacement for MySQL 5.5. If you ask about Mariadb 5.5
taking over a MySQL 5.5 data dir then it looks doable. Otto might
have better idea here.<br>
<br>
I would also like to add that I did not consider making all variants
co-installable. I was under impression that we want to keep only one
variant running. Still, some of what I said earlier remains valid
and I would still stress on keeping /var/lib/mysql as a fork
directory for MySQL because keeping 5.5 and 5.6 co-installable does
not be give additional benefit.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
I still don't like the idea of not having a clean break from the former
confusion of ownership of /var/lib/mysql. You might say that MySQL 5.6
owns the directory now, but what stops older versions of (for example)
MariaDB from writing to it? Or are we going to have some kind of
Breaks/Replaces the world and update all variants at once?</pre>
</blockquote>
I agree that each fork should have its own data dir but separate
data dir for each version should be created only if there is a
requirement. I see following possibilities here:<br>
<ol>
<li>User already has MySQL 5.5.37/ 5.6.17 or lesser and upgrades
to:</li>
<ol>
<li>Older/ latest version of MySQL 5.5/ 5.6 and this will keep
him on /var/lib/mysql.</li>
<li>Older version of fork that is present as of today, and
continues to use /var/lib/mysql.</li>
<li>Latest version of fork that detects use of /var/lib/mysql,
moves data dir to /var/lib/fork and continues.<br>
</li>
</ol>
<li>User has older version of Fork installed that is currently
writing to /var/lib/mysql and upgrades to:</li>
<ol>
<li>Latest version of fork that detects use of /var/lib/mysql,
moves data dir to /var/lib/fork.</li>
<li>Fork2 or MySQL that "can" carry the current state of data
dir forward. It moves data dir to /var/lib/fork2 or
/var/lib/mysql as needed.</li>
<li>Fork2 or MySQL that "cannot" carry the current state of data
dir forward. It creates fresh data dir /var/lib/fork2 or
/var/lib/mysql as needed.</li>
</ol>
</ol>
<p>This invalidates the case of creating controlled links and admin
scripts will need to be modified as needed. Apart from this, we
should quiz the users if they want to move existing data dir or
create fresh blank setup ready for imports to be done later. In
case, user chooses to import then he will need to take backup/
dump of current data dir on separate terminal prompt before he
continues further. We may show a debconf notice to take backup and
only then continue.<br>
</p>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">Another trouble that I see with forking the data dir is related to the
size of data dir. In case user has large data dir that goes into GBs or
even TBs then the copy would take noticeable amount of time. We might
need to show them a progress status/ bar and also handle situations like
copy of data dir ending midway with an error.
</pre>
</blockquote>
<pre wrap="">
How about a one-off rename instead? That should be quick.</pre>
</blockquote>
Yes, that should do.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">I would like to present a slight modification to the proposal with help
of controlled links. This would keep /var/lib/mysql intact owned by the
MySQL 5.x series keeping things simpler for all existing users who wish
to continue with MySQL.
</pre>
</blockquote>
<pre wrap="">
I don't think it's simpler, because variants already assume ownership of
this directory, and have potentially modified it. What if we instead
renamed /var/lib/mysql to a new name, so that the directory has clear
ownership again?</pre>
</blockquote>
I think newer versions of variants can move the data dir to new fork
specific location making /var/lib/mysql free for MySQL 5.x. For the
variants already installed and writing to /var/lib/mysql,
installation of MySQL can create new data dir at /var/lib/mysql
after moving existing data dir to some backup location like
/var/lib/mysql.backup. This would need manual import or movement of
individual databases as needed by user. This is similar to upgrade
options I listed above.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap=""> In case a user wants to install another fork of
MySQL then during installation fork.{pre,post}inst would detect existing
MySQL 5.6 and would ask if the user wishes to fork the data dir or start
fresh.
</pre>
</blockquote>
<pre wrap="">
I'm with you so far.
</pre>
<blockquote type="cite">
<pre wrap=""> 1. Fresh start with blank data dir.
1. Uninstall MySQL 5.x as regular removal.
</pre>
</blockquote>
<pre wrap="">
We want co-installability though, right? So removal would no longer be
required?</pre>
</blockquote>
As mentioned by Otto in other mail, this would need a secondary
config file my.cnf for each variant to use non-standard port, socket
file etc. If a variant is to be installed second then it should
consider the secondary config file. This can be achieved but we may
want to re-check if we should allow it immediately or keep it for
future implementation.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap=""> 2. Create fork specific data dir /var/lib/fork or
/var/lib/fork-version in case there can be compatibility issues.
</pre>
</blockquote>
<pre wrap="">
Agreed.
</pre>
<blockquote type="cite">
<pre wrap=""> 3. Move /var/lib/mysql to /var/lib/mysql.backup (snapshot)
4. Create link /var/lib/mysql pointing to /var/lib/fork
5. Install fork as regular installation.
The user can decide after installation to remove, backup or keep
/var/lib/mysql.backup.
</pre>
</blockquote>
<pre wrap="">
This is where I think the confusion starts, but I don't think it makes
sense to go into it without deciding on the co-installation question
first.</pre>
</blockquote>
I see co-installable setup as an interesting option but I am not
sure about amount of time it would need and how much time we have
before we freeze 14.10 code base. Testing effort should be
considered here.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap=""> Fork would be able to access data dir directly at
/var/lib/fork. Same will remain accessible via /var/lib/mysql if needed
for administration purpose by user scripts.
</pre>
</blockquote>
<pre wrap="">
I think this could lead to trouble. Scripts should know what they're
accessing, in terms of their compatibility. It'd be better for scripts
to break, and be fixed to use some helper provided by mysql-defaults to
figure out which directory to use.</pre>
</blockquote>
Agreed, even if we do not get votes for co-installable setup. Admin
scripts should break and be modified.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap=""> 2. Fork the data dir.
1. Uninstall MySQL 5.x as regular removal.
2. Move /var/lib/mysql to /var/lib/fork and optionally copy
/var/lib/fork to /var/lib/mysql.backup (snapshot)
3. Create link /var/lib/mysql pointing to /var/lib/fork
4. Install fork as regular installation.
</pre>
</blockquote>
<pre wrap="">
So what happens now if the user wants to go back to the MySQL variant?
With co-installability, the way I see it is that there's nothing that
needs to be done. The variant forks the data, and the original MySQL
carries on with the original side of the fork.</pre>
</blockquote>
Note: Point 3 is now invalid with co-installable setup under
consideration.<br>
<br>
I agree with you here but copying the whole db cleanly looks less
than possible to me. We may want to ask an additional question to
user if he wants a MySQL snapshot at /var/lib/mysql and has enough
disk space. This way if users says yes then we copy data dir to
/var/lib/fork else we move. Still the copy may fail for reasons
other than disk space and we will need to know what should be done
in that situation.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">Please let me know your views on the modifications suggested above to
the initial proposal.
</pre>
</blockquote>
<pre wrap="">
It feels more complicated to me, because to me it seems that it
necessiates consideration of more edge cases. IMHO, deprecating the
confused-ownership /var/lib/mysql is cleaner, since it massively reduces
the problem space where edge cases may lie.
What exactly is the problem with my initial proposal that you're trying
to solve here?</pre>
</blockquote>
I see it this way:<br>
<ol>
<li>User using MySQL and decide to continue with the same variant
should have no impact.</li>
<li>User migrating to different variants should be quick by
avoiding copy.</li>
<li>User already on different variant should be in control on how
he wishes to proceed with changes on data dir.<br>
</li>
</ol>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
Would a rename solve your concern about time for the 5.5 to 5.6 (or
updated 5.6 with this proposal implemented) upgrade?</pre>
</blockquote>
Yes, rename does the job. Copy looks somewhat difficult to me.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
Giving the user the option of starting with a fresh empty database for
all variants (over automatically migrating data from a different
variant) makes sense to me - that could be a debconf question for which
the default could be decided later. But I think that's orthogonal to my
proposal, which only really provides the mechanism for migrations,
leaving it for variants to implement (with common code in mysql-defaults
as makes sense).</pre>
</blockquote>
Yes, it will give some control to user over the situation and will
make us look more user-friendly.<br>
<blockquote cite="mid:20140708170416.GA17765@mal.justgohome.co.uk"
type="cite">
<pre wrap="">
Robie
</pre>
</blockquote>
<br>
</body>
</html>