[debian-mysql] Discussion on handling 5.6 related matters

[Akhil Mohan]

Hi Robie,

On Tuesday 08 July 2014 10:34 PM, Robie Basak wrote:
> Hi Akhil,
>
> On Mon, Jul 07, 2014 at 07:30:59PM +0530, Akhil Mohan wrote:
>> Thanks for your response which makes it easier to sort out the priority
>> list at this moment. As I get it, I am placing the tasks in order of
>> priority below:
>>
>>  1. Upgrade Paths:
>>      1. Separation of data directory.
>>      2. mysql-defaults package should contain common functionality and
>>         own data directory as well.
>>      3. Handling upgrade scenarios from old format to new format that
>>         depends on mysql-defaults.
>>  2. Get remaining delta from 5.5 into 5.6 packaging source.
>>  3. Import latest released version of 5.6 into experimental and trusty
>>     universe.
>>  4. Get 5.6 as default in Debian testing and Ubuntu 14.10.
> Agreed, and thank you for the summary.
>
>> For now, as you said already, lets discuss on upgrade paths first.
>>
>> You proposal looks promising but I suspect that users might find it
>> difficult to handle change of data directory location who wish continue
>> with MySQL. Moreover, MySQL 5.5 data dir can be directly taken over by
>> MySQL 5.6 using simple mysql_upgrade utility as it is being done at this
>> moment. There will no need for having separate /var/lib/mysql-5.6 data
>> dir for reason of binary incompatibility.
> Does this include all variants which uptil now have written to
> /var/lib/mysql/ as they wish?
MySQL 5.x should not be a problem and for Mariadb 5.5 is said to be a
drop-in replacement for MySQL 5.5. If you ask about Mariadb 5.5 taking
over a MySQL 5.5 data dir then it looks doable. Otto might have better
idea here.

I would also like to add that I did not consider making all variants
co-installable. I was under impression that we want to keep only one
variant running. Still, some of what I said earlier remains valid and I
would still stress on keeping /var/lib/mysql as a fork directory for
MySQL because keeping 5.5 and 5.6 co-installable does not be give
additional benefit.
>
> I still don't like the idea of not having a clean break from the former
> confusion of ownership of /var/lib/mysql. You might say that MySQL 5.6
> owns the directory now, but what stops older versions of (for example)
> MariaDB from writing to it? Or are we going to have some kind of
> Breaks/Replaces the world and update all variants at once?
I agree that each fork should have its own data dir but separate data
dir for each version should be created only if there is a requirement. I
see following possibilities here:

 1. User already has MySQL 5.5.37/ 5.6.17 or lesser and upgrades to:
     1. Older/ latest version of MySQL 5.5/ 5.6 and this will keep him
        on /var/lib/mysql.
     2. Older version of fork that is present as of today, and continues
        to use /var/lib/mysql.
     3. Latest version of fork that detects use of /var/lib/mysql, moves
        data dir to /var/lib/fork and continues.
 2. User has older version of Fork installed that is currently writing
    to /var/lib/mysql and upgrades to:
     1. Latest version of fork that detects use of /var/lib/mysql, moves
        data dir to /var/lib/fork.
     2. Fork2 or MySQL that "can" carry the current state of data dir
        forward. It moves data dir to /var/lib/fork2 or /var/lib/mysql
        as needed.
     3. Fork2 or MySQL that "cannot" carry the current state of data dir
        forward. It creates fresh data dir /var/lib/fork2 or
        /var/lib/mysql as needed.

This invalidates the case of creating controlled links and admin scripts
will need to be modified as needed. Apart from this, we should quiz the
users if they want to move existing data dir or create fresh blank setup
ready for imports to be done later. In case, user chooses to import then
he will need to take backup/ dump of current data dir on separate
terminal prompt before he continues further. We may show a debconf
notice to take backup and only then continue.

>
>> Another trouble that I see with forking the data dir is related to the
>> size of data dir. In case user has large data dir that goes into GBs or
>> even TBs then the copy would take noticeable amount of time. We might
>> need to show them a progress status/ bar and also handle situations like
>> copy of data dir ending midway with an error.
> How about a one-off rename instead? That should be quick.
Yes, that should do.
>
>> I would like to present a slight modification to the proposal with help
>> of controlled links. This would keep /var/lib/mysql intact owned by the
>> MySQL 5.x series keeping things simpler for all existing users who wish
>> to continue with MySQL.
> I don't think it's simpler, because variants already assume ownership of
> this directory, and have potentially modified it. What if we instead
> renamed /var/lib/mysql to a new name, so that the directory has clear
> ownership again?
I think newer versions of variants can move the data dir to new fork
specific location making /var/lib/mysql free for MySQL 5.x. For the
variants already installed and writing to /var/lib/mysql, installation
of MySQL can create new data dir at /var/lib/mysql after moving existing
data dir to some backup location like /var/lib/mysql.backup. This would
need manual import or movement of individual databases as needed by
user. This is similar to upgrade options I listed above.
>
>>                         In case a user wants to install another fork of
>> MySQL then during installation fork.{pre,post}inst would detect existing
>> MySQL 5.6 and would ask if the user wishes to fork the data dir or start
>> fresh.
> I'm with you so far.
>
>>  1. Fresh start with blank data dir.
>>      1. Uninstall MySQL 5.x as regular removal.
> We want co-installability though, right? So removal would no longer be
> required?
As mentioned by Otto in other mail, this would need a secondary config
file my.cnf for each variant to use non-standard port, socket file etc.
If a variant is to be installed second then it should consider the
secondary config file. This can be achieved but we may want to re-check
if we should allow it immediately or keep it for future implementation.
>
>>      2. Create fork specific data dir /var/lib/fork or
>>         /var/lib/fork-version in case there can be compatibility issues.
> Agreed.
>
>>      3. Move /var/lib/mysql to /var/lib/mysql.backup (snapshot)
>>      4. Create link /var/lib/mysql pointing to /var/lib/fork
>>      5. Install fork as regular installation.
>>
>> The user can decide after installation to remove, backup or keep
>> /var/lib/mysql.backup.
> This is where I think the confusion starts, but I don't think it makes
> sense to go into it without deciding on the co-installation question
> first.
I see co-installable setup as an interesting option but I am not sure
about amount of time it would need and how much time we have before we
freeze 14.10 code base. Testing effort should be considered here.
>
>>                        Fork would be able to access data dir directly at
>> /var/lib/fork. Same will remain accessible via /var/lib/mysql if needed
>> for administration purpose by user scripts.
> I think this could lead to trouble. Scripts should know what they're
> accessing, in terms of their compatibility. It'd be better for scripts
> to break, and be fixed to use some helper provided by mysql-defaults to
> figure out which directory to use.
Agreed, even if we do not get votes for co-installable setup. Admin
scripts should break and be modified.
>
>>  2. Fork the data dir.
>>      1. Uninstall MySQL 5.x as regular removal.
>>      2. Move /var/lib/mysql to /var/lib/fork and optionally copy
>>         /var/lib/fork to /var/lib/mysql.backup (snapshot)
>>      3. Create link /var/lib/mysql pointing to /var/lib/fork
>>      4. Install fork as regular installation.
> So what happens now if the user wants to go back to the MySQL variant?
> With co-installability, the way I see it is that there's nothing that
> needs to be done. The variant forks the data, and the original MySQL
> carries on with the original side of the fork.
Note: Point 3 is now invalid with co-installable setup under consideration.

I agree with you here but copying the whole db cleanly looks less than
possible to me. We may want to ask an additional question to user if he
wants a MySQL snapshot at /var/lib/mysql and has enough disk space. This
way if users says yes then we copy data dir to /var/lib/fork else we
move. Still the copy may fail for reasons other than disk space and we
will need to know what should be done in that situation.
>
>> Please let me know your views on the modifications suggested above to
>> the initial proposal.
> It feels more complicated to me, because to me it seems that it
> necessiates consideration of more edge cases. IMHO, deprecating the
> confused-ownership /var/lib/mysql is cleaner, since it massively reduces
> the problem space where edge cases may lie.
>
> What exactly is the problem with my initial proposal that you're trying
> to solve here?
I see it this way:

 1. User using MySQL and decide to continue with the same variant should
    have no impact.
 2. User migrating to different variants should be quick by avoiding copy.
 3. User already on different variant should be in control on how he
    wishes to proceed with changes on data dir.

>
> Would a rename solve your concern about time for the 5.5 to 5.6 (or
> updated 5.6 with this proposal implemented) upgrade?
Yes, rename does the job. Copy looks somewhat difficult to me.
>
> Giving the user the option of starting with a fresh empty database for
> all variants (over automatically migrating data from a different
> variant) makes sense to me - that could be a debconf question for which
> the default could be decided later. But I think that's orthogonal to my
> proposal, which only really provides the mechanism for migrations,
> leaving it for variants to implement (with common code in mysql-defaults
> as makes sense).
Yes, it will give some control to user over the situation and will make
us look more user-friendly.
>
> Robie

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-mysql-maint/attachments/20140709/52a48f1e/attachment-0001.html>