[Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3

Barry Warsaw-103
Public bug reported:

We need a script, documentation, or other procedure to help people
migrate from Mailman 2 to Mailman 3.

** Affects: mailman
     Importance: Undecided
         Status: New


** Tags: mailman3

--
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/965532

Title:
  Need a script to upgrade from MM2 to MM3

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions
_______________________________________________
Mailman-coders mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/mailman-coders
Reply | Threaded
Open this post in threaded view
|

[Bug 965532] Re: Need a script to upgrade from MM2 to MM3

Barry Warsaw-103
** Changed in: mailman
       Status: New => Confirmed

** Changed in: mailman
   Importance: Undecided => High

** Changed in: mailman
   Importance: High => Critical

** Changed in: mailman
    Milestone: None => 3.0.0b2

--
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/965532

Title:
  Need a script to upgrade from MM2 to MM3

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions
_______________________________________________
Mailman-coders mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/mailman-coders
Reply | Threaded
Open this post in threaded view
|

[Bug 965532] Re: Need a script to upgrade from MM2 to MM3

Barry Warsaw-103
In reply to this post by Barry Warsaw-103
** Changed in: mailman
     Assignee: (unassigned) => Richard Wackerbarth (wacky)

--
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/965532

Title:
  Need a script to upgrade from MM2 to MM3

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions
_______________________________________________
Mailman-coders mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/mailman-coders
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3

Richard Wackerbarth
In reply to this post by Barry Warsaw-103
On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:

> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:

> Need a script to upgrade from MM2 to MM3
> https://bugs.launchpad.net/bugs/965532

Here are some thoughts on a possible migration technique.
I would request discussion and suggestions.

In particular, what about the idea of converting the configuration file to HTML as an intermediate file format?
Selectable css could easily render it as a viewable report. It could still be edited by hand without too much difficulty.

Richard "Wacky" Wackerbarth
- - - - - -

Steps to migrate from MM2 to MM3

1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This
should include testing to assure that things are ready to create new
lists.

2) Translate list configurations

  a) Use TOOL1 to extract the set of list configurations from MM2. Pipe
this to TOOL2 which generates a tree of MM2 configurations. That tree
hierarchy would be Root-->World-->Site-->Domain-->List-->Subscriber.
TOOL2 would populate configurations at the List level. It might also
reformat selected parameters. In particular, various <option
type="radio" > entries might be transformed into enumerations such as
"Yes"/"No" or "Hidden"/"Private"/"Public" rather than numerical values.
This would enhance readability.

  b) TOOL3 would populate the World level with the MM2 defaults and
recursively promote common values up the tree, leaving only those
entries which would need to override their parent to derive the current
value. Values which match the parent would be flagged. (The inheritance
flag should be tri-state. "Differs from parent", "Same as parent",
"Inherited from parent")

  c) At this point, the user might edit some of the configurations and
rerun TOOL3 adjusting the inheritance flag as appropriate.

  d) Now, we begin translation to MM3 configuration options. For each MM3 option, TOOL4 computes the equivalent value from the MM2 values. Each computed value also gets the corresponding inheritance flag. Values that cannot be computed from the available information get the "Inherited from parent" flag. MM2 values used in computations are marked as "translated".
  e) After a chance to edit the MM3 configurations, TOOL5 would recompute inheritance flags, report any MM2 values that have not been translated and produce a copy of the configuration file simplified by removing all inherited entries.

  f) After a final inspection TOOL7 would actually import the
configurations, committing entries to the MM3 database.

3) For the migration of rosters, we should be able to do it one
subscription at a time through a pipeline that permits pre- and post-
hooks.  A --dry-run option would be appropriate.

  a) We can assume that each email address is a distinct person.  The
subscribers can utilize the UI to merge email addresses into a common
persona.

  b) We can also assume that each subscription overrides its parent in
the Persona-->EMailAddress-->Subscription hierarchy. The individual
users can use the UI to consolidate their selections.

Some additional thoughts:

        All of the tools should be written in Python, hopefully in a dialect that is common to all of the versions supported by MM3.
        TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes as they come in. Similarly, in step 3, we can use TOOL1 and discard the list configuration information.
        TOOL2 can reformat the XML as HTML, thus making the input data into a viewable report. The inheritance flag would become a class attribute on the option. Would it make sense to go a step further and generate html forms and run a trivial http server on localhost? It might be easier to do this in django, but I think that requiring that level of installation is probably too much for the current situation.

--
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/965532

Title:
  Need a script to upgrade from MM2 to MM3

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions
_______________________________________________
Mailman-coders mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/mailman-coders
Reply | Threaded
Open this post in threaded view
|

Re: [Mailman-Developers] [Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3

Barry Warsaw-103
On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:

>On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
>
>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>
>> Need a script to upgrade from MM2 to MM3
>> https://bugs.launchpad.net/bugs/965532
>
>Here are some thoughts on a possible migration technique.
>I would request discussion and suggestions.
>
>In particular, what about the idea of converting the configuration file to
>HTML as an intermediate file format?  Selectable css could easily render it
>as a viewable report. It could still be edited by hand without too much
>difficulty.

It's an interesting idea.  As you observed, mm2 can export to XML, so it's not
such a big stretch.

w>Steps to migrate from MM2 to MM3
>
>1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>include testing to assure that things are ready to create new lists.

Right, and it should be doable even while mm2 is still functional.

>2) Translate list configurations
>
>  a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
>  to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
>  would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would
>  populate configurations at the List level. It might also reformat selected
>  parameters. In particular, various <option type="radio" > entries might be
>  transformed into enumerations such as "Yes"/"No" or
>  "Hidden"/"Private"/"Public" rather than numerical values. This would
>  enhance readability.
>
>  b) TOOL3 would populate the World level with the MM2 defaults and
>  recursively promote common values up the tree, leaving only those entries
>  which would need to override their parent to derive the current
>  value. Values which match the parent would be flagged. (The inheritance
>  flag should be tri-state. "Differs from parent", "Same as parent",
>  "Inherited from parent")
>
>  c) At this point, the user might edit some of the configurations and rerun
>  TOOL3 adjusting the inheritance flag as appropriate.

I think the trickiest part will be what to do about subscriber information.
In mm2, this is always list-centric, but in mm3, you need to collate and
globalize all the membership information into the user database.  You can
probably do the same kind of up-promotion there, but it would be from
member->address->user.  IOW, if you see an address subscribed to a mailing
list with the same values across all those lists, put the preferences in the
user.  What happens if you see [hidden email] subscribed to three different
lists with three different passwords?  That's a tough one because there's no
way to express that in mm3 (nor probably should there be).

So I think you will occasionally have to just resolve some conflicts by
flipping a coin.  In the case of passwords, perhaps you'd always make the user
do a password reset.

>  d) Now, we begin translation to MM3 configuration options. For each MM3
>  option, TOOL4 computes the equivalent value from the MM2 values. Each
>  computed value also gets the corresponding inheritance flag. Values that
>  cannot be computed from the available information get the "Inherited from
>  parent" flag. MM2 values used in computations are marked as "translated".

The user herself could probably write a script for this pipeline you're
proposing, that would allow her to do bulk transformations of configuration
variable.

>  e) After a chance to edit the MM3 configurations, TOOL5 would recompute
>  inheritance flags, report any MM2 values that have not been translated and
>  produce a copy of the configuration file simplified by removing all
>  inherited entries.
>
>  f) After a final inspection TOOL7 would actually import the configurations,
>  committing entries to the MM3 database.
>
>3) For the migration of rosters, we should be able to do it one subscription
>at a time through a pipeline that permits pre- and post- hooks.  A --dry-run
>option would be appropriate.

Almost definitely.  The --dry-run step which would produce an output of those
conflicts, and impossible situations.  The user would then have a chance to
re-edit the intermediate file so that the values can be better mapped to mm3.
It's probably worth doing for both the rosters and list configurations.

>  a) We can assume that each email address is a distinct person.  The
>  subscribers can utilize the UI to merge email addresses into a common
>  persona.

We'll probably need a "claim and merge" operation in the system.  And a way to
purge unclaimed addresses after a while.

>  b) We can also assume that each subscription overrides its parent in the
>  Persona-->EMailAddress-->Subscription hierarchy. The individual users can
>  use the UI to consolidate their selections.

A good challenge for the Postoriusians :)

> All of the tools should be written in Python, hopefully in a dialect that is
> common to all of the versions supported by MM3.

+1.  Today that would be 2.6 and 2.7.

> TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes
> as they come in. Similarly, in step 3, we can use TOOL1 and discard the list
> configuration information.

> TOOL2 can reformat the XML as HTML, thus making the input data into a
> viewable report. The inheritance flag would become a class attribute on the
> option. Would it make sense to go a step further and generate html forms and
> run a trivial http server on localhost? It might be easier to do this in
> django, but I think that requiring that level of installation is probably
> too much for the current situation.

It wouldn't be hard to do in standard Python.  OTOH, I'm not sure we want to
maintain a stack of html templates, forms, and form processing in the
conversion tool.

It's sounding like this suite of conversion tools may need to be a separate
sub-project.

-Barry

--
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/965532

Title:
  Need a script to upgrade from MM2 to MM3

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions
_______________________________________________
Mailman-coders mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/mailman-coders
Reply | Threaded
Open this post in threaded view
|

Re: [Mailman-Developers] [Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3

Richard Wackerbarth
Many MM admins are NOT programmers. I could do the transforms in either
Python or XSL.

Which would be more acceptable to our users who might want to customize
the transform? In either case, the transforms would be kept in a
separate file from the rest of the engine and pipeline so that "where to
customize" is clear.

Richard

On Apr 10, 2012, at 3:46 PM, Barry Warsaw <[hidden email]>
wrote:

> On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:
>
>> On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
>>
>>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>>
>>> Need a script to upgrade from MM2 to MM3
>>> https://bugs.launchpad.net/bugs/965532
>>
>> Here are some thoughts on a possible migration technique.
>> I would request discussion and suggestions.
>>
>> In particular, what about the idea of converting the configuration file to
>> HTML as an intermediate file format?  Selectable css could easily render it
>> as a viewable report. It could still be edited by hand without too much
>> difficulty.
>
> It's an interesting idea.  As you observed, mm2 can export to XML, so it's not
> such a big stretch.
>
> w>Steps to migrate from MM2 to MM3
>>
>> 1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>> include testing to assure that things are ready to create new lists.
>
> Right, and it should be doable even while mm2 is still functional.
>
>> 2) Translate list configurations
>>
>> a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
>> to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
>> would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would
>> populate configurations at the List level. It might also reformat selected
>> parameters. In particular, various <option type="radio" > entries might be
>> transformed into enumerations such as "Yes"/"No" or
>> "Hidden"/"Private"/"Public" rather than numerical values. This would
>> enhance readability.
>>
>> b) TOOL3 would populate the World level with the MM2 defaults and
>> recursively promote common values up the tree, leaving only those entries
>> which would need to override their parent to derive the current
>> value. Values which match the parent would be flagged. (The inheritance
>> flag should be tri-state. "Differs from parent", "Same as parent",
>> "Inherited from parent")
>>
>> c) At this point, the user might edit some of the configurations and rerun
>> TOOL3 adjusting the inheritance flag as appropriate.
>
> I think the trickiest part will be what to do about subscriber information.
> In mm2, this is always list-centric, but in mm3, you need to collate and
> globalize all the membership information into the user database.  You can
> probably do the same kind of up-promotion there, but it would be from
> member->address->user.  IOW, if you see an address subscribed to a mailing
> list with the same values across all those lists, put the preferences in the
> user.  What happens if you see [hidden email] subscribed to three different
> lists with three different passwords?  That's a tough one because there's no
> way to express that in mm3 (nor probably should there be).
>
> So I think you will occasionally have to just resolve some conflicts by
> flipping a coin.  In the case of passwords, perhaps you'd always make the user
> do a password reset.
>
>> d) Now, we begin translation to MM3 configuration options. For each MM3
>> option, TOOL4 computes the equivalent value from the MM2 values. Each
>> computed value also gets the corresponding inheritance flag. Values that
>> cannot be computed from the available information get the "Inherited from
>> parent" flag. MM2 values used in computations are marked as "translated".
>
> The user herself could probably write a script for this pipeline you're
> proposing, that would allow her to do bulk transformations of configuration
> variable.
>
>> e) After a chance to edit the MM3 configurations, TOOL5 would recompute
>> inheritance flags, report any MM2 values that have not been translated and
>> produce a copy of the configuration file simplified by removing all
>> inherited entries.
>>
>> f) After a final inspection TOOL7 would actually import the configurations,
>> committing entries to the MM3 database.
>>
>> 3) For the migration of rosters, we should be able to do it one subscription
>> at a time through a pipeline that permits pre- and post- hooks.  A --dry-run
>> option would be appropriate.
>
> Almost definitely.  The --dry-run step which would produce an output of those
> conflicts, and impossible situations.  The user would then have a chance to
> re-edit the intermediate file so that the values can be better mapped to mm3.
> It's probably worth doing for both the rosters and list configurations.
>
>> a) We can assume that each email address is a distinct person.  The
>> subscribers can utilize the UI to merge email addresses into a common
>> persona.
>
> We'll probably need a "claim and merge" operation in the system.  And a way to
> purge unclaimed addresses after a while.
>
>> b) We can also assume that each subscription overrides its parent in the
>> Persona-->EMailAddress-->Subscription hierarchy. The individual users can
>> use the UI to consolidate their selections.
>
> A good challenge for the Postoriusians :)
>
>> All of the tools should be written in Python, hopefully in a dialect that is
>> common to all of the versions supported by MM3.
>
> +1.  Today that would be 2.6 and 2.7.
>
>> TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes
>> as they come in. Similarly, in step 3, we can use TOOL1 and discard the list
>> configuration information.
>
>> TOOL2 can reformat the XML as HTML, thus making the input data into a
>> viewable report. The inheritance flag would become a class attribute on the
>> option. Would it make sense to go a step further and generate html forms and
>> run a trivial http server on localhost? It might be easier to do this in
>> django, but I think that requiring that level of installation is probably
>> too much for the current situation.
>
> It wouldn't be hard to do in standard Python.  OTOH, I'm not sure we want to
> maintain a stack of html templates, forms, and form processing in the
> conversion tool.
>
> It's sounding like this suite of conversion tools may need to be a separate
> sub-project.
>
> -Barry
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/965532
>
> Title:
>  Need a script to upgrade from MM2 to MM3
>
> Status in GNU Mailman:
>  Confirmed
>
> Bug description:
>  We need a script, documentation, or other procedure to help people
>  migrate from Mailman 2 to Mailman 3.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions

--
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/965532

Title:
  Need a script to upgrade from MM2 to MM3

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions
_______________________________________________
Mailman-coders mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/mailman-coders