Switching servers, migration plan

I am starting to slowly switch services over from charon to neko-base (two servers here at home). The first step has been started, which is to duplicate all DNS information to both and start switching over domains to point to the new servers. If you’re a charon user, you may have already received mail from me about that.

The idea behind a slow migration rather than a quick one is so that each service will be impacted for a minimal amount of time — hopefully no more than an hour of inconvenience to the owners, and no Web/Mail outage observed by their clients. For Mush clients there will be an interruption, either the users will not be able to play during the switchover, or if the owner prefers, the users can stay logged in to the old server while the new one is being set up (and they should not do any building or send/change their +mail). The slower migration of services to the new host also allows me to double-check the services we are providing, to drop any services that are no longer required, and change the setup on the new machine if needed.

The rest of the message describes how I am going to move users over from one to the other. If this affects you, I will be emailing you anyway, but comments are welcome from others as well.

Preparation phase
Not time-critical, check off all before starting migration

  1. [All] Create new account and make sure user is able to log in. (Verify ssh client, since we are no longer supporting telnet logins)
  2. [All] Have user do the first pass of copying files: rsync –rsh=ssh -a ~/ neko-base-old:~
  3. [Web] Set up an alternate hostname to point to neko-base, such as test.mydomain.com (or test.mydomain.nekodojo.org)
  4. [Web] Add the alternate hostname to the VirtualHost section (/usr/local/adm/apache/virtual.conf)
  5. [Web] View the alternate URL and make sure that the web content and behavior is correct.
  6. [All] Verify that the user is able to get in and modify files (ftp, scp — adjust /usr/local/adm/firewall/firewall)
  7. [Mush] Test run the game on neko-base to make sure it works. Shut down the test game after verifying.
  8. [Mush] Also make a backup copy of the last known-good save in a separate dir.

First phase, mail and mush
At this point we are ready to begin the migration.

  1. [Mush] Shut down the game on charon. Alternately, save the game and announce to users that they should make no more changes to buildings or +mail or the changes will be lost.
  2. [All] Ask user to not update any files (home directory or web) until everything is finished.
  3. [All] Rsync again to copy any last-minute changes.
  4. [Mush] Start the game on neko-base after data files are syncd over.
  5. [Mush] For mush accounts, we may want to go ahead and update the A record in DNS immediately and not wait for mail.
  6. [Mail] If the mail is being forwarded, OK to skip the following [Mail] steps.
  7. [Mail] Rsync the mail spool also (need instructions here)
  8. [Mail] Switch virtusertable on both charon and neko-base to point to neko-base. (/usr/local/adm/mail/virtusertable — Update neko-base first :)
  9. [Mail] After change is made, send test message and verify it is received at neko-base (cat /var/spool/mail/yourname)
  10. [Mail] Check old account on charon for any new messages, if there are any, forward/bounce them to the new account
  11. [Mail] Update client-side mail settings (if needed)

Second phase, DNS and Web
At this point Mail is set up on the new host, but Web is still being served from the old host.

  1. [Web] Change the DNS for mydomain.com so that Web requests now go to neko-base
  2. [Mail] While updating DNS, also change the MX to add neko-base and remove charon
  3. [Mail,Web] Force the DNS changes to all three servers and verify with nslookup.
  4. [Mush] If the game was still running on charon, go announce to users that the new site is ready and they should drop and reconnect, then shut down the game.
  5. [Web] Verify that the web page displays correctly.
  6. [Web] If the user has any files in ~/www, adjust apache on charon to provide a redirect to neko-base (this is only needed for nekodojo.org/~user requests since nekodojo.org www will be moved last — anything using their personal domain name will already be moved by DNS change — /usr/local/adm/apache/virtual.conf)

Done.
Watch the logs for anything suspicious.

Comments welcome.

0 thoughts on “Switching servers, migration plan

Leave a Reply