Creating a manual Moxie failover environment

Summary

When working on a production network, a Data Server failover is a best-practice. This minimizes the possibility of data loss in the event of a catastrophic hardware failure. Automated failover support is available in Moxie 7.05.

Scope

This document is intended as a reference for creating and configuring a Moxie failover environment. It covers new Moxie installations as well as Moxie upgrades.

Notes:

  • For the purposes of this document, the active Data Server has been named "DS1" and the passive or failover Data Server has been named "DS2". This naming convention is used for illustration and clarity purposes only and does not have to be used for implementation of the failover environment.
  • This document shows an environment with a single, active Data Server and a single, passive Data Server. Although there can be only one active Data Server at a time, it is possible to configure multiple passive Data Servers, using these techniques.

Architecture

Prerequisites

  1. A fully qualified domain name for Data Server, set in DNS server. The fully qualified domain name for the Data Server should be created with a CNAME record, for example "ds.domain.com". In the CNAME record properties, set the Time to Live (TTL), which tells the client when to refresh the value from the DNS Server, to 1 minute.



    Based on this value, clients typically reconnect within 2-3 minutes (dependant on various factors such as when the DNS Time to Live expires and the reconnect timer within the client).
  2. Use the fully qualified domain name in the Data Server Connection Wizard for Moxie components, instead of the named Data Server’s hostname.

    Set this up is using an A record for each Data Server (with their IP address) and a CNAME record for the fully qualified domain name:
    • DS1.domain.com A 192.168.200.101
    • DS2.domain.com A 192.168.200.102
    • DS.domain.com CNAME DS1.domain.com (on failover this would be set to DS2.domain.com)

    Note: DNS entries’ TTL can be verified at the client using ipconfig /displaydns

  3. Create a file share to be used for both Moxie Data Servers’ caches. In this example, a network share is used, but this could also be done with another technology, such as replicated folders using DFS, etc.

    Optionally, a folder for both Log Server logs can be created, if using Log Server. This isn’t required, but can help in reducing duplicates.
  4. Map the Network share using a symbolic link (mklink /D [link] [target]) at the command prompt on both Data Server computers. Ensure the symbolic link is the same for each Server (ie. “C:\MoxieShare” is symbolically linked to \\FileServer\MoxieShare). Services using this symbolic link must run as a named user account.

    Note: This step is only applicable if using a file share, as tested by Omnivex, and may not be applicable if using other file-replication technologies.

Installing the Data Servers

  1. Install the active Moxie Data Server (DS1) first. Follow the typical installation instructions, with the following exceptions:
    • in the MDSC, create the database and point the cache location to the symbolic link folder. Take note of each setting, as these need to be used when installing the passive Data Server (DS2).

      Note: All MDSC settings for subsequent, passive Data Server installations must match the active Data Server’s configured settings, including: database setup, SQL server account, Data Server setup (port, dedicated DataPipe port, cache location), Client identification (player ID, passphrase, default player content), encryption and permission settings.
    • in Data Server Connection Wizard, in Address or host name, use localhost.

    • if installing Log Server on same computer, then use the fully qualified domain name (ie. DS.domain.com) in the Log Server Config Wizard, so that CLAs get the fully qualified domain name to connect to.
  2. Set all Moxie services to Manual so that in the case of a server failure, the services do not reconnect to the database on reboot.

    This means that on a Server reboot, the services must be started manually if not doing a failover at that time.

    There are other ways to accomplish this, such as the SQL cluster having a firewall scope to only accept the active Server’s address for connection (this firewall scope will alternate to the other server during failover).
  3. Prior to installation on DS2, stop all Moxie services on DS1.
  4. Install Moxie Data Server on DS2:
    • in the MDSC, target the same database, symbolic link folder and settings as for DS1
    • in the Data Server Connection Wizard, use localhost
    • if installing Log Server on same computer, then use the fully qualified domain name
  5. Set all Moxie services to Manual, as this is the passive server.
  6. Both Data Servers must be registered separately with their own license. The licenses should match in terms of user and player counts.

Installing Studios and Players

  1. Install Studios and Players, following the standard installation instructions.
  2. In the Data Server Connection wizard, in Address or host name, use the domain name specified in the CNAME record.

Failover steps

  1. Ensure all Moxie services on the active Data Server (DS1) are stopped before starting the services on the passive Data Server (DS2).
  2. Change the CNAME DNS entry for DS.domain.com to the passive server (DS2.domain.com).

Notes:

  • In 7.01, Studio and Player detect a Data Server name change and restart after reconnection (if it was running prior to failover). This purges all caches. The Studio downloads modules again and the player downloads its assigned content again. This can be problematic depending on how much content the players have. If a Studio or Player is started from a stopped state, this check is for the network folder’s GUID instead, and does not prompt for cache purge or restarts.
  • In 7.02 and higher, the reconnection check has been changed to the initial connection check (via network folder GUID) and Studios/Players reconnect to the failover server without a purge and restart as expected (going from DS1 to DS2).

Upgrade path

  1. Upgrade active Data Server (DS1) as per usual, making sure services are stopped on the passive Data Server (DS2). Complete the MDSC, Data Server Connection Wizard and Log Server Connection Wizard (if applicable).
  2. Before upgrading DS2, stop all services on DS1.
  3. Upgrade DS2 and complete the MDSC, Data Server Connection Wizard and Log Server Connection Wizard (if applicable).
  4. Stop the Moxie services on DS2, and restart the services on DS1. Clients reconnect to DS1.

After each Data Server installation/upgrade, you must:

  • set service accounts back to using named accounts for services configured with symbolic links
  • set Moxie services back to Manual startup
  • create on install (or restore on upgrade) any custom metadata and Data Server configuration settings

FAQ

  1. How do I know when to start the backup Moxie Data Server service?
    1. Manually determine there is a problem.
    2. Stop the services on the active Data Server, if possible.
    3. Start the services on the passive Data Server. This becomes the new "active" Data Server, once the DNS is changed.
  2. How do I fail over the Moxie Players to the new active server?

    Change global DNS manually
  3. Do I need a failover environment for the DataPipe Server and the Data Suite products?
    • Secondary DataPipe Server is already configurable
    • Other Data Suite products can reside on the same server as the DataPipe Server
  4. What about my SQL Server Cluster?

    SQL Server Cluster is redundant, by definition
  5. Do I need a redundant Moxie cache location?

    Yes, the Moxie share should be made redundant by whatever technology you use
  6. Do I still need to plan for a disaster recovery situation?

    Yes. Even though, by definition, all components are made redundant by having multiple Moxie services, SQL cluster and a redundant file share for the cache, this does not alleviate the need for a disaster recovery scenario. Plan backup of SQL databases and the Moxie file share accordingly.

Known issues 

  1. When changing Data Servers, both Data Servers will be listed in System Manager and their status will always show as currently connected, as denoted by a green checkmark, even though neither may be running at the time.
  2. When changing Data Servers via DNS entry (as described in this document) or by changing the HOSTS file, the Data Server name that appears:
    • on status bar of Moxie Studio
    • in Data Manager
    • on any property dialog box that displays the Data Server name (for example, the Data Source dialog box)

    is the name of the first Data Server the Studio connected to. If the Data Server is changed using the Data Server Connection Wizard, the Studio shows the correct Data Server name.

NOTE: Both of these issues have been resolved in Moxie 7.03 and higher.