Skip to main content

Enabling High Availability

Documentation Team
  • Updated

Enabling High Availability

High availability (HA) allows you host two or more separate instances and having them read data from a shared location, such as a network share or drive.

High availability on Dashboard Server works by mirroring the files on the share back onto the local disk.

This style of deployment is typically used to:

  • Enable load-balancing between servers hosting the same content
  • Create two or more different access points into Dashboard Server with different modes of authentication (e.g. one instance with Windows authentication and the other with forms)

Distributed file system (DFS) shares are supported, under the proviso that DFSR (replication) is either disabled, or works in a limited capacity on Dashboard Server files. For example, performing replication only at specific times or manually when Dashboard Server is not running. If DFSR is used then the Primary and Secondary servers will not see the same information.

You may also like to watch this training webinar about 'Dashboard Server v4 High availability' (25 mins):

Requirements

  • Two or more separate servers for installing Dashboard Server v4 or above. For high availability to work, every SquaredUp server must be running exactly the same version.
  • A shared network folder or drive
  • A Dashboard Server license with a Secondary activation key

A Teams edition license (or above) is required for this feature.

To upgrade please contact sales@squaredup.com.

To check the license edition you are using see How to check which license key is being used. To see what is included in different product edition licenses see the Licensing Overview.

Understanding Primary and Secondary servers

Decide which server will be your Primary SquaredUp server and which will be your Secondary server(s):

  • A high availability setup must consist of at least one Primary server. Without a Primary server, the servers licensed as Secondary will operate with reduced functionality and named users.
  • The Primary server will use the Primary license key, and any existing dashboards on this server will become the dashboards on the share, to be used by all servers.
  • The Secondary server(s) will use the Secondary license key, the dashboards on these servers will be ignored and the content of the Primary server will take precedence. You can have one or more Secondary servers.

If a Secondary server is out of contact with the Primary server you will see a notification in the notifications area from the right-hand menu > notifications.

  • Within 2-3 hours of the Primary server being unavailable for any reason you will receive a notification.
  • After 3 days of no contact with the Primary server a critical notification is shown warning that shut down is imminent.
  • After 5 days of no contact the SquaredUp server will no longer be available, effectively 'shut down'.
  • Once the SquaredUp server has shut down you can recycle the Secondary server's Dashboard Server application pool to restore limited access. This will only give you 5 named users and features such as Open Access, VADA, Web API and Visio tiles will be unavailable.

Configure permissions on the share

Before configuring HA it is important to check the permissions on the share itself. The share has its own permissions which cannot be viewed via Windows Explorer, and they always take precedence over the file and folder permissions. If the Dashboard Server application pool identity (How to check and modify the application pool identity) is not allowed to read and change, then the permissions added to the folders by the squaredup4 ha or squaredup5 ha command (depending on the version of Dashboard Server installed) are overridden/ignored by the share, and Dashboard Server will fail to start with an 'Access denied' error.

In Computer Management check that the Dashboard Server application pool account has 'Read' and 'Change' permissions to the share itself, as described below:

  1. On the machine that hosts the share, click on the Start button > type Computer Management > open Computer Management.
  2. Navigate to System Tools > Shared Folders > Shares.
  3. Right-click on the SquaredUp share and then on Properties.

  4. On the Share Permissions tab, check that all Dashboard Server application pool accounts have read and change permissions:

  5. If your servers are using a domain service account as the Dashboard Server application pool identity (How to check and modify the application pool identity) then this account should be given read and change permissions.

    If your SquaredUp servers are using Network Service as the Dashboard Server application pool account (How to check and modify the application pool identity), then all the machine accounts for the SquaredUp Primary and Secondary servers should be given permissions, for example, Squpserver01$ and Squpserver02$.

    It should not be necessary to grant full control, and this could pose a security risk.

Configuring each server for high availability

The first server that you configure for HA, will be the one from which dashboards and profiles are copied to the HA share. Follow the steps below for each server, one at a time:

  1. Install and activate Dashboard Server using the appropriate Primary or Secondary activation key (see How to install Dashboard Server SCOM Edition and How to activate your license).
  2. On the server open a command prompt as an administrator (from Start > type cmd, right-click on the Command Prompt icon and click Run as administrator).

  3. Change to the Dashboard Server installation location, for example type:

    cd C:\inetpub\wwwroot\SquaredUpv5

    Name of the Dashboard Server folder

    The name of the Dashboard Server folder is SquaredUpv followed by the product version number.

    Location of the Dashboard Server folder

    The default location for the Dashboard Server folder is C:\inetpub\wwwroot\SquaredUpv[Version Number], but a custom location may have been chosen during the installation.
    For example, for Dashboard Server v5 the default folder location is C:\inetpub\wwwroot\SquaredUpv5 and for v4 C:\inetpub\wwwroot\SquaredUpv4

  4. Identify whether the Dashboard Server application pool identity is running as a domain service account or as NetworkService (How to check and modify the application pool identity) and run the appropriate squaredup4 ha or squaredup5 ha command as described below to configure Dashboard Server for HA:

    If the Dashboard Server application pool identity is a domain service account use this account in the command:

    On Dashboard Server v5:

    squaredup5 ha --path=<network share path> --user=domain\user

    On Dashboard Server v4:

    squaredup4 ha --path=<network share path> --user=domain\user

    where domain\user is the Dashboard Server application pool identity. First check what the Dashboard Server application pool account is for each server is using(How to check and modify the application pool identity). In a load balanced environment using Windows authentication (Kerberos), both servers must be using the same application pool identity. (In environments not using Kerberos, it is possible for the servers to use different application pool identities). This allows the tool to give the specified Dashboard Server application pool account read and write permissions to the folder and files on the share.

    and where <network share path> should be replaced by a drive or path specification for your network share. The folder/share must already exist: Dashboard Server cannot create it automatically (for example, specifying \\myhost\folder is invalid if folder is not already shared by myhost). The path should not contain a filename (i.e. it should be \myshare etc. and not \myshare\squaredup.index).

    For example:

    • X:\
    • \\myhost\folder (UNC path)

    If your Dashboard Server application pool identity is NetworkService you should use the SquaredUp server name followed by the $ dollar symbol as the username when you run the command on that server.

    On Dashboard Server v5:

    squaredup5 ha --path=<network share path> --user=domain\SquaredUpServer$

    On Dashboard Server v4:

    squaredup4 ha --path=<network share path> --user=domain\SquaredUpServer$

    where SquaredUpServer$ is the SquaredUp server name followed by $, for example, Squpserver01$.

    Remember, that the first server that you run this command on will be the one from which dashboards and profiles are copied to the HA share. Later, after checking the Primary server is configured correctly you will run this command on all your other SquaredUp servers to configure each of them to use HA AND to give them permissions to the HA share.

  5. Navigate to Dashboard Server using a web browser either on the server itself, or from a client machine. (Note: The previous command will have automatically recycled Dashboard Server, so you will need to login again).

  6. After logging in, the server should behave identically to how it did post-installation. There are several ways to confirm that HA is in effect:

    • The path to which HA has been pointed (e.g. \\myhost\folder) should contain a file called squaredup_scom_[version].index (or squaredup_[version].index if HA was configured on v5.0 or below).
    • The Dashboard Server log (\SquaredUpv5\transient\log\rolling.log) should contain the text

    For the Primary server:

    [WRN] Shared cryptography is enabled: behaving as a PRIMARY server

    For a Secondary server:

    [WRN] Shared cryptography is enabled: behaving as a SECONDARY server

  7. Once the Primary is confirmed to be running in HA mode, the Secondary server(s) can be configured using the Secondary license key. Repeat the above steps for the Secondary servers. Content already present on these servers will be ignored and not displayed. Instead, each Secondary server will now behave as an exact mirror of the Primary server.

  8. After configuring HA, log in to the Secondary server(s) and check the configuration:

    • The licensing details for the Secondary in the right-hand menu ☰ > system > named users should reflect the overall quantity of users that your license was purchased for
    • The Secondary should now be displaying the same dashboards and content as the Primary.
    • Newly created content on Primary or Secondary should be visible to both nodes.

Upgrading Dashboard Server when using high availability

For high availability to work, every SquaredUp server must be running exactly the same version.

Upgrading Dashboard Server v4 and above in a high availability setup no longer requires all servers go offline for upgrade. Instead, each server can now be upgraded one at a time, and the other servers will continue to serve dashboards.

However, once upgrade of at least one server is performed, changes made to dashboards by servers running the old version of Dashboard Server are ignored, and will be lost when those servers are eventually upgraded (that is to say, the dashboards and content in the upgraded servers becomes authoritative over dashboards and content from the older un-upgraded servers).

  • 10 minutes after upgrade of a server is performed, the servers yet-to-be-upgraded will automatically enter a read-only state, to prevent new dashboards being created or edited. This is visible in Dashboard Server as a yellow banner at the top of the page.
  • Dashboards (or other content) created or edited on an un-upgraded server before the server automatically goes read-only, will be permanently lost once the server is upgraded to the newer version. For this reason, we recommend advising users to not make changes once you start your upgrade process.

Consider initiating your upgrades out-of-hours - even if you don't finish them on all servers. This way, all of the servers yet to be upgraded will have entered read-only mode automatically by the time users come to view or edit their content.

Dashboard Server v4.5 fixes the issue of the named user list not being synchronised across HA servers. When upgrading to v4.5 the first server that you upgrade will be the one from which the named user list is copied to the HA share and that list becomes the authoritative version. The named user list from subsequently upgraded servers will be lost, as the single authoritative named user list is synchronised across HA servers.

Follow the steps below to upgrade each server:

  1. Prepare to take the server you are upgrading offline. For example, notify users, disable the load balancer allocation for it, put it in maintenance mode etc.

  2. Download the latest version of Dashboard Server, run the installer and upgrade the server.

    Get the latest version of SCOM Edition

  3. Log on to the Dashboard Server on the server once upgraded.

  4. Upgrade any other SquaredUp servers.

Disabling high availability

You may wish to switch high availability off for one or more servers, for example if one of the servers is going to be unavailable for a long period of time. Remember, for Dashboard Server to work with your full license allocation, you need to ensure that there is still a Primary licensed server available.

Dashboard Server mirrors the content of the share location back to the local disk: So that each member of a high availability set is an approximate replica, even when disconnected from the share. This means that HA can safely be disabled at any time, and once disabled the content of Dashboard Server will reflect the network share data at the point just before the link was broken.

  1. On the SquaredUp server open a command prompt as an administrator (from Start > type cmd, right-click on the Command Prompt icon and click Run as administrator).

  2. Change to the Dashboard Server installation location, for example type:

    cd C:\inetpub\wwwroot\squaredupv5

    Name of the Dashboard Server folder

    The name of the Dashboard Server folder is SquaredUpv followed by the product version number.

    Location of the Dashboard Server folder

    The default location for the Dashboard Server folder is C:\inetpub\wwwroot\SquaredUpv[Version Number], but a custom location may have been chosen during the installation.
    For example, for Dashboard Server v5 the default folder location is C:\inetpub\wwwroot\SquaredUpv5 and for v4 C:\inetpub\wwwroot\SquaredUpv4

  3. Run the following command to disable HA:

    On Dashboard Server v5:

    squaredup5 ha --disable

    On Dashboard Server v4:

    squaredup4 ha --disable

You will also need to reconfigure the load balancer allocation to take into account the server(s) no longer available.

Moving the HA share

To move the HA share, the initial setup steps can simply be repeated as below, just specifying a new share path. This is because each server maintains a local disk copy of the share content - and the first server to startup after being pointed at the new share will build the share’s content from its own.

  1. Create the new share and follow the steps to Configure permissions on the share.

  2. On any SquaredUp server run the steps to Configuring each server for high availability using the path of the new share.

  3. Browse to Dashboard Server in a web browser and log in. This will restart Dashboard Server as the previous command will have automatically recycled the Dashboard Server application pool. The files will be copied to the share from the first Dashboard Server instance to start up. It should not matter which Dashboard Server instance this is, as they should all be in sync, having been using HA previously. If you need to specify form which Dashboard Server instance the files are copied to the new share, you should take care to open this Dashboard Server instance by browsing to it directly, for example

    https://SquaredUpServer1/SquaredUpv5

    or

    https://SquaredUpServer1/SquaredUpv4

  4. Run the steps to Configuring each server for high availability on all the other servers using the path of the new share. This will point all the servers to the new share and the files that were copied there in the previous step.

Backing up and restoring the HA share

Backup Dashboard Server on each server as normal (How to backup and restore Dashboard Server SCOM Edition). No additional actions are required: Any SquaredUp server can rebuild the content of the network share from its local disk data. So backing up individual Dashboard Server instances (and not the network share) is sufficient.

If the share does have to be recreated, ensure you configure permissions on the share first:

Before configuring HA it is important to check the permissions on the share itself. The share has its own permissions which cannot be viewed via Windows Explorer, and they always take precedence over the file and folder permissions. If the Dashboard Server application pool identity (How to check and modify the application pool identity) is not allowed to read and change, then the permissions added to the folders by the squaredup4 ha or squaredup5 ha command (depending on the version of Dashboard Server installed) are overridden/ignored by the share, and Dashboard Server will fail to start with an 'Access denied' error.

In Computer Management check that the Dashboard Server application pool account has 'Read' and 'Change' permissions to the share itself, as described below:

  1. On the machine that hosts the share, click on the Start button > type Computer Management > open Computer Management.
  2. Navigate to System Tools > Shared Folders > Shares.
  3. Right-click on the SquaredUp share and then on Properties.

  4. On the Share Permissions tab, check that all Dashboard Server application pool accounts have read and change permissions:

  5. If your servers are using a domain service account as the Dashboard Server application pool identity (How to check and modify the application pool identity) then this account should be given read and change permissions.

    If your SquaredUp servers are using Network Service as the Dashboard Server application pool account (How to check and modify the application pool identity), then all the machine accounts for the SquaredUp Primary and Secondary servers should be given permissions, for example, Squpserver01$ and Squpserver02$.

    It should not be necessary to grant full control, and this could pose a security risk.

To restore the files to a new or empty HA share run the HA configuration steps from any SquaredUp server, to copy the files from this server to the share. See Enabling High Availability

To restore to an existing share folder you must delete the file SquaredUp_scom_[version].index, if it exists, in order for the restore to proceed. Restoring will then recreate the SquaredUp_scom_[version].index file on the share. (This file will be called SquaredUp_[version].index if HA was configured on v5.0 or below).

FAQs

Can I configure each server to use a different SCOM server, Data Warehouse or Open Access loopback URL?

Yes, see How to set up per-server configuration when using high availability (HA).

Troubleshooting

Only 5 users can login and several features such as Open Access, VADA, Web API and Visio tiles are unavailable

Check that HA mode is enabled, and that the Primary server is available. If the Secondary server has not been able to contact the Primary server for 5 days, or if HA mode is disabled, then the Secondary server will drop back to 5 named users and features such as Open Access, Web API and Visio tiles will be unavailable.

Dashboard Server is displaying a message that it is in read-only mode

A Dashboard Server instance will go into read-only mode if it notices that it is running an older version to another server.

Read-only mode will end once the server is upgraded to the same version.

My providers, PowerShell profiles, and PowerShell Run As accounts are out of sync across HA servers

In a High Availability environment, it can happen that the following items are not properly synced across servers:

Providers contain the connection details to external platforms. A provider only needs to be set up once and can then be used when creating tiles on a dashboard.

There are two types of integrations and therefore providers:

  • generic Web API providers that can connect to any REST API

  • dedicated providers that connect to a specific external platform or database (SQL, ServiceNow, Azure Active App Insights, Elasticsearch, etc.)

PowerShell profiles contain re-usable scripts with encrypted sensitive data.

A PowerShell profile is created once and then can be re-used in PowerShell tiles. Only administrators can create PowerShell profiles. Since PowerShell profile scripts are encrypted and can only be seen by administrators, you can safely store scripts that contain credentials, authentication tokens, etc. You can also load external modules in a profile (e.g. a VMWare module downloaded from the internet).

You can also use PowerShell profiles for more sophisticated code, for example if your tile needs to combine data from two different API connections, you can put credentials for both connections in the profile. Make sure to give your profile a meaningful description to remember which provider(s) the profile connects to and what it does with the data.

Dashboard Server users who can edit tiles due to their Team Folder permissions can use PowerShell profiles in their PowerShell tiles, but they can’t see the underlying script.

PowerShell Run As accounts contain the credentials that define the permissions deciding how PowerShell scripts are run (both the script in the tile and the profile script chosen for the tile).

The PowerShell Run As account Default comes with every Dashboard Server installation and uses the Dashboard Server app pool identity to run the scripts. Since running PowerShell scripts within the Dashboard Server application pool process can pose a security risk and affect Dashboard Server performance, you can change the default Run As to use a different account.

You can also add new Run As accounts to be able to execute scripts with different credentials.

Solution:

On your primary server, make an edit to any of your providers, PowerShell profiles, or Run As accounts.

Editing one provider will trigger a sync between all your providers across all your servers. Editing one PowerShell profile will sync all your PowerShell profiles, and editing one Run As account will sync all your Run As accounts.