Update Sharded Cluster to Keyfile Authentication (No Downtime)

Overview

Important

The following procedure applies to sharded clusters using MongoDB3.4 or later.

Earlier versions of MongoDB do not support no-downtime upgrade. Forsharded clusters using earlier versions of MongoDB, seeUpdate Sharded Cluster to Keyfile Authentication.

A MongoDB sharded cluster can enforce user authentication as well as internal authentication of its components to secure against unauthorizedaccess.

The following tutorial describes a procedure usingsecurity.transitionToAuth to transition an existing shardedcluster to enforce authentication without incurring downtime.

Before you attempt this tutorial, please familiarize yourself with thecontents of this document.

Considerations

Cloud Manager and Ops Manager

If you are using Cloud Manager or Ops Manager to manage your deployment,refer to _Configure Access Control for MongoDB Deployments_in the Cloud Manager manualor Ops Manager manual to enforce authentication.

IP Binding

Changed in version 3.6.

Starting with MongoDB 3.6, MongoDB binaries, mongod andmongos, bind to localhost by default.From MongoDB versions 2.6 to 3.4, only the binaries from theofficial MongoDB RPM (Red Hat, CentOS, Fedora Linux, and derivatives)and DEB (Debian, Ubuntu, and derivatives) packages would bind tolocalhost by default. To learn more about this change, seeLocalhost Binding Compatibility Changes.

Internal and Client Authentication Mechanisms

This tutorial configures authentication usingSCRAM for client authentication and akeyfile for internal authentication.

Refer to the Authentication documentation for a complete list ofavailable client and internal authentication mechanisms.

Architecture

This tutorial assumes that each shard replica set, as well as the configserver replica set, can elect a new primary after stepping down itsexisting primary.

A replica set can elect a primary only if both of the following conditions aretrue:

Minimum number of mongos instances

Ensure your sharded cluster has at least two mongos instancesavailable. This tutorial requires restarting each mongos inthe cluster. If your sharded cluster has only one mongosinstance, this results in downtime during the period that themongos is offline.

Enforce Keyfile Access Control on an Existing Sharded Cluster

Create and Distribute the Keyfile

With keyfile authentication, eachmongod or mongos instances in the sharded cluster uses the contents of the keyfile as theshared password for authenticating other members in the deployment. Onlymongod or mongos instances with the correct keyfile can join the sharded cluster.

Note

Starting in MongoDB 4.2, keyfiles for internal membershipauthentication use YAML format to allow formultiple keys in a keyfile. The YAML format accepts content of:

  • a single key string (same as in earlier versions),
  • multiple key strings (each string must be enclosed in quotes), or
  • sequence of key strings.

The YAML format is compatible with the existing single-keykeyfiles that use the text file format.

A key’s length must be between 6 and 1024 characters and may onlycontain characters in the base64 set. All members of thesharded cluster must share at least one common key.

Note

On UNIX systems, the keyfile must not have group or worldpermissions. On Windows systems, keyfile permissions are not checked.

You can generate a keyfile using any method you choose. For example,the following operation uses openssl to generate a complexpseudo-random 1024 character string to use as a shared password. It thenuses chmod to change file permissions to provide readpermissions for the file owner only:

  1. openssl rand -base64 755 > <path-to-keyfile>
  2. chmod 400 <path-to-keyfile>

Copy the keyfile to each server hosting the sharded cluster members.Ensure that the user running the mongod or mongos instances is the owner of thefile and can access the keyfile.

Avoid storing the keyfile on storage mediums that can be easilydisconnected from the hardware hosting the mongod or mongos instances, such as aUSB drive or a network attached storage device.

For more information on using keyfiles for internal authentication, refer toKeyfiles.

Configure Sharded Cluster Admin User and Client Users

You must connect to a mongos to complete the steps in thissection. The users created in these steps are cluster-level users andcannot be used for accessing individual shard replica sets.

Create the adminstrator user.

Use the db.createUser() method to create an administratoruser and assign it the following roles:

Clients performing maintenance operations or user administrativeoperations on the sharded cluster must authenticate as this userat the completion of this tutorial. Create this user now to ensurethat you have access to the cluster after enforcing authentication.

  1. admin = db.getSiblingDB("admin");
  2. admin.createUser(
  3.   {
  4.     user: "admin",
  5.     pwd: "<password>",
  6.     roles: [
  7.       { role: "clusterAdmin", db: "admin" },
  8.       { role: "userAdmin", db: "admin" }
  9.     ]
  10.   }
  11. );

Important

Passwords should be random, long, and complex to prevent or hindermalicious access.

Optional: Create additional users for client applications.

In addition to the administrator user, you can create additionalusers before enforcing authentication.. This ensures access to thesharded cluster once you fully enforce authentication.

Example

The following operation creates the user joe on themarketing database, assigning to this user thereadWrite role on the marketing database`.

  1. db.getSiblingDB("marketing").createUser(
  2.   {
  3.     "user": "joe",
  4.     "pwd": "<password>",
  5.     "roles": [ { "role" : "readWrite", "db" : "marketing" } ]
  6.   }
  7. )

Clients authenticating as "joe" can perform read and writeoperations on the marketing database.

See Database User Roles for roles provided by MongoDB.

See the Add Users tutorial for more information onadding users. Consider security best practices when adding new users.

Optional: Update client applications to specify authentication credentials.

While the sharded cluster does not currently enforce authentication, you canstill update client applications to specify authentication credentials whenconnecting to the sharded cluster. This may prevent loss of connectivity atthe completion of this tutorial.

Example

The following operation connects to the sharded cluster using themongo shell, authenticating as the user joe on themarketing database.

  1. mongo  --username "joe" --password "<password>" \
  2.   --authenticationDatabase "marketing" --host mongos1.example.net:27017

If your application uses a MongoDB driver, see the associateddriver documentation for instructions on creatingan authenticated connection.

Transition Each mongos Instance to Enforce Authentication

Create a new mongos configuration file.

For each mongos:

  • Copy the existing mongos configuration file, giving it adistinct name such as <filename>-secure.conf (or .cfg ifusing Windows). You will use this new configuration file totransition the mongos to enforce authentication in thesharded cluster. Retain the original configuration file for backuppurposes.

  • To the new configuration file, add the following settings:

If using a different internal authentication mechanism, specifysettings appropriate for the mechanism.

  1. security:
  2.    transitionToAuth: true
  3.    keyFile: <path-to-keyfile>

The new configuration file should contain all of the configurationsettings previously used by the mongos as well as the newsecurity settings.

One at a time, restart the mongos with the new configuration file.

Note

If your cluster has only one mongos, this step resultsin downtime while the mongos is offline.

Follow the procedure to restart the mongos instance, onemongos at a time:

  1. db.getSiblingDB("admin").shutdownServer()
  • Restart mongos with the new configuration file,specifying the path to the config file using—config. For example, if the new configuration filewere named mongos-secure.conf:
  1. mongos --config <path>/mongos-secure.conf

where <path> represents the system path to the foldercontaining the new configuration file.

Repeat the restart process for the next mongos instanceuntil all mongos instances in the sharded cluster havebeen restarted.

At the end of this section, all mongos instances in thesharded cluster are running with security.transitionToAuthand security.keyFile internal authentication.

Transition Config Server Replica Set Members to Enforce Authentication

Create a new mongod configuration file.

For each mongod in the config server replica set,

  • Copy the existing mongod configuration file, giving it adistinct name such as <filename>-secure.conf (or .cfg ifusing Windows). You will use this new configuration file totransition the mongod to enforce authentication in thesharded cluster. Retain the original configuration file forbackup purposes.

  • To the new configuration file, add the following settings:

If using a different internal authentication mechanism, specifysettings appropriate for the mechanism.

  1. security:
  2.    transitionToAuth: true
  3.    keyFile: <path-to-keyfile>

One at a time, restart the mongod with the new configuration file.

Restart the replica set, one member at a time, starting with thesecondary members.

  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the newconfiguration file, specifying the path to the config fileusing —config. For example, if the newconfiguration file were named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the new configuration file.

Once this member is up, repeat for the next secondary member.

  • Once all the secondary members have restarted and are up,restart the primary:

    • Connect to the mongod.

    • Use the rs.stepDown() method to step down the primary and trigger an election.

  1. rs.stepDown()

You can use the rs.status() method to ensure thereplica set has elected a new primary.

  • Once you step down the primary and a new primary has beenelected, shut down the old primary using thedb.shutdownServer() method against the admindatabase.
  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the newconfiguration file, specifying the path to the config file using—config. For example, if the new configuration filewere named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the new configuration file.

At the end of this section, all mongod instances in the configserver replica set is running with security.transitionToAuth andsecurity.keyFile internal authentication.

Transition Each Shard Replica Set Members to Enforce Authentication

Create the shard-local administrator

In a sharded cluster that enforces authentication, each shard replicaset should have its own shard-local administrator. You cannot use a shard-local administrator forone shard to access another shard or the sharded cluster.

Connect to the primary member of each shard replica set and create auser with the db.createUser() method, assigning it the followingroles:

Tip

Starting in version 4.2 of the mongo shell, you canuse the passwordPrompt() method in conjunction withvarious user authentication/management methods/commands to promptfor the password instead of specifying the password directly in themethod/command call. However, you can still specify the passworddirectly as you would with earlier versions of themongo shell.

  1. admin = db.getSiblingDB("admin")
  2. admin.createUser(
  3.   {
  4.     user: "admin",
  5.     pwd: passwordPrompt(),  // or cleartext password
  6.     roles: [
  7.       { role: "clusterAdmin", db: "admin" },
  8.       { role: "userAdmin", db: "admin" }
  9.     ]
  10.   }
  11. )

At the completion of this tutorial, if you want to connect to the shardto perform maintenance operation that require direct connection to ashard, you must authenticate as the shard-local administrator.

Note

Direct connections to a shard should only be for shard-specificmaintenance and configuration. In general, clients should connect tothe sharded cluster through the mongos.

Procedure

Transitioning one shard replica set at a time, repeat these steps foreach shard replica set in the sharded cluster.

Create a new mongod configuration file.

For each mongod in the shard replica set,

  • Copy the existing mongod configuration file, giving it adistinct name such as <filename>-secure.conf (or .cfg ifusing Windows). You will use this new configuration file totransition the mongod to enforce authentication in thesharded cluster. Retain the original configuration file forbackup purposes.

  • To the new configuration file, add the following settings:

If using a different internal authentication mechanism, specifysettings appropriate for the mechanism.

  1. security:
  2.    transitionToAuth: true
  3.    keyFile: <path-to-keyfile>
One at a time, restart the mongod with the new configuration file.

Restart the replica set, one member at a time, starting with thesecondary members.

  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the newconfiguration file, specifying the path to the config fileusing —config. For example, if the newconfiguration file were named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the new configuration file.

Once this member is up, repeat for the next secondary member of the replica set until all secondaries have been updated.

  • Once all the secondary members have restarted and are up,restart the primary:

    • Connect to the mongod.

    • Use the rs.stepDown() method to step down the primary and trigger an election.

  1. rs.stepDown()

You can use the rs.status() method to ensure thereplica set has elected a new primary.

  • Once you step down the primary and a new primary has beenelected, shut down the old primary using thedb.shutdownServer() method against the admindatabase.
  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the newconfiguration file, specifying the path to the config file using—config. For example, if the new configuration filewere named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the new configuration file.

At this point in the tutorial, every component of the sharded cluster isrunning with —transitionToAuth and security.keyFileinternal authentication. The sharded cluster has at least one administrativeuser, and each shard replica set has a shard-local administrative user.

The remaining sections involve taking the sharded cluster out of thetransition state to fully enforce authentication.

Restart Each mongos Instance without transitionToAuth

Important

At the end of this section, clients must specify authenticationcredentials to connect to the sharded cluster. Update clients to specifyauthentication credentials before completing this section to avoid loss ofconnectivity.

To complete the transition to fully enforcing authentication in thesharded cluster, you must restart each mongos instance withoutthe security.transitionToAuth setting.

Remove transitionToAuth from the mongos configuration files.

Remove the security.transitionToAuth key and its valuefrom the mongos configuration files created during thistutorial. Leave the security.keyFile setting added in thetutorial.

  1. security:
  2.    keyFile: <path-to-keyfile>

Restart the mongos with the updated configuration file.

Note

If your cluster has only one mongos, this step resultsin downtime while the mongos is offline.

Follow the procedure to restart mongos instance, onemongos at a time:

  1. db.getSiblingDB("admin").shutdownServer()
  • Restart mongos with the updated configuration file,specifying the path to the config file using—config. For example, if the updated configuration filewere named mongos-secure.conf:
  1. mongos --config <path>/mongos-secure.conf

At the end of this section, all mongos instances enforce clientauthentication and security.keyFile internal authentication.

Restart Each Config Server Replica Set Member without transitionToAuth

Important

At the end of this step, clients must specify authentication credentials toconnect to the config server replica set. Update clients to specifyauthentication credentials before completing this section to avoid loss ofconnectivity.

To complete the transition to fully enforcing authentication in thesharded cluster, you must restart each mongod instance withoutthe security.transitionToAuth setting.

Remove transitionToAuth from the mongod configuration files.

Remove the security.transitionToAuth key and its valuefrom the config server configuration files created during thistutorial. Leave the security.keyFile setting added in thetutorial.

  1. security:
  2.    keyFile: <path-to-keyfile>

One at a time, restart the mongod with the updated configuration file.

Restart the replica set, one member at a time, starting with thesecondary members.

  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the updatedconfiguration file, specifying the path to the config fileusing —config. For example, if the newconfiguration file were named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the updated configuration file.

Once this member is up, repeat for the next secondary member.

  • Once all the secondary members have restarted and are up,restart the primary:

    • Connect to the mongod.

    • Use the rs.stepDown() method to step down the primary and trigger an election.

  1. rs.stepDown()

You can use the rs.status() method to ensure thereplica set has elected a new primary.

  • Once you step down the primary and a new primary has beenelected, shut down the old primary using thedb.shutdownServer() method against the admindatabase.
  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the updatedconfiguration file, specifying the path to the config file using—config. For example, if the new configuration filewere named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the updated configuration file.

At the end of this section, all mongod instances in the configserver replica set enforce client authentication andsecurity.keyFile internal authentication.

Restart Each Member in Each Shard Replica Set without transitionToAuth

Important

At the end of this step, clients must specify authentication credentials toconnect to the shard replica set. Update clients to specify authenticationcredentials before completing this section to avoid loss of connectivity.

To complete the transition to fully enforcing authentication in the shardedcluster, you must restart every member of every shard replica set in thesharded cluster without the security.transitionToAuth setting.

Transitioning one shard replica set at a time, repeat these steps foreach shard replica set in the sharded cluster.

Remove transitionToAuth from the mongod configuration files.

Remove the security.transitionToAuth key and its valuefrom the config server configuration files created during thistutorial. Leave the security.keyFile setting added in thetutorial.

  1. security:
  2.    keyFile: <path-to-keyfile>

One at a time, restart the mongod with the updated configuration file.

Restart the replica set, one member at a time, starting with thesecondary members.

  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the updatedconfiguration file, specifying the path to the config fileusing —config. For example, if the newconfiguration file were named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the updated configuration file.

Once this member is up, repeat for the next secondary member.

  • Once all the secondary members have restarted and are up,restart the primary:

    • Connect to the mongod.

    • Use the rs.stepDown() method to step down the primary and trigger an election.

  1. rs.stepDown()

You can use the rs.status() method to ensure thereplica set has elected a new primary.

  • Once you step down the primary and a new primary has beenelected, shut down the old primary using thedb.shutdownServer() method against the admindatabase.
  1. db.getSiblingDB("admin").shutdownServer()
  • Restart the mongod with the updatedconfiguration file, specifying the path to the config file using—config. For example, if the new configuration filewere named mongod-secure.conf:
  1. mongod --config <path>/mongod-secure.conf

where <path> represents the system path to the foldercontaining the updated configuration file.

At the end of this section, all mongos and mongodinstances in the sharded cluster enforce client authentication andsecurity.keyFile internal authentication. Clients can only connectto the sharded cluster by using the configured client authentication mechanism.Additional components can only join the cluster by specifying the correctkeyfile.

x.509 Certificate Internal Authentication

MongoDB supports x.509 certificate authentication for use with a secure TLS/SSLconnection. Sharded cluster members and replica set members can use x.509certificates to verify their membership to the cluster or the replica setinstead of using Keyfiles.

For details on using x.509 certificates for internal authentication, seeUse x.509 Certificate for Membership Authentication.

Upgrade from Keyfile Authentication to x.509 Authentication describes how to upgrade adeployment’s internal auth mechanism from keyfile-based authentication to x.509certificate-based auth.