mapReduce

  • mapReduce
  • The mapReduce command allows you to runmap-reduce aggregation operations over a collection.

Note

Starting in version 4.2, MongoDB deprecates:

  • The map-reduce option to create a new sharded collection as wellas the use of the sharded option formap-reduce. To output to a sharded collection, create the shardedcollection first. MongoDB 4.2 also deprecates the replacement ofan existing sharded collection.
  • The explicit specification of nonAtomic: false option.

ThemapReduce command has the following prototypeform:

  1. db.runCommand(
  2.                {
  3.                  mapReduce: <collection>,
  4.                  map: <function>,
  5.                  reduce: <function>,
  6.                  finalize: <function>,
  7.                  out: <output>,
  8.                  query: <document>,
  9.                  sort: <document>,
  10.                  limit: <number>,
  11.                  scope: <document>,
  12.                  jsMode: <boolean>,
  13.                  verbose: <boolean>,
  14.                  bypassDocumentValidation: <boolean>,
  15.                  collation: <document>,
  16.                  writeConcern: <document>
  17.                }
  18.              )

Pass the name of the collection to the mapReduce command(i.e. <collection>) to use as the source documents to performthe map-reduce operation.

Note

Views do not support map-reduce operations.

The command also accepts the following parameters:

FieldTypeDescriptionmapReducecollectionThe name of the collection on which you want to perform map-reduce.This collection will be filtered using query before being processedby the map function.mapfunctionA JavaScript function that associates or “maps” a value with akey and emits the key and value pair.

See Requirements for the map Function for more information.reducefunctionA JavaScript function that “reduces” to a single object all thevalues associated with a particular key.

See Requirements for the reduce Function for more information.outstring or documentSpecifies where to output the result of the map-reduce operation. Youcan either output to a collection or return the result inline. On aprimary member of a replica set you can output either to a collectionor inline, but on a secondary, only inline output ispossible.

See out Options for more information.querydocumentOptional. Specifies the selection criteria using query operators for determining the documents input to themap function.sortdocumentOptional. Sorts the input documents. This option is useful foroptimization. For example, specify the sort key to be the same asthe emit key so that there are fewer reduce operations. The sort keymust be in an existing index for this collection.limitnumberOptional. Specifies a maximum number of documents for the input into themap function.finalizefunctionOptional. Follows the reduce method and modifies the output.

See Requirements for the finalize Function for more information.scopedocumentOptional. Specifies global variables that are accessible in the map,reduce and finalize functions.jsModebooleanOptional. Specifies whether to convert intermediate data into BSONformat between the execution of the map and reducefunctions.

Defaults to false.

If false:

  • Internally, MongoDB converts the JavaScript objects emittedby the mapfunction to BSON objects. These BSONobjects are then converted back to JavaScript objects whencalling the reduce function.

  • The map-reduce operation places the intermediate BSON objectsin temporary, on-disk storage. This allows the map-reduceoperation to execute over arbitrarily large data sets.If true:

  • Internally, the JavaScript objects emitted during mapfunction remain as JavaScript objects. There is no need toconvert the objects for the reduce function, whichcan result in faster execution.

  • You can only use jsMode for result sets with fewer than500,000 distinct key arguments to the mapper’s emit()function.verbosebooleanOptional. Specifies whether to include the timing information in theresult information. Set verbose to true to includethe timing information.

Defaults to false.bypassDocumentValidationbooleanOptional. Enables mapReduce to bypass document validationduring the operation. This lets you insert documents that do notmeet the validation requirements.

New in version 3.2.

Note

If the output option is set toinline, no document validation occurs. If the output goes toa collection, mapReduce observes any validationrules which the collection has and does not insert any invaliddocuments unless the bypassDocumentValidation parameter isset to true.

collationdocumentOptional.

Specifies the collation to use for the operation.

Collation allows users to specifylanguage-specific rules for string comparison, such as rules forlettercase and accent marks.

The collation option has the following syntax:

  1. collation: {
  2.    locale: <string>,
  3.    caseLevel: <boolean>,
  4.    caseFirst: <string>,
  5.    strength: <int>,
  6.    numericOrdering: <boolean>,
  7.    alternate: <string>,
  8.    maxVariable: <string>,
  9.    backwards: <boolean>
  10. }

When specifying collation, the locale field is mandatory; allother collation fields are optional. For descriptions of the fields,see Collation Document.

If the collation is unspecified but the collection has adefault collation (see db.createCollection()), theoperation uses the collation specified for the collection.

If no collation is specified for the collection or for theoperations, MongoDB uses the simple binary comparison used in priorversions for string comparisons.

You cannot specify multiple collations for an operation. Forexample, you cannot specify different collations per field, or ifperforming a find with a sort, you cannot use one collation for thefind and another for the sort.

New in version 3.4.

writeConcerndocumentOptional. A document that expresses the write concern to use when outputing to a collection.Omit to use the default write concern.

The following is a prototype usage of the mapReducecommand:

  1. var mapFunction = function() { ... };
  2. var reduceFunction = function(key, values) { ... };
  3.  
  4. db.runCommand(
  5.                {
  6.                  mapReduce: <input-collection>,
  7.                  map: mapFunction,
  8.                  reduce: reduceFunction,
  9.                  out: { merge: <output-collection> },
  10.                  query: <query>
  11.                }
  12.              )

JavaScript in MongoDB

Although mapReduce uses JavaScript, mostinteractions with MongoDB do not use JavaScript but use anidiomatic driver in the languageof the interacting application.

Requirements for the map Function

The map function is responsible for transforming each input document intozero or more documents. It can access the variables defined in the scopeparameter, and has the following prototype:

  1. function() {
  2.    ...
  3.    emit(key, value);
  4. }

The map function has the following requirements:

  • In the map function, reference the current document as thiswithin the function.
  • The map function should not access the database for any reason.
  • The map function should be pure, or have no impact outside ofthe function (i.e. side effects.)
  • A single emit can only hold half of MongoDB’s maximum BSONdocument size.
  • The map function may optionally call emit(key,value) any number oftimes to create an output document associating key with value.
  • Starting in version 4.2.1, MongoDB deprecates the use of JavaScriptwith scope (i.e. BSON type 15) forthe map function. To scope variables, use the scope parameterinstead.

The following map function will call emit(key,value) either0 or 1 times depending on the value of the input document’sstatus field:

  1. function() {
  2.     if (this.status == 'A')
  3.         emit(this.cust_id, 1);
  4. }

The following map function may call emit(key,value)multiple times depending on the number of elements in the inputdocument’s items field:

  1. function() {
  2.     this.items.forEach(function(item){ emit(item.sku, 1); });
  3. }

Requirements for the reduce Function

The reduce function has the following prototype:

  1. function(key, values) {
  2.    ...
  3.    return result;
  4. }

The reduce function exhibits the following behaviors:

  • The reduce function should not access the database,even to perform read operations.
  • The reduce function should not affect the outsidesystem.
  • MongoDB will not call the reduce function for a keythat has only a single value. The values argument is an arraywhose elements are the value objects that are “mapped” to thekey.
  • MongoDB can invoke the reduce function more than once for thesame key. In this case, the previous output from the reducefunction for that key will become one of the input values to the nextreduce function invocation for that key.
  • The reduce function can access the variables definedin the scope parameter.
  • The inputs to reduce must not be larger than half of MongoDB’smaximum BSON document size. Thisrequirement may be violated when large documents are returned and thenjoined together in subsequent reduce steps.
  • Starting in version 4.2.1, MongoDB deprecates the use of JavaScriptwith scope (i.e. BSON type 15) forthe reduce function. To scope variables, use the scopeparameter instead.

Because it is possible to invoke the reduce functionmore than once for the same key, the followingproperties need to be true:

  • the type of the return object must be identicalto the type of the value emitted by the mapfunction.

  • the reduce function must be associative. The following statement must betrue:

  1. reduce(key, [ C, reduce(key, [ A, B ]) ] ) == reduce( key, [ C, A, B ] )
  • the reduce function must be idempotent. Ensurethat the following statement is true:
  1. reduce( key, [ reduce(key, valuesArray) ] ) == reduce( key, valuesArray )
  • the reduce function should be commutative: that is, the order of theelements in the valuesArray should not affect the output of thereduce function, so that the following statement is true:
  1. reduce( key, [ A, B ] ) == reduce( key, [ B, A ] )

Requirements for the finalize Function

The finalize function has the following prototype:

  1. function(key, reducedValue) {
  2.    ...
  3.    return modifiedObject;
  4. }

The finalize function receives as its arguments a keyvalue and the reducedValue from the reduce function. Beaware that:

  • The finalize function should not access the database forany reason.
  • The finalize function should be pure, or have no impactoutside of the function (i.e. side effects.)
  • The finalize function can access the variables defined inthe scope parameter.
  • Starting in version 4.2.1, MongoDB deprecates the use of JavaScriptwith scope (i.e. BSON type 15) forthe finalize function. To scope variables, use the scopeparameter instead.

out Options

You can specify the following options for the out parameter:

Output to a Collection

This option outputs to a new collection, and is not available on secondarymembers of replica sets.

  1. out: <collectionName>

Output to a Collection with an Action

Note

Starting in version 4.2, MongoDB deprecates:

  • The map-reduce option to create a new sharded collection as wellas the use of the sharded option formap-reduce. To output to a sharded collection, create the shardedcollection first. MongoDB 4.2 also deprecates the replacement ofan existing sharded collection.
  • The explicit specification of nonAtomic: false option.

This option is only available when passing a collection thatalready exists to out. It is not availableon secondary members of replica sets.

  1. out: { <action>: <collectionName>
  2.         [, db: <dbName>]
  3.         [, sharded: <boolean> ]
  4.         [, nonAtomic: <boolean> ] }

When you output to a collection with an action, the out has thefollowing parameters:

  • <action>: Specify one of the following actions:

    • replace

Replace the contents of the <collectionName> if thecollection with the <collectionName> exists.

  • merge

Merge the new result with the existing result if theoutput collection already exists. If an existing documenthas the same key as the new result, overwrite thatexisting document.

  • reduce

Merge the new result with the existing result if theoutput collection already exists. If an existing documenthas the same key as the new result, apply the reducefunction to both the new and the existing documents andoverwrite the existing document with the result.

  • db:

Optional. The name of the database that you want the map-reduceoperation to write its output. By default this will be the samedatabase as the input collection.

  • sharded:

Note

Starting in version 4.2, the use of the sharded option isdeprecated.

Optional. If true and you have enabled sharding on outputdatabase, the map-reduce operation will shard the output collectionusing the _id field as the shard key.

If true and collectionName is an existing unsharded collection,map-reduce fails.

  • nonAtomic:

Note

Starting in MongoDB 4.2, explicitly setting nonAtomic to false isdeprecated.

Optional. Specify output operation as non-atomic. This applies onlyto the merge and reduce output modes, which may take minutes toexecute.

By default nonAtomic is false, and the map-reduceoperation locks the database during post-processing.

If nonAtomic is true, the post-processing step preventsMongoDB from locking the database: during this time, other clientswill be able to read intermediate states of the output collection.

Output Inline

Perform the map-reduce operation in memory and return the result. Thisoption is the only available option for out on secondary members ofreplica sets.

  1. out: { inline: 1 }

The result must fit within the maximum size of a BSON document.

Required Access

If your MongoDB deployment enforces authentication, the user executingthe mapReduce command must possess the followingprivilege actions:

Map-reduce with {out : inline} output option:

Map-reduce with the replace action when outputting to acollection:

Map-reduce with the merge or reduce actions whenoutputting to a collection:

The readWrite built-in role provides the necessarypermissions to perform map-reduce aggregation.

Restrictions

MongoDB drivers automatically set afterClusterTime for operations associated with causallyconsistent sessions. Starting in MongoDB 4.2, themapReduce command no longer support afterClusterTime. As such, mapReduce cannot beassociatd with causally consistent sessions.

Map-Reduce Examples

In the mongo shell, the db.collection.mapReduce()method is a wrapper around the mapReduce command. Thefollowing examples use the db.collection.mapReduce() method:

Consider the following map-reduce operations on a collectionorders that contains documents of the following prototype:

  1. {
  2.      _id: ObjectId("50a8240b927d5d8b5891743c"),
  3.      cust_id: "abc123",
  4.      ord_date: new Date("Oct 04, 2012"),
  5.      status: 'A',
  6.      price: 25,
  7.      items: [ { sku: "mmm", qty: 5, price: 2.5 },
  8.               { sku: "nnn", qty: 5, price: 2.5 } ]
  9. }

Return the Total Price Per Customer

Perform the map-reduce operation on the orders collection to groupby the cust_id, and calculate the sum of the price for eachcust_id:

  • Define the map function to process each input document:

    • In the function, this refers to the document that themap-reduce operation is processing.
    • The function maps the price to the cust_id for eachdocument and emits the cust_id and price pair.
  1. var mapFunction1 = function() {
  2.                        emit(this.cust_id, this.price);
  3.                    };

  • Define the corresponding reduce function with two argumentskeyCustId and valuesPrices:

    • The valuesPrices is an array whose elements are the pricevalues emitted by the map function and grouped by keyCustId.
    • The function reduces the valuesPrice array to thesum of its elements.
  1. var reduceFunction1 = function(keyCustId, valuesPrices) {
  2.                           return Array.sum(valuesPrices);
  3.                       };
  • Perform the map-reduce on all documents in the orders collectionusing the mapFunction1 map function and the reduceFunction1reduce function.
  1. db.orders.mapReduce(
  2.                      mapFunction1,
  3.                      reduceFunction1,
  4.                      { out: "map_reduce_example" }
  5.                    )

This operation outputs the results to a collection namedmap_reduce_example. If the map_reduce_example collectionalready exists, the operation will replace the contents with theresults of this map-reduce operation:

Calculate Order and Total Quantity with Average Quantity Per Item

In this example, you will perform a map-reduce operation on theorders collection for all documents that have an ord_datevalue greater than 01/01/2012. The operation groups by theitem.sku field, and calculates the number oforders and the total quantity ordered for each sku. The operation concludes bycalculating the average quantity per order for each sku value:

  • Define the map function to process each input document:

    • In the function, this refers to the document that themap-reduce operation is processing.
    • For each item, the function associates the sku with a newobject value that contains the count of 1 and theitem qty for the order and emits the sku and value pair.
  1. var mapFunction2 = function() {
  2.                        for (var idx = 0; idx < this.items.length; idx++) {
  3.                            var key = this.items[idx].sku;
  4.                            var value = {
  5.                                          count: 1,
  6.                                          qty: this.items[idx].qty
  7.                                        };
  8.                            emit(key, value);
  9.                        }
  10.                     };

  • Define the corresponding reduce function with two argumentskeySKU and countObjVals:

    • countObjVals is an array whose elements are the objectsmapped to the grouped keySKU values passed by mapfunction to the reducer function.
    • The function reduces the countObjVals array to a singleobject reducedValue that contains the count and theqty fields.
    • In reducedVal, the count field contains the sum of thecount fields from the individual array elements, and theqty field contains the sum of the qty fields from theindividual array elements.
  1. var reduceFunction2 = function(keySKU, countObjVals) {
  2.                      reducedVal = { count: 0, qty: 0 };
  3.  
  4.                      for (var idx = 0; idx < countObjVals.length; idx++) {
  5.                          reducedVal.count += countObjVals[idx].count;
  6.                          reducedVal.qty += countObjVals[idx].qty;
  7.                      }
  8.  
  9.                      return reducedVal;
  10.                   };
  • Define a finalize function with two arguments key andreducedVal. The function modifies the reducedVal objectto add a computed field named avg and returns the modifiedobject:
  1. var finalizeFunction2 = function (key, reducedVal) {
  2.  
  3.                        reducedVal.avg = reducedVal.qty/reducedVal.count;
  4.  
  5.                        return reducedVal;
  6.  
  7.                     };
  • Perform the map-reduce operation on the orders collection usingthe mapFunction2, reduceFunction2, andfinalizeFunction2 functions.
  1. db.orders.mapReduce( mapFunction2,
  2.                      reduceFunction2,
  3.                      {
  4.                        out: { merge: "map_reduce_example" },
  5.                        query: { ord_date:
  6.                                   { $gt: new Date('01/01/2012') }
  7.                               },
  8.                        finalize: finalizeFunction2
  9.                      }
  10.                    )

This operation uses the query field to select only thosedocuments with ord_date greater than newDate(01/01/2012). Then it output the results to a collectionmap_reduce_example. If the map_reduce_example collectionalready exists, the operation will merge the existing contents withthe results of this map-reduce operation.

For more information and examples, see theMap-Reduce page andPerform Incremental Map-Reduce.

Output

The mapReduce command adds support for thebypassDocumentValidation option, which lets you bypassdocument validation wheninserting or updating documents in a collection with validationrules.

If you set the out parameter to write theresults to a collection, the mapReduce command returns adocument in the following form:

  1. {
  2.     "result" : <string or document>,
  3.     "timeMillis" : <int>,
  4.     "counts" : {
  5.         "input" : <int>,
  6.         "emit" : <int>,
  7.         "reduce" : <int>,
  8.         "output" : <int>
  9.     },
  10.     "ok" : <int>,
  11. }

If you set the out parameter to output theresults inline, the mapReduce command returns a documentin the following form:

  1. {
  2.     "results" : [
  3.        {
  4.           "_id" : <key>,
  5.           "value" :<reduced or finalizedValue for key>
  6.        },
  7.        ...
  8.     ],
  9.     "timeMillis" : <int>,
  10.     "counts" : {
  11.        "input" : <int>,
  12.        "emit" : <int>,
  13.        "reduce" : <int>,
  14.        "output" : <int>
  15.     },
  16.     "ok" : <int>
  17. }
  • mapReduce.result

  • For output sent to a collection, this value is either:

    • a string for the collection name if outdid not specify the database name, or
    • a document with both db and collection fields if out specified both a database and collection name.
  • mapReduce.results

  • For output written inline, an array of resulting documents. Eachresulting document contains two fields:

    • _id field contains the key value,
    • value field contains the reduced or finalized value for theassociated key.
  • mapReduce.timeMillis
  • The command execution time in milliseconds.
  • mapReduce.counts
  • Various count statistics from the mapReduce command.
  • mapReduce.counts.input
  • The number of input documents, which is the number of times themapReduce command called the map function.
  • mapReduce.counts.emit
  • The number of times the mapReduce command called theemit function.
  • mapReduce.counts.reduce
  • The number of times the mapReduce command called thereduce function.
  • mapReduce.counts.output
  • The number of output values produced.
  • mapReduce.ok
  • A value of 1 indicates the mapReduce command ransuccessfully. A value of 0 indicates an error.

Additional Information