Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| mongo_management [2020/03/06 08:41] – andonovj | mongo_management [2020/12/16 09:42] (current) – [Index] andonovj | ||
|---|---|---|---|
| Line 9: | Line 9: | ||
| ===Start=== | ===Start=== | ||
| - | <sxh bash> | + | <Code:bash> |
| [root@tain-cx-mdb1 scripts]# ./ | [root@tain-cx-mdb1 scripts]# ./ | ||
| about to fork child process, waiting until server is ready for connections. | about to fork child process, waiting until server is ready for connections. | ||
| forked process: 67867 | forked process: 67867 | ||
| child process started successfully, | child process started successfully, | ||
| - | </sxh> | + | </Code> |
| Where the script contains: | Where the script contains: | ||
| **startMongoDB** | **startMongoDB** | ||
| - | <sxh bash> | + | <Code:bash> |
| #!/bin/bash | #!/bin/bash | ||
| sudo /bin/mongod --auth -f / | sudo /bin/mongod --auth -f / | ||
| - | </sxh> | + | </Code> |
| and the contents of the configuration file is: | and the contents of the configuration file is: | ||
| **mongo.conf** | **mongo.conf** | ||
| - | <sxh bash> | + | <Code:bash> |
| port = 9005 | port = 9005 | ||
| logpath = / | logpath = / | ||
| Line 36: | Line 36: | ||
| smallfiles = true | smallfiles = true | ||
| maxConns = 16000 | maxConns = 16000 | ||
| - | </sxh> | + | </Code> |
| The database can also be shutdown as follows: | The database can also be shutdown as follows: | ||
| Line 42: | Line 42: | ||
| ===Stop=== | ===Stop=== | ||
| - | <sxh bash> | + | <Code:bash> |
| ~/ | ~/ | ||
| - | </sxh> | + | </Code> |
| or you can do it manually using | or you can do it manually using | ||
| - | <sxh bash> | + | <Code:bash> |
| mongo admin -u ' | mongo admin -u ' | ||
| - | </sxh> | + | </Code> |
| Line 58: | Line 58: | ||
| - | <sxh bash> | + | <Code:bash> |
| prompt = function() { | prompt = function() { | ||
| user = db.runCommand({connectionStatus: | user = db.runCommand({connectionStatus: | ||
| Line 78: | Line 78: | ||
| | | ||
| Username IP Address Port Database | Username IP Address Port Database | ||
| - | </sxh> | + | </Code> |
| - | =====Change | + | |
| + | =====Storage===== | ||
| + | |||
| + | The storage and memory can be overviewed as follows: | ||
| + | |||
| + | |||
| + | ===Using Server Status=== | ||
| + | Server status will give you some rough overview of how the memory is allocated: | ||
| + | |||
| + | < | ||
| + | > db.serverStatus().mem | ||
| + | { " | ||
| + | |||
| + | > db.serverStatus().tcmalloc | ||
| + | ... not easy to read! ... | ||
| + | var mem = db.serverStatus().tcmalloc; | ||
| + | |||
| + | mem.tcmalloc.formattedString | ||
| + | > db.serverStatus().tcmalloc.tcmalloc.formattedString | ||
| + | " | ||
| + | MALLOC: | ||
| + | MALLOC: + 949514240 ( 905.5 MiB) Bytes in page heap freelist | ||
| + | MALLOC: + 135775296 ( 129.5 MiB) Bytes in central cache freelist | ||
| + | MALLOC: + 2762912 ( 2.6 MiB) Bytes in transfer cache freelist | ||
| + | MALLOC: + | ||
| + | MALLOC: + | ||
| + | MALLOC: | ||
| + | MALLOC: = | ||
| + | MALLOC: + 147517440 ( 140.7 MiB) Bytes released to OS (aka unmapped) | ||
| + | MALLOC: | ||
| + | MALLOC: = | ||
| + | MALLOC: | ||
| + | MALLOC: | ||
| + | MALLOC: | ||
| + | MALLOC: | ||
| + | ------------------------------------------------ | ||
| + | Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). | ||
| + | Bytes released to the OS take up virtual address space but no physical memory. | ||
| + | </ | ||
| + | |||
| + | So you can see how much memory your database is storing. | ||
| + | |||
| + | The storage can be checked as follows: | ||
| + | < | ||
| + | > db.getCollectionNames().map(name => ({totalIndexSize: | ||
| + | |||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | { " | ||
| + | </ | ||
| + | |||
| + | Of course you can investigate it further with: | ||
| + | |||
| + | < | ||
| + | > db.rounds.stats().indexSizes | ||
| + | { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } | ||
| + | > | ||
| + | </ | ||
| + | |||
| + | ===Get Storage Size for each Colleciton=== | ||
| + | < | ||
| + | var collectionNames = db.getCollectionNames(), | ||
| + | collectionNames.forEach(function (n) { stats.push(db[n].stats()); | ||
| + | for (var c in stats) { | ||
| + | print(stats[c][' | ||
| + | } | ||
| + | </ | ||
| + | |||
| + | < | ||
| + | var collectionNames = db.getCollectionNames() | ||
| + | var col_stats = []; | ||
| + | |||
| + | // Get stats for every collections | ||
| + | collectionNames.forEach(function (n) { | ||
| + | col_stats.push(db.getCollection(n).stats()); | ||
| + | }); | ||
| + | |||
| + | |||
| + | for (var item of col_stats) { | ||
| + | print(`${item[' | ||
| + | (${(item[' | ||
| + | (${(item[' | ||
| + | } | ||
| + | </ | ||
| + | |||
| + | |||
| + | ====Compact Data==== | ||
| + | Data in Mongo Database can easily become fragmented. Especially with frequent additions and removals. Let's check one example: | ||
| + | |||
| + | |||
| + | < | ||
| + | > db.files.chunks.stats() | ||
| + | { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | </ | ||
| + | |||
| + | On our example, this collection " | ||
| + | |||
| + | In order to defremant a collection we have two options, depending on the storage engine: | ||
| + | |||
| + | * MMAPv1 : Repair a whole database or directory location | ||
| + | * WiredTiger: Compact Collection. | ||
| + | |||
| + | Let's see how this is done in each of them: | ||
| + | |||
| + | ===MMAPv1=== | ||
| + | If your storage engine is MMAPv1, this is your way forward. The repairDatabase command is used for checking and repairing errors and inconsistencies in your data. It performs a rewrite of your data, freeing up any unused disk space along with it. Like compact, it will block all other operations on your database. Running repairDatabase can take a lot of time depending on the amount of data in your db, and it will also completely remove any corrupted data it finds. | ||
| + | |||
| + | ==Linux Shell== | ||
| + | < | ||
| + | mongod --repair --repairpath /mnt/vol1 | ||
| + | </ | ||
| + | |||
| + | ==Mongo Shell== | ||
| + | < | ||
| + | db.repairDatabase() | ||
| + | </ | ||
| + | |||
| + | or | ||
| + | |||
| + | < | ||
| + | db.runCommand({repairDatabase: | ||
| + | </ | ||
| + | |||
| + | RepairDatabase needs free space equivalent to the data in your database and an additional 2GB more. It can be run either from the system shell or from within the mongo shell. Depending on the amount of data you have, it may be necessary to assign a sperate volume for this using the --repairpath option. | ||
| + | |||
| + | |||
| + | ===Wired Tiger=== | ||
| + | he compact command works at the collection level, so each collection in your database will have to be compacted one by one. This completely rewrites the data and indexes to remove fragmentation. In addition, if your storage engine is WiredTiger, the compact command will also release unused disk space back to the system. You're out of luck if your storage engine is the older MMAPv1 though; it will still rewrite the collection, but it will not release the unused disk space. Running the compact command places a block on all other operations at the database level, so you have to plan for some downtime. | ||
| + | |||
| + | < | ||
| + | db.runCommand({compact:' | ||
| + | </ | ||
| + | |||
| + | ===Example=== | ||
| + | So let's check it in action with WiredTiger: | ||
| + | |||
| + | < | ||
| + | > db.files.chunks.storageSize() | ||
| + | 394018263040 | ||
| + | > db.runCommand({compact:' | ||
| + | { " | ||
| + | > db.files.chunks.storageSize() | ||
| + | 162991509504 | ||
| + | > | ||
| + | </ | ||
| + | |||
| + | WoW, we saved more than half of the space :) | ||
| + | |||
| + | ====Change storage engine==== | ||
| From 4.2 the: MMAPv1 is depricated, so if you are thinking of upgrading to 4.2, we have to change the storage engine. The engine isn't easy though. | From 4.2 the: MMAPv1 is depricated, so if you are thinking of upgrading to 4.2, we have to change the storage engine. The engine isn't easy though. | ||
| In nutshell we should: | In nutshell we should: | ||
| Line 94: | Line 269: | ||
| The export can be done wherever you can, in our case we will create a new directory for that backup: | The export can be done wherever you can, in our case we will create a new directory for that backup: | ||
| - | <sxh bash> | + | <Code:bash> |
| [root@localhost mongo]# mkdir -p / | [root@localhost mongo]# mkdir -p / | ||
| [root@localhost mongo]# cd /app/data | [root@localhost mongo]# cd /app/data | ||
| Line 110: | Line 285: | ||
| drwxr-xr-x. 4 root root 33 Sep 10 15:22 . | drwxr-xr-x. 4 root root 33 Sep 10 15:22 . | ||
| drwxr-xr-x. 6 root root 62 Sep 10 15:25 backup <- The backup (export) location | drwxr-xr-x. 6 root root 62 Sep 10 15:25 backup <- The backup (export) location | ||
| - | </sxh> | + | </Code> |
| The export is done very easy: | The export is done very easy: | ||
| - | <sxh bash> | + | <Code:bash> |
| [root@localhost backup]# mongodump | [root@localhost backup]# mongodump | ||
| 2019-09-10T15: | 2019-09-10T15: | ||
| Line 128: | Line 303: | ||
| 2019-09-10T15: | 2019-09-10T15: | ||
| 2019-09-10T15: | 2019-09-10T15: | ||
| - | </sxh> | + | </Code> |
| ===Modify the mongo config file=== | ===Modify the mongo config file=== | ||
| We have to modify the config file so it will point to the new folder and with the correct engine: | We have to modify the config file so it will point to the new folder and with the correct engine: | ||
| - | <sxh bash> | + | <Code:bash> |
| # Where and how to store data. | # Where and how to store data. | ||
| storage: | storage: | ||
| Line 140: | Line 315: | ||
| enabled: true | enabled: true | ||
| engine: wiredTiger <- The New engine | engine: wiredTiger <- The New engine | ||
| - | </sxh> | + | </Code> |
| P.S. Disable the authentication for now, since it will be problem later: | P.S. Disable the authentication for now, since it will be problem later: | ||
| - | <sxh bash> | + | <Code:bash> |
| security: | security: | ||
| authorization: | authorization: | ||
| - | </sxh> | + | </Code> |
| ===Restart the mongod=== | ===Restart the mongod=== | ||
| Feel free to restart the mongo however you want :) | Feel free to restart the mongo however you want :) | ||
| - | <sxh bash> | + | <Code:bash> |
| [root@localhost mongo]# mongo -u adminDBA -p password123 localhost: | [root@localhost mongo]# mongo -u adminDBA -p password123 localhost: | ||
| MongoDB shell version v4.0.12 | MongoDB shell version v4.0.12 | ||
| Line 206: | Line 381: | ||
| > | > | ||
| bye | bye | ||
| - | </sxh> | + | </Code> |
| Line 212: | Line 387: | ||
| If the authentication isn't disabled, that will create a problem since in the config file the authentication is enabled, but the user (which is in the admin database in our case) cannot be used. So ensure the authentication is disabled: | If the authentication isn't disabled, that will create a problem since in the config file the authentication is enabled, but the user (which is in the admin database in our case) cannot be used. So ensure the authentication is disabled: | ||
| - | <sxh bash> | + | <Code:bash> |
| [root@localhost mongo]# mongorestore / | [root@localhost mongo]# mongorestore / | ||
| 2019-09-10T15: | 2019-09-10T15: | ||
| Line 225: | Line 400: | ||
| 2019-09-10T15: | 2019-09-10T15: | ||
| 2019-09-10T15: | 2019-09-10T15: | ||
| - | </sxh> | + | </Code> |
| ===Verify the data== | ===Verify the data== | ||
| We can verify the data, for some reason mongo was showing me 0000, even though there was data in the databases: | We can verify the data, for some reason mongo was showing me 0000, even though there was data in the databases: | ||
| - | <sxh bash> | + | <Code:bash> |
| [root@localhost mongo]# mongo | [root@localhost mongo]# mongo | ||
| MongoDB shell version v4.0.12 | MongoDB shell version v4.0.12 | ||
| Line 289: | Line 464: | ||
| { " | { " | ||
| { " | { " | ||
| - | </sxh> | + | </Code> |
| - | =====Compact Data===== | + | =====Logical Structure===== |
| - | Data in Mongo Database can easily become fragmented. Especially with frequent additions and removals. Let's check one example: | + | As any database, mongo has objects to store information into, we already discussed them: |
| + | ====Data Objects==== | ||
| + | * Collections | ||
| + | * Documents | ||
| - | <sxh bash> | + | As you know, a collection can store many documents, which doesn' |
| - | > db.files.chunks.stats() | + | |
| - | { | + | |
| - | " | + | |
| - | " | + | |
| - | " | + | |
| - | " | + | |
| - | " | + | |
| - | " | + | |
| - | " | + | |
| - | </ | + | |
| - | On our example, this collection | + | ====Index==== |
| + | Indexes support the efficient execution of queries in MongoDB. Without indexes, MongoDB must perform a collection | ||
| - | In order to defremant | + | Indexes are special data structures [1] that store a small portion of the collection’s data set in an easy to traverse form. The index stores the value of a specific field or set of fields, ordered by the value of the field. The ordering of the index entries supports efficient equality matches and range-based query operations. In addition, MongoDB can return sorted results by using the ordering in the index. |
| - | * MMAPv1 : Repair | + | The following diagram illustrates |
| - | * WiredTiger: Compact Collection. | + | |
| - | Let's see how this is done in each of them: | + | {{:mongoindexoverview.jpg? |
| - | ===MMAPv1=== | + | ===Index types=== |
| - | If your storage engine is MMAPv1, this is your way forward. The repairDatabase command is used for checking and repairing errors and inconsistencies in your data. It performs a rewrite of your data, freeing up any unused disk space along with it. Like compact, it will block all other operations on your database. Running repairDatabase can take a lot of time depending on the amount of data in your db, and it will also completely remove any corrupted data it finds. | + | |
| - | ==Linux Shell== | + | ==Single Field== |
| - | <sxh bash> | + | In addition to the MongoDB-defined _id index, MongoDB supports the creation of user-defined ascending/descending indexes on a single field of a document. |
| - | mongod | + | |
| - | </sxh> | + | |
| - | ==Mongo Shell== | + | {{: |
| - | <sxh bash> | + | |
| - | db.repairDatabase() | + | |
| - | </ | + | |
| - | or | + | ==Compound Index== |
| + | MongoDB also supports user-defined indexes on multiple fields, i.e. compound indexes. | ||
| - | <sxh bash> | + | The order of fields listed in a compound index has significance. For instance, if a compound index consists of { userid: 1, score: -1}, the index sorts first by userid and then, within each userid value, sorts by score. |
| - | db.runCommand({repairDatabase:1}) | + | For compound indexes and sort operations, the sort order (i.e. ascending or descending) of the index keys can determine whether the index can support a sort operation. See Sort Order for more information on the impact of index order on results in compound indexes. |
| - | </ | + | |
| - | RepairDatabase needs free space equivalent to the data in your database and an additional 2GB more. It can be run either from the system shell or from within the mongo shell. Depending on the amount of data you have, it may be necessary to assign a sperate volume for this using the --repairpath option. | + | {{: |
| - | ===Wired Tiger=== | + | ==Multikey Index== |
| - | he compact command works at the collection level, so each collection | + | MongoDB uses multikey indexes to index the content stored |
| - | <sxh bash> | + | {{:mongomultikeyindex.jpg? |
| - | db.runCommand({compact:' | + | |
| - | </ | + | |
| - | ===Example=== | + | ==Geospatial Index== |
| - | So let's check it in action with WiredTiger: | + | To support efficient queries of geospatial coordinate data, MongoDB provides two special indexes: 2d indexes that uses planar geometry when returning results and 2dsphere indexes that use spherical geometry to return results. |
| - | <sxh bash> | ||
| - | > db.files.chunks.storageSize() | ||
| - | 394018263040 | ||
| - | > db.runCommand({compact:' | ||
| - | { " | ||
| - | > db.files.chunks.storageSize() | ||
| - | 162991509504 | ||
| - | > | ||
| - | </ | ||
| - | WoW, we saved more than half of the space :) | + | ==Text Indexes== |
| - | =====Storage & Memory Overview===== | + | MongoDB provides a text index type that supports searching for string content in a collection. These text indexes do not store language-specific stop words (e.g. “the”, “a”, “or”) |
| - | The storage | + | |
| - | ===Using Server Status=== | + | ==Hashed Indexes== |
| - | Server status will give you some rough overview | + | To support hash based sharding, MongoDB provides a hashed index type, which indexes the hash of the value of a field. These indexes have a more random distribution of values along their range, but only support equality matches and cannot support range-based queries. |
| - | <sxh bash> | + | ===Index Properties=== |
| - | > db.serverStatus().mem | + | |
| - | { " | + | |
| - | > db.serverStatus().tcmalloc | + | ==Unique Indexes== |
| - | ... not easy to read! ... | + | The unique property for an index causes MongoDB |
| - | var mem = db.serverStatus().tcmalloc; | + | |
| - | mem.tcmalloc.formattedString | + | ==Partial Indexes== |
| - | > db.serverStatus().tcmalloc.tcmalloc.formattedString | + | New in version 3.2. |
| - | " | + | |
| - | MALLOC: | + | |
| - | MALLOC: + 949514240 ( 905.5 MiB) Bytes in page heap freelist | + | |
| - | MALLOC: + 135775296 ( 129.5 MiB) Bytes in central cache freelist | + | |
| - | MALLOC: + 2762912 ( 2.6 MiB) Bytes in transfer cache freelist | + | |
| - | MALLOC: + | + | |
| - | MALLOC: + | + | |
| - | MALLOC: | + | |
| - | MALLOC: | + | |
| - | MALLOC: + 147517440 ( 140.7 MiB) Bytes released to OS (aka unmapped) | + | |
| - | MALLOC: | + | |
| - | MALLOC: | + | |
| - | MALLOC: | + | |
| - | MALLOC: | + | |
| - | MALLOC: | + | |
| - | MALLOC: | + | |
| - | ------------------------------------------------ | + | |
| - | Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). | + | |
| - | Bytes released to the OS take up virtual address space but no physical memory. | + | |
| - | </ | + | |
| - | So you can see how much memory your database is storing. | + | Partial indexes only index the documents in a collection that meet a specified filter expression. By indexing a subset of the documents in a collection, partial indexes have lower storage requirements and reduced performance costs for index creation and maintenance. |
| - | The storage can be checked as follows: | ||
| - | <sxh bash> | ||
| - | > db.getCollectionNames().map(name => ({totalIndexSize: | ||
| - | { " | + | ==Sparse Indexes== |
| - | { " | + | The sparse property of an index ensures that the index only contain entries for documents that have the indexed field. The index skips documents that do not have the indexed field. |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | { " | + | |
| - | </ | + | |
| - | Of course you can investigate it further | + | You can combine the sparse index option |
| + | |||
| + | ==TTL Indexes== | ||
| + | TTL indexes are special indexes that MongoDB can use to automatically remove documents from a collection after a certain amount of time. This is ideal for certain types of information like machine generated event data, logs, and session information that only need to persist in a database for a finite amount of time. | ||
| - | <sxh bash> | + | ==Hidden Indexes== |
| - | > db.rounds.stats().indexSizes | + | New in version 4.4. |
| + | |||
| + | Hidden indexes are not visible to the query planner and cannot be used to support a query. | ||
| + | |||
| + | By hiding an index from the planner, users can evaluate the potential impact of dropping an index without actually dropping the index. If the impact is negative, the user can unhide the index instead of having to recreate a dropped index. And because indexes are fully maintained while hidden, the indexes are immediately available for use once unhidden. | ||
| + | |||
| + | Except for the _id index, you can hide any indexes. | ||
| + | |||
| + | |||
| + | ====Management==== | ||
| + | Let's see how to get what indexes a collection has and how to create or re-create and index: | ||
| + | |||
| + | ===Get Indexes=== | ||
| + | To get the indexes for a certain collection we can use the following function in any collection: | ||
| + | |||
| + | <Code:bash|Get Indexes> | ||
| + | > db.files.files.getIndexes() | ||
| + | [ | ||
| + | { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | }, | ||
| + | " | ||
| + | }, | ||
| + | { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | }, | ||
| + | " | ||
| + | } | ||
| + | ] | ||
| + | > | ||
| + | </ | ||
| + | |||
| + | ===Create an Index=== | ||
| + | To create index we can use: | ||
| + | |||
| + | < | ||
| + | db.files.chunks.createIndex( | ||
| + | { | ||
| + | " | ||
| + | }, | ||
| + | { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } | ||
| + | } | ||
| + | ) | ||
| + | </ | ||
| + | |||
| + | ===Reindex=== | ||
| + | Reindex will re-create all indexes on a collection (after mongoimport for example) | ||
| + | |||
| + | < | ||
| + | > db.files.files.reIndex() | ||
| { | { | ||
| - | "_id_" : 64692224, | + | "nIndexesWas" : 2, |
| - | "rounds_unique" : 58056704, | + | "nIndexes" : 2, |
| - | "rounds_sort_dealer_name" : 18669568, | + | "indexes" : [ |
| - | "rounds_sort_is_closed" : 17887232, | + | { |
| - | "rounds_sort_video_id" : 18415616, | + | " |
| - | "rounds_sort_started_at" : 55402496, | + | "key" : { |
| - | "rounds_text_game_code" : 113971200, | + | " |
| - | "rounds_sort_is_cancelled" : 21516288 | + | }, |
| + | "name" : " | ||
| + | }, | ||
| + | { | ||
| + | | ||
| + | "unique" : true, | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | }, | ||
| + | " | ||
| + | } | ||
| + | ], | ||
| + | "ok" : 1 | ||
| } | } | ||
| > | > | ||
| - | </sxh> | + | </Code> |
| =====Execute a script===== | =====Execute a script===== | ||
| If you have a JSON file " | If you have a JSON file " | ||
| - | <sxh bash> | + | <Code:bash> |
| [root@tbp-mts-mdb02 ~]# / | [root@tbp-mts-mdb02 ~]# / | ||
| MongoDB shell version v4.2.2 | MongoDB shell version v4.2.2 | ||
| Line 451: | Line 637: | ||
| { " | { " | ||
| { " | { " | ||
| - | </sxh> | + | </Code> |