mongo_management

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
mongo_management [2020/12/16 09:35] – [Index] andonovjmongo_management [2020/12/16 09:42] (current) – [Index] andonovj
Line 9: Line 9:
 ===Start=== ===Start===
  
-<sxh bash>+<Code:bash>
 [root@tain-cx-mdb1 scripts]# ./startMongoDB.sh [root@tain-cx-mdb1 scripts]# ./startMongoDB.sh
 about to fork child process, waiting until server is ready for connections. about to fork child process, waiting until server is ready for connections.
 forked process: 67867 forked process: 67867
 child process started successfully, parent exiting child process started successfully, parent exiting
-</sxh>+</Code>
  
 Where the script contains: Where the script contains:
  
 **startMongoDB** **startMongoDB**
-<sxh bash>+<Code:bash>
 #!/bin/bash #!/bin/bash
 sudo /bin/mongod --auth -f /app/mongo/hunterServer/conf/mongo.conf --fork sudo /bin/mongod --auth -f /app/mongo/hunterServer/conf/mongo.conf --fork
-</sxh>+</Code>
  
 and the contents of the configuration file is: and the contents of the configuration file is:
  
 **mongo.conf** **mongo.conf**
-<sxh bash>+<Code:bash>
 port = 9005 port = 9005
 logpath = /var/mongodb/logs/mongodb.log logpath = /var/mongodb/logs/mongodb.log
Line 36: Line 36:
 smallfiles = true smallfiles = true
 maxConns = 16000 maxConns = 16000
-</sxh>+</Code>
  
 The database can also be shutdown as follows: The database can also be shutdown as follows:
Line 42: Line 42:
 ===Stop=== ===Stop===
  
-<sxh bash>+<Code:bash>
 ~/mongodb/bin/mongo localhost:9005/admin --eval "db.shutdownServer()" ~/mongodb/bin/mongo localhost:9005/admin --eval "db.shutdownServer()"
-</sxh>+</Code>
  
 or you can do it manually using or you can do it manually using
  
  
-<sxh bash>+<Code:bash>
 mongo admin -u  'adminDBA' -p 'password' --port 9005 --eval "db.shutdownServer()" mongo admin -u  'adminDBA' -p 'password' --port 9005 --eval "db.shutdownServer()"
-</sxh>+</Code>
  
  
Line 58: Line 58:
  
  
-<sxh bash>+<Code:bash>
 prompt = function() { prompt = function() {
     user = db.runCommand({connectionStatus:1}).authInfo.authenticatedUsers[0]     user = db.runCommand({connectionStatus:1}).authInfo.authenticatedUsers[0]
Line 78: Line 78:
    |         |       |    |    |         |       |    |
 Username IP Address Port Database Username IP Address Port Database
-</sxh>+</Code>
  
  
Line 90: Line 90:
 Server status will give you some rough overview of how the memory is allocated: Server status will give you some rough overview of how the memory is allocated:
  
-<sxh bash>+<Code:bash>
 > db.serverStatus().mem > db.serverStatus().mem
 { "bits" : 64, "resident" : 27, "virtual" : 397, "supported" : true } { "bits" : 64, "resident" : 27, "virtual" : 397, "supported" : true }
Line 119: Line 119:
 Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
 Bytes released to the OS take up virtual address space but no physical memory. Bytes released to the OS take up virtual address space but no physical memory.
-</sxh>+</Code>
  
 So you can see how much memory your database is storing. So you can see how much memory your database is storing.
  
 The storage can be checked as follows: The storage can be checked as follows:
-<sxh bash>+<Code:bash>
 > db.getCollectionNames().map(name => ({totalIndexSize: db.getCollection(name).stats().totalIndexSize, name: name})).sort((a, b) => a.totalIndexSize - b.totalIndexSize).forEach(printjson) > db.getCollectionNames().map(name => ({totalIndexSize: db.getCollection(name).stats().totalIndexSize, name: name})).sort((a, b) => a.totalIndexSize - b.totalIndexSize).forEach(printjson)
  
Line 141: Line 141:
 { "totalIndexSize" : 114831360, "name" : "audit.trails" } { "totalIndexSize" : 114831360, "name" : "audit.trails" }
 { "totalIndexSize" : 368611328, "name" : "rounds" } { "totalIndexSize" : 368611328, "name" : "rounds" }
-</sxh>+</Code>
  
 Of course you can investigate it further with: Of course you can investigate it further with:
  
-<sxh bash>+<Code:bash>
 > db.rounds.stats().indexSizes > db.rounds.stats().indexSizes
 { {
Line 158: Line 158:
 } }
 > >
-</sxh>+</Code>
  
 ===Get Storage Size for each Colleciton=== ===Get Storage Size for each Colleciton===
Line 191: Line 191:
  
  
-<sxh bash>+<Code:bash>
 > db.files.chunks.stats() > db.files.chunks.stats()
 { {
Line 201: Line 201:
         "capped" : false,         "capped" : false,
         "wiredTiger" : {....}         "wiredTiger" : {....}
-</sxh>+</Code>
  
 On our example, this collection "file.chunks" is 148870322967 bytes in data but it occupies 394018263040 on disk level. In other words the data is more than 100% fragmented. On our example, this collection "file.chunks" is 148870322967 bytes in data but it occupies 394018263040 on disk level. In other words the data is more than 100% fragmented.
Line 216: Line 216:
  
 ==Linux Shell== ==Linux Shell==
-<sxh bash>+<Code:bash>
 mongod --repair --repairpath /mnt/vol1 mongod --repair --repairpath /mnt/vol1
-</sxh>+</Code>
  
 ==Mongo Shell== ==Mongo Shell==
-<sxh bash>+<Code:bash>
 db.repairDatabase() db.repairDatabase()
-</sxh>+</Code>
  
 or or
  
-<sxh bash>+<Code:bash>
 db.runCommand({repairDatabase:1}) db.runCommand({repairDatabase:1})
-</sxh>+</Code>
  
 RepairDatabase needs free space equivalent to the data in your database and an additional 2GB more. It can be run either from the system shell or from within the mongo shell. Depending on the amount of data you have, it may be necessary to assign a sperate volume for this using the --repairpath option. RepairDatabase needs free space equivalent to the data in your database and an additional 2GB more. It can be run either from the system shell or from within the mongo shell. Depending on the amount of data you have, it may be necessary to assign a sperate volume for this using the --repairpath option.
Line 237: Line 237:
 he compact command works at the collection level, so each collection in your database will have to be compacted one by one. This completely rewrites the data and indexes to remove fragmentation. In addition, if your storage engine is WiredTiger, the compact command will also release unused disk space back to the system. You're out of luck if your storage engine is the older MMAPv1 though; it will still rewrite the collection, but it will not release the unused disk space. Running the compact command places a block on all other operations at the database level, so you have to plan for some downtime. he compact command works at the collection level, so each collection in your database will have to be compacted one by one. This completely rewrites the data and indexes to remove fragmentation. In addition, if your storage engine is WiredTiger, the compact command will also release unused disk space back to the system. You're out of luck if your storage engine is the older MMAPv1 though; it will still rewrite the collection, but it will not release the unused disk space. Running the compact command places a block on all other operations at the database level, so you have to plan for some downtime.
  
-<sxh bash>+<Code:bash>
 db.runCommand({compact:'collectionName'}) db.runCommand({compact:'collectionName'})
-</sxh>+</Code>
  
 ===Example=== ===Example===
 So let's check it in action with WiredTiger: So let's check it in action with WiredTiger:
  
-<sxh bash>+<Code:bash>
 > db.files.chunks.storageSize() > db.files.chunks.storageSize()
 394018263040 394018263040
Line 252: Line 252:
 162991509504 162991509504
 > >
-</sxh>+</Code>
  
 WoW, we saved more than half of the space :)  WoW, we saved more than half of the space :) 
Line 269: Line 269:
 The export can be done wherever you can, in our case we will create a new directory for that backup: The export can be done wherever you can, in our case we will create a new directory for that backup:
  
-<sxh bash>+<Code:bash>
 [root@localhost mongo]# mkdir -p /app/data/mongo [root@localhost mongo]# mkdir -p /app/data/mongo
 [root@localhost mongo]# cd /app/data [root@localhost mongo]# cd /app/data
Line 285: Line 285:
 drwxr-xr-x. 4 root root 33 Sep 10 15:22 . drwxr-xr-x. 4 root root 33 Sep 10 15:22 .
 drwxr-xr-x. 6 root root 62 Sep 10 15:25 backup <- The backup (export) location drwxr-xr-x. 6 root root 62 Sep 10 15:25 backup <- The backup (export) location
-</sxh>+</Code>
  
 The export is done very easy: The export is done very easy:
  
-<sxh bash> +<Code:bash> 
 [root@localhost backup]# mongodump  --out /app/data/backup -u adminDBA -p password123 --authenticationDatabase admin [root@localhost backup]# mongodump  --out /app/data/backup -u adminDBA -p password123 --authenticationDatabase admin
 2019-09-10T15:25:28.642-0400    writing admin.system.indexes to 2019-09-10T15:25:28.642-0400    writing admin.system.indexes to
Line 303: Line 303:
 2019-09-10T15:25:28.691-0400    done dumping ExampleDB.ExampleCol (2 documents) 2019-09-10T15:25:28.691-0400    done dumping ExampleDB.ExampleCol (2 documents)
 2019-09-10T15:25:28.691-0400    done dumping test.mycol1 (20 documents) 2019-09-10T15:25:28.691-0400    done dumping test.mycol1 (20 documents)
-</sxh>+</Code>
  
 ===Modify the mongo config file=== ===Modify the mongo config file===
 We have to modify the config file so it will point to the new folder and with the correct engine: We have to modify the config file so it will point to the new folder and with the correct engine:
  
-<sxh bash>+<Code:bash>
 # Where and how to store data. # Where and how to store data.
 storage: storage:
Line 315: Line 315:
     enabled: true     enabled: true
   engine: wiredTiger <- The New engine   engine: wiredTiger <- The New engine
-</sxh>+</Code>
  
 P.S. Disable the authentication for now, since it will be problem later: P.S. Disable the authentication for now, since it will be problem later:
  
-<sxh bash>+<Code:bash>
 security: security:
   authorization: "disabled"   authorization: "disabled"
-</sxh>+</Code>
  
 ===Restart the mongod=== ===Restart the mongod===
 Feel free to restart the mongo however you want :) Feel free to restart the mongo however you want :)
  
-<sxh bash>+<Code:bash>
 [root@localhost mongo]# mongo -u adminDBA -p password123 localhost:27017/admin --eval "db.shutdownServer()" [root@localhost mongo]# mongo -u adminDBA -p password123 localhost:27017/admin --eval "db.shutdownServer()"
 MongoDB shell version v4.0.12 MongoDB shell version v4.0.12
Line 381: Line 381:
 > >
 bye bye
-</sxh>+</Code>
  
  
Line 387: Line 387:
 If the authentication isn't disabled, that will create a problem since in the config file the authentication is enabled, but the user (which is in the admin database in our case) cannot be used. So ensure the authentication is disabled: If the authentication isn't disabled, that will create a problem since in the config file the authentication is enabled, but the user (which is in the admin database in our case) cannot be used. So ensure the authentication is disabled:
  
-<sxh bash>+<Code:bash>
 [root@localhost mongo]# mongorestore /app/data/backup/ [root@localhost mongo]# mongorestore /app/data/backup/
 2019-09-10T15:33:08.051-0400    preparing collections to restore from 2019-09-10T15:33:08.051-0400    preparing collections to restore from
Line 400: Line 400:
 2019-09-10T15:33:08.125-0400    restoring users from /app/data/backup/admin/system.users.bson 2019-09-10T15:33:08.125-0400    restoring users from /app/data/backup/admin/system.users.bson
 2019-09-10T15:33:08.185-0400    done 2019-09-10T15:33:08.185-0400    done
-</sxh>+</Code>
  
 ===Verify the data== ===Verify the data==
 We can verify the data, for some reason mongo was showing me 0000, even though there was data in the databases: We can verify the data, for some reason mongo was showing me 0000, even though there was data in the databases:
  
-<sxh bash>+<Code:bash>
 [root@localhost mongo]# mongo [root@localhost mongo]# mongo
 MongoDB shell version v4.0.12 MongoDB shell version v4.0.12
Line 464: Line 464:
 { "_id" : ObjectId("5d00f61c4422539b4d67539d"), "count" : 9, "username" : "user9", "password" : "u2ff0kvs4i", "createdOn" : ISODate("2019-06-12T12:54:52.378Z"), "score" : 0 } { "_id" : ObjectId("5d00f61c4422539b4d67539d"), "count" : 9, "username" : "user9", "password" : "u2ff0kvs4i", "createdOn" : ISODate("2019-06-12T12:54:52.378Z"), "score" : 0 }
 { "_id" : ObjectId("5d00f61c4422539b4d67539e"), "count" : 10, "username" : "user10", "password" : "a7ny85xw29", "createdOn" : ISODate("2019-06-12T12:54:52.379Z"), "score" : 0 } { "_id" : ObjectId("5d00f61c4422539b4d67539e"), "count" : 10, "username" : "user10", "password" : "a7ny85xw29", "createdOn" : ISODate("2019-06-12T12:54:52.379Z"), "score" : 0 }
-</sxh>+</Code>
  
  
Line 477: Line 477:
  
 ====Index==== ====Index====
-Indexes help with searching through mongo, as such there are a lot of indexes, depending what you need:+Indexes support the efficient execution of queries in MongoDB. Without indexes, MongoDB must perform a collection scan, i.e. scan every document in a collection, to select those documents that match the query statement. If an appropriate index exists for a query, MongoDB can use the index to limit the number of documents it must inspect. 
 + 
 +Indexes are special data structures [1] that store a small portion of the collection’s data set in an easy to traverse form. The index stores the value of a specific field or set of fields, ordered by the value of the field. The ordering of the index entries supports efficient equality matches and range-based query operations. In addition, MongoDB can return sorted results by using the ordering in the index. 
 + 
 +The following diagram illustrates a query that selects and orders the matching documents using an index: 
 + 
 +{{:mongoindexoverview.jpg?600|}}
  
 ===Index types=== ===Index types===
Line 540: Line 546:
  
  
-====Practice====+====Management====
 Let's see how to get what indexes a collection has and how to create or re-create and index: Let's see how to get what indexes a collection has and how to create or re-create and index:
  
Line 621: Line 627:
 If you have a JSON file "js" which you would like to execute on the mongo, you can do as follows: If you have a JSON file "js" which you would like to execute on the mongo, you can do as follows:
  
-<sxh bash>+<Code:bash>
 [root@tbp-mts-mdb02 ~]# /usr/bin/mongo -u root -p "password" --authenticationDatabase=admin --host localhost < script.js [root@tbp-mts-mdb02 ~]# /usr/bin/mongo -u root -p "password" --authenticationDatabase=admin --host localhost < script.js
 MongoDB shell version v4.2.2 MongoDB shell version v4.2.2
Line 631: Line 637:
 { "nIndexesWas" : 8, "ok" : 1 } { "nIndexesWas" : 8, "ok" : 1 }
 { "nIndexesWas" : 7, "ok" : 1 } { "nIndexesWas" : 7, "ok" : 1 }
-</sxh>+</Code>
  • mongo_management.1608111332.txt.gz
  • Last modified: 2020/12/16 09:35
  • by andonovj