Code RemiForest40 gets you a 25% discount off the list price

Back to Basics 2016 : Webinar 2

Votre première application MongoDB

Rémi ForestSenior Solution Architect

MongoDB#back2basics

V1.0

4

Résumé de la partie 1

• Pourquoi NoSQL existe• Les différents types de bases NoSQL• Principales caractéristiques de MongoDB• Durabilité des données avec MongoDB – Replica Sets• Scalabilité avec MongoDB - Sharding

5

Agenda

• Concepts et terminologie• Installation de MongoDB• Une application basique de blogging• Ajouter un index• Optimisation des requêtes avec Explain()

6

Concepts et terminologie

Relational MongoDBDatabase DatabaseTable CollectionRow DocumentIndex IndexJoin LookupForeign Key ReferenceMulti-table transaction Single document transaction

7

Installation de MongoDB

$ curl -O https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-3.2.6.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 60.9M 100 60.9M 0 0 2730k 0 0:00:22 0:00:22 --:--:-- 1589k$ tar xzvf mongodb-osx-x86_64-3.2.6.tgz x mongodb-osx-x86_64-3.2.6/READMEx mongodb-osx-x86_64-3.2.6/THIRD-PARTY-NOTICESx mongodb-osx-x86_64-3.2.6/MPL-2x mongodb-osx-x86_64-3.2.6/GNU-AGPL-3.0x mongodb-osx-x86_64-3.2.6/bin/mongodumpx mongodb-osx-x86_64-3.2.6/bin/mongorestorex mongodb-osx-x86_64-3.2.6/bin/mongoexportx mongodb-osx-x86_64-3.2.6/bin/mongoimportx mongodb-osx-x86_64-3.2.6/bin/mongostatx mongodb-osx-x86_64-3.2.6/bin/mongotopx mongodb-osx-x86_64-3.2.6/bin/bsondumpx mongodb-osx-x86_64-3.2.6/bin/mongofilesx mongodb-osx-x86_64-3.2.6/bin/mongooplogx mongodb-osx-x86_64-3.2.6/bin/mongoperfx mongodb-osx-x86_64-3.2.6/bin/mongosniffx mongodb-osx-x86_64-3.2.6/bin/mongodx mongodb-osx-x86_64-3.2.6/bin/mongosx mongodb-osx-x86_64-3.2.6/bin/mongo$ ln -s mongodb-osx-x86_64-3.2.6 mongodb

8

Lancer MongodJD10Gen:mongodb jdrumgoole$ ./bin/mongod --dbpath /data/b2b2016-05-23T19:21:07.767+0100 I CONTROL [initandlisten] MongoDB starting : pid=49209 port=27017 dbpath=/data/b2b 64-bit host=JD10Gen.local2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] db version v3.2.62016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b252016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] allocator: system2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] modules: none2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] build environment:2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] distarch: x86_642016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] target_arch: x86_642016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] options: { storage: { dbPath: "/data/b2b" } }2016-05-23T19:21:07.769+0100 I - [initandlisten] Detected data files in /data/b2b created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.2016-05-23T19:21:07.769+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),2016-05-23T19:21:08.837+0100 I CONTROL [initandlisten] 2016-05-23T19:21:08.838+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 10002016-05-23T19:21:08.840+0100 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker2016-05-23T19:21:08.840+0100 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/b2b/diagnostic.data'2016-05-23T19:21:08.841+0100 I NETWORK [initandlisten] waiting for connections on port 270172016-05-23T19:21:09.148+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:59213 #1 (1 connection now open)

9

Connection au shell$ ./bin/mongoMongoDB shell version: 3.2.6connecting to: testServer has startup warnings: 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000>

10

Insérer son premier enregistrement> show databaseslocal 0.000GB> use testswitched to db test> show databaseslocal 0.000GB> db.demo.insert( { "key" : "value" } )WriteResult({ "nInserted" : 1 })> show databaseslocal 0.000GBtest 0.000GB> show collectionsdemo> db.demo.findOne(){ "_id" : ObjectId("573af7085ee4be80385332a6"), "key" : "value" }>

11

Object ID

573af7085ee4be80385332a6TS------ID----PID-Count-

12

Une application basique

• Créons une application de blogging qui contient:– Articles– Users– Commentaires

13

Typical Entity Relation Diagram

14

Représentation dans MongoDB> use blogswitched to db blog> db.users.insert( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" } )WriteResult({ "nInserted" : 1 })> db.users.findOne(){

"_id" : ObjectId("573afff65ee4be80385332a7"),"username" : "jdrumgoole","password" : "top secret","lang" : "EN"

}

15

Comment coder ça ?'''Created on 17 May 2016

@author: jdrumgoole'''import pymongo ## client defaults to localhost and port 27017. eg MongoClient('localhost', 27017)client = pymongo.MongoClient()blogDatabase = client[ "blog" ]usersCollection = blogDatabase[ "users" ]

usersCollection.insert_one( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" })

user = usersCollection.find_one()

print( user )

16

Next up Articles

…articlesCollection = blogDatabase[ "articles" ]

author = "jdrumgoole"

article = { "title" : "This is my first post", "body" : "The is the longer body text for my blog post. We can add lots of text here.", "author" : author, "tags" : [ "joe", "general", "Ireland", "admin" ] }

## Lets check if our author exists#

if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( article )else: raise ValueError( "Author %s does not exist" % author )

17

Création d’un nouveau type d’article## Lets add a new type of article with a posting date and a section#author = "jdrumgoole"title = "This is a post on MongoDB" newPost = { "title" : title, "body" : "MongoDB is the worlds most popular NoSQL database. It is a document database", "author" : author, "tags" : [ "joe", "mongodb", "Ireland" ], "section" : "technology", "postDate" : datetime.datetime.now(), }

## Lets check if our author exists#

if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( newPost )

18

Création d’articles - 1import pymongoimport stringimport datetimeimport random def randomString( size, letters = string.letters ): return "".join( [random.choice( letters ) for _ in xrange( size )] )

client = pymongo.MongoClient()

def makeArticle( count, author, timestamp ): return { "_id" : count, "title" : randomString( 20 ), "body" : randomString( 80 ), "author" : author, "postdate" : timestamp }

def makeUser( username ): return { "username" : username, "password" : randomString( 10 ) , "karma" : random.randint( 0, 500 ), "lang" : "EN" }

19

Création d’articles - 2blogDatabase = client[ "blog" ]usersCollection = blogDatabase[ "users" ]articlesCollection = blogDatabase[ "articles" ]

bulkUsers = usersCollection.initialize_ordered_bulk_op()bulkArticles = articlesCollection.initialize_ordered_bulk_op()

ts = datetime.datetime.now()

for i in range( 1000000 ) : #username = randomString( 10, string.ascii_uppercase ) + "_" + str( i ) username = "USER_" + str( i ) bulkUsers.insert( makeUser( username ) ) ts = ts + datetime.timedelta( seconds = 1 ) bulkArticles.insert( makeArticle( i, username, ts )) if ( i % 500 == 0 ) : bulkUsers.execute() bulkArticles.execute() bulkUsers = usersCollection.initialize_ordered_bulk_op() bulkArticles = articlesCollection.initialize_ordered_bulk_op() bulkUsers.execute()bulkArticles.execute()

20

Trouver un User> db.users.findOne(){

"_id" : ObjectId("5742da5bb26a88bc00e941ac"),"username" : "FLFZQLSRWZ_0","lang" : "EN","password" : "vTlILbGWLt","karma" : 448

}> db.users.find( { "username" : "VHXDAUUFJW_45" } ).pretty(){

"_id" : ObjectId("5742da5bb26a88bc00e94206"),"username" : "VHXDAUUFJW_45","lang" : "EN","password" : "GmRLnCeKVp","karma" : 284

}

21

Trouver les Users avec un Karma élevé> db.users.find( { "karma" : { $gte : 450 }} ).pretty(){

"_id" : ObjectId("5742da5bb26a88bc00e941ae"),"username" : "JALLFRKBWD_1","lang" : "EN","password" : "bCSKSKvUeb","karma" : 487

}{

"_id" : ObjectId("5742da5bb26a88bc00e941e4"),"username" : "OTKWJJBNBU_28","lang" : "EN","password" : "HAWpiATCBN","karma" : 473

}{

…

22

Utiliser la projection > db.users.find( { "karma" : { $gte : 450 }}, { "_id" : 0, username : 1, karma : 1 } ){ "username" : "JALLFRKBWD_1", "karma" : 487 }{ "username" : "OTKWJJBNBU_28", "karma" : 473 }{ "username" : "RVVHLKTWHU_31", "karma" : 493 }{ "username" : "JBNESEOOEP_48", "karma" : 464 }{ "username" : "VSTBDZLKQQ_51", "karma" : 487 }{ "username" : "UKYDTQJCLO_61", "karma" : 493 }{ "username" : "HZFZZMZHYB_106", "karma" : 493 }{ "username" : "AAYLPJJNHO_113", "karma" : 455 }{ "username" : "CXZZMHLBXE_128", "karma" : 460 }{ "username" : "KKJXBACBVN_134", "karma" : 460 }{ "username" : "PTNTIBGAJV_165", "karma" : 461 }{ "username" : "PVLCQJIGDY_169", "karma" : 463 }

23

Ajouter des commentaires - 1> db.articles.find( { "_id" : 19 } ).pretty(){

"_id" : 19,"body" :

"nTzOofOcnHKkJxpjKAyqTTnKZMFzzkWFeXtBRuEKsctuGBgWIrEBrYdvFIVHJWaXLUTVUXblOZZgUqWu",

"postdate" : ISODate("2016-05-23T12:02:46.830Z"),"author" : "ASWTOMMABN_19","title" : "CPMaqHtAdRwLXhlUvsej"

} > db.articles.update( { _id : 18 }, { $set : { comments : [] }} )WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

24

Ajouter des commentaires - 2> db.articles.find( { _id :18 } ).pretty(){

"_id" : 18,"body" :

"KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgXUV",

"postdate" : ISODate("2016-05-23T16:04:39.497Z"),"author" : "USER_18","title" : "wTLreIEyPfovEkBhJZZe","comments" : [ ]

}>

25

Ajouter des commentaires - 3> db.articles.update( { _id : 18 }, { $push : { comments : { username : "joe", comment : "hey first post" }}} )WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

> db.articles.find( { _id :18 } ).pretty(){

"_id" : 18,"body" : "KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgXUV","postdate" : ISODate("2016-05-23T16:04:39.497Z"),"author" : "USER_18","title" : "wTLreIEyPfovEkBhJZZe","comments" : [{"username" : "joe","comment" : "hey first post"}]

}>

26

Supprimer un Article

> db.articles.remove( { "_id" : 25 } )WriteResult({ "nRemoved" : 1 })> db.articles.remove( { "_id" : 25 } )WriteResult({ "nRemoved" : 0 })> db.articles.remove( { "_id" : { $lte : 5 }} )WriteResult({ "nRemoved" : 6 })

• Les supressions laissent des trous• Supprimer une collection est plus efficace que de supprimer

l’ensemble de ses éléments un par un.

27

A Quick Look at Users and Articles Again> db.users.findOne(){

"_id" : ObjectId("57431c07b26a88bf060e10cb"),"username" : "USER_0","lang" : "EN","password" : "kGIxPxqKGJ","karma" : 266

}> db.articles.findOne(){

"_id" : 0,"body" :

"hvJLnrrfZQurmtjPfUWbMhaQWbNjXLzjpuGLZjsxHXbUycmJVZTeOZesTnZtojThrebRcUoiYwivjpwG","postdate" : ISODate("2016-05-23T16:04:39.246Z"),"author" : "USER_0","title" : "gpNIoPxpfTAxWjzAVoTJ"

}>

28

Trouver un User> db.users.find( { "username" : "ABOXHWKBYS_199" } ).explain(){

"queryPlanner" : {"plannerVersion" : 1,"namespace" : "blog.users","indexFilterSet" : false,"parsedQuery" : {

"username" : {"$eq" : "ABOXHWKBYS_199"

}},"winningPlan" : {

"stage" : "COLLSCAN","filter" : {

"username" : {"$eq" : "ABOXHWKBYS_199"

}},"direction" : "forward"

},"rejectedPlans" : [ ]

},"serverInfo" : {

"host" : "JD10Gen.local","port" : 27017,"version" : "3.2.6","gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25"

},"ok" : 1

}

29

Trouver un User – Execution Stats> db.users.find( {"username" : "USER_999999" } ).explain( "executionStats" ).executionStats{

"executionSuccess" : true,"nReturned" : 1,"executionTimeMillis" : 433,"totalKeysExamined" : 0,"totalDocsExamined" : 1000000,"executionStages" : {

"stage" : "COLLSCAN","filter" : {

"username" : {"$eq" : "USER_999999"

}},"nReturned" : 1,"executionTimeMillisEstimate" : 330,"works" : 1000002,"advanced" : 1,"needTime" : 1000000,"needYield" : 0,"saveState" : 7812,"restoreState" : 7812,"isEOF" : 1,"invalidates" : 0,"direction" : "forward","docsExamined" : 1000000

30

Il faut un Index !

> db.users.createIndex( { username : 1 } ){

"createdCollectionAutomatically" : false,"numIndexesBefore" : 1,"numIndexesAfter" : 2,"ok" : 1

}>

31

Indexes Overview

• Parameters– Background : Create an index in the background as opposed to locking the database– Unique : All keys in the collection must be unique. Duplicate key insertions will be

rejected with an error.– Name : explicitly name an index. Otherwise the index name is autogenerated from the

index field.• Deleting an Index

– db.users.dropIndex({ “username” : 1 })• Get All the Indexes on a collection

– db.users.getIndexes()

32

Query Plan Execution Stages

• COLLSCAN : scan de toute la collection• IXSCAN : scan d’un indexe• FETCH : récupération de documents• SHARD_MERGE : fusionne les

33

Ajouter un Index> db.users.find( {"username" : "USER_999999”} ).explain("executionStats”).executionStats{

"executionSuccess" : true,"nReturned" : 1,"executionTimeMillis" : 0,"totalKeysExamined" : 1,"totalDocsExamined" : 1,

…

34

Execution Stage"executionStages" : {

"stage" : "FETCH","nReturned" : 1,"executionTimeMillisEstimate" : 0,"docsExamined" : 1,,"inputStage" : {

"stage" : "IXSCAN","nReturned" : 1,"executionTimeMillisEstimate" : 0,"keyPattern" : {

"username" : 1},"indexName" : "username_1","isMultiKey" : false,"isUnique" : false,"isSparse" : false,"isPartial" : false,"indexVersion" : 1,"direction" : "forward","indexBounds" : {

"username" : ["[\"USER_999999\", \"USER_999999\"]"

]},"keysExamined" : 1,"seenInvalidated" : 0

}}

}

35

Drivers et Frameworks

Morphia

MEAN Stack

36

Ce que nous avons appris

• Comment créer une database et une collection• Comment insérer du contenu dans cette collection• Comment requêter une collection• Comment mettre à jour un document• Comment supprimer un document• Comment évaluer la performance d’une requête• Comment ajouter un index• Comment vérifier si un index est utilisé lors d’une requête

37

Prochain Webinar : Penser “Document”

• Plutôt que de normaliser, il faut considérer une approche hybride de schéma pour proposer une relation cohérente entre les objets dans l’application et dans la base de données.

• Nous optimiserons ensuite ce schéma en fonction des patterns de requêtes que nous nous attendons à trouver.

• Enfin, nous verrons comment les schémas dynamiques et la validation des schémas permettent de faire évoluer le schéma de façon contrôlée .

23 Juin 2016, 14:00 CET.

Q&A