Use oplog_replay option to optimize queries on oplog#3
Open
poudro wants to merge 2 commits intorick446:masterfrom
poudro:master
Open
Use oplog_replay option to optimize queries on oplog#3poudro wants to merge 2 commits intorick446:masterfrom poudro:master
poudro wants to merge 2 commits intorick446:masterfrom
poudro:master
Conversation
…ce index on capped collections don't work so well...
…ast mark point when mmm crashes
gukoff
suggested changes
Aug 15, 2017
gukoff
left a comment
gukoff
left a comment
There was a problem hiding this comment.
It is straight up wrong in so many ways.
Isn't this enough?
q = self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)| @@ -1,16 +1,16 @@ | |||
| #!/usr/bin/env python | |||
| #!/usr/bin/python | |||
| # for name,m in masters.items(): | ||
| # pprint.pprint(name) | ||
| # pprint.pprint(m._coll) | ||
| # pprint.pprint('masters output') |
There was a problem hiding this comment.
There shouldn't be any commented code. If you don't use it, remove it.
| '''Create a bson.Timestamp. Factored into a method so we can delay | ||
| importing socket until gevent.monkey.patch_all() is called. You could | ||
| also insert another timestamp function to set options if you so desire. | ||
| ''' |
There was a problem hiding this comment.
Here is a clear reason not to remove this method.
| master_uri = self._topology[name_by_id[master['_id']]]['uri'] | ||
| conn = Connection(master_uri) | ||
| coll = conn.local[MMM_DB_NAME] | ||
| for cp in coll.find(dict(_id='master_checkpoint')): |
There was a problem hiding this comment.
Don't use dict for it: {'_id': 'master_checkpoint'}
| if masters[s._topology[name]['id']] == True: | ||
| masters[s._topology[name]['id']] = s | ||
| except: | ||
| True |
| try: | ||
| if masters[s._topology[name]['id']] == True: | ||
| masters[s._topology[name]['id']] = s | ||
| except: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.
Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).
pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.
Please consider applying this patch if pymongo updates the available options in the Cursor class init.
Thx :)