Add support for skip history (write last state to sink)

[gkorland]

In case when only the last key state is relevant we should be able to skip updates.

e.g.

HSET person:1 name v1
HSET person:1 name v2
HSET person:1 name v3

Only one call to the DB should sent

UPDATE person SET name = v3 WHERE id=1; 

[K-Jo]

@gkorland could this be generalised as a feature of streams?

[mcazzador]

Hi, is it possible to write only first key occurrence (in final database table) and dont' make a "REPLACE INTO" if a replicated key is inserted in redis?
Thanks

[MeirShpilraien]

@mcazzador Theoretically, it's possible but requires some code changes, if you describe a little more the usecase and why you need it maybe we can come up with a solution.

[mcazzador]

Hi, "REPLACE INTO" is not the best solution for me because i need to flag every DB records inserted on backend. REPLACE INTO in case of duplicate key (I want to protect myself from a possible case of this type without bloking error), erase my previous update fields. Maybe i can change REPLACE INTO with something like that:

INSERT INTO tablename (id, value...)
VALUES('$id', '$value',....)
ON DUPLICATE KEY UPDATE ....

I thought it was already implemented in the code.

[MeirShpilraien]

@mcazzador you can easily change the add query by inheriting the MySqlConnector (for example in case of mysql backend) and override def PrepereQueries(self, mappings):

rgsync/rgsync/Connectors/sql_connectors.py

Line 181 in c16a737

def PrepereQueries(self, mappings):

Then you can use your new connector as it was MySql connector.

Is this helps?

[mcazzador]

Thank's a lot

[mcazzador]

I do that, maybe it could be doing better, thanks

def GetUpdateQuery_noreplace(tableName, mappings, pk):
query = 'INSERT INTO %s' % tableName
values = [val for kk, val in mappings.items() if not kk.startswith('_')]
values = [self.pk] + values
values.sort()
query = '%s(%s) values(%s) ON DUPLICATE KEY UPDATE %s=:%s' % (query, ','.join(values), ','.join([':%s' % a for a in values]),pk,pk)
return query