Optimistic Concurrency – Where is your merge code?

So next time someone waves off scalability issues by sprinkling magic ‘optimistic concurrency’ pixie dust make sure to ask them who is going to write the merge code. The answer is likely to be “Um… the client?” Yeah, sure.

I believe the author is assuming that OCC is claiming to solve the merge conflict and I never thought it did. In database systems, OCC allows for improved pipelining of transaction requests by being optimistic and assuming that not many transactions involve the same working sets. It is an scalability feature because it is lockless, thus eliminating all of the overhead introduced by lock managers.

If OCC was so bad, I would think that most of the web services and applications would be implementing using locking mechanisms, but that does not seem to be the case. On the other hand, I believe most applications are implementing a half-baked OCC. I mean this because they are lockless but either just explode on conflicts or silently override the information. Therefore, I think that by Google advertising this computer science gem and suggesting a method for using with the Atom protocol is a step in the right direction.

What does OCC as described by Google gives us? In my opinion, it takes Atom Protocol implementors (client and server) closer to a level where they can start thinking about merge procedures but they are not there yet. Let me take this a bit further. Source Control Management protocols like CVS and SVN dictate that clients cannot commit a file change unless they have updated to the most recent version of that file. This is good. However, the Atom syntax is somewhat vague (in my opinion) in regards to versioning. How do we compare two versions? I am sure something could be done with Atom id and updated elements, but that is definitely un(der)-specified in the protocol. If we were to leave things as they are, every client developer would have to write this comparison code and that would take longer to get right. Fortunately, we can make use of the opaque URLs inside the link elements in the Atom entry to help clients provide the server with implementation specific versioning information. Now a server can extract the version from the URL as it pleases and has the option of throwing a conflict error rather effortless. Should we disregard this improvement just because it does not have merge code? I do not think so.

The main reason why I am posting on OCC, is because as Yaron says, implementing OCC is a joke given that you have a half-decent transactional system and we do, DB2. In any case, Queso now implements OCC too. The first benefit is that we can, like Google, begin to create a de-facto protocol for Atom clients to give reliable version information to the server and a best practice for Atom servers to return 409 conflict errors. If anything, the client can always choose to just override and commit their changes, at least Atom entries are quite small in structure, maybe not in content (like this post). But more importantly, once that is established, we can begin to introduce the possibility of introducing merge procedures in an update request. The best example I could come up for this was the scenario of scheduling resources as described in the Managing Update Conflicts in Bayou paper which I think is a great read. I will not go onto describe it here because you’ll stop reading this, but the only hint I will give you is that Queso uses also a SQL-like language for pre-conditions and merge procedures: SPARQL.