Warning: Table '.\dcgforum\cache_page' is marked as crashed and should be repaired query: SELECT data, created, headers, expire FROM cache_page WHERE cid = 'http://forums.dataphor.org/index.php?q=node/15' in C:\Sites\DCGForum\includes\database.mysql.inc on line 172
Improving database constraints performance. | Dataphor Forums

Improving database constraints performance.

Hello,

I am currently working on a model that represents 'temporal' trees (nodes/edges + validity from/to dates).

To enforce integrity I use database constraints. And because database constraints are queries that access all the data of a subset of the base tables (and not a subset of the tuples of base tables), I am running into performance problems.

So I tried several solution, but no one is completely satisfying.

1. Instead of having 1 base table per entity, partition the data into 2 base tables OLD and NEW. OLD contains 'validated, old' data and NEW contains 'new' data. DB constraints refer to the NEW table only.

Create a view from the union of these 2 tables. Control the update propagation of that view so that:
- inserts are done on the NEW table.Thus, inserts will trigger the DB constraints on NEW table only.
- Updated/deleted tuples are first deleted from OLD and copied to NEW and then are updated in /deleted from NEW.

After a certain time or after the NEW table is too large, transfer the data from NEW to OLD.
This solution works fine providing that DB constraints do not need to access to OLD table.

2. Precompute the queries (or part of the queries) that appear in the DB constraints. This can be done using 'after' deferred event handlers.
This solution is rather complex because the pre-computation must be done incrementaly (otherwise the gain in performance would dissapear). But DB constraints can still be expressed as non-restricted queries and we have one entity per table.

3. For each base table, create a companion log table that stores, per transaction, all the insert/update/delete on that base table.
Fill these log tables using event handlers.
Express the DB constraint queries in terms of the base tables and log tables, using the tuples in the log table to restrict the queries.
The problem here is to delete the contents of the log tables before a transaction starts. I wonder whether it is possible to do that inside the server (and not from the frontend).

4. Use event handlers instead of db constraints. But in that case, the constraints are no longer declarative, it is much like putting business rules in stored procedures or in the business layer (à la java or C# with all known bad consequences). It looks like a step backwards.
Moreover when a db constraint refers to several tables (as it is often the case), the event handlers of each table must somewhat collaborate to make sure that this constraint is not checked twice.

So far, I got the best results with method 4 but, as I said, this is not really a solution.
I think that method 3 is the closest to a good solution but, to be useable, it would require some support from the server to collect all the data that is modified during a transaction.

Unless there is another solution that I am not aware of...

Thank you for your comments

Regards

J-C

Improving Constraint Performance

Hi J-C,

Transition constraints are what the system uses internally to rewrite reference constraints, and often result in significant performance improvements for equivalent constraints. I agree that the cost is high in terms of development because of the complexity of transforming a database constraint to an equivalent set of transition constraints, but the run-time performance should be better than the equivalent implementation using deferred event handlers.

Regards,
Bryn Rhodes
Alphora

Re: Improving database constraints performance.

Have you tried transition constraints? That's what I use for my temporal data.

Improving database constraints performance

Thank you for your tip.

In fact, transition constraints or event handlers have pretty much the same cost in terms of development.

Anyway, I'd be interested in knowing exactly how you check these constraints.

Do you think we could exchange some code snippets?

Regards

J-C

Code snippet

Here's a simple example---simpler than what you need, but it's what I've got. I believe there are examples in the Dataphor documentation as well. The transition constraint here makes sure that no two rows with the same TeamID have overlapping LocationFrom/LocationTo intervals.


create table FieldTeamLocation
{
LocationID : FieldTeamLocationID { default GetNextGenerator("FieldTeamLocationID") },
KommuneCode : KommuneCode,
LocationName : Name,
TeamID : FieldTeamID,
LocationAddress : LongDescription,
LocationFrom : Date,
LocationTo : Date,
constraint ValidPeriod LocationTo >= LocationFrom
tags { DAE.SimpleMessage = "'To' may not be earlier than 'From'" },
transition constraint NoOverlap
on insert
not exists (FieldTeamLocation where LocationID <> new.LocationID
and TeamID = new.TeamID
and new.LocationFrom <= LocationTo
and new.LocationTo >= LocationFrom)
on update
new.LocationID = old.LocationID and
not exists (FieldTeamLocation where LocationID <> new.LocationID
and TeamID = new.TeamID
and new.LocationFrom <= LocationTo
and new.LocationTo >= LocationFrom)
tags { DAE.SimpleMessage = "Location periods for a particular Field Team may not overlap" },
key { LocationID },
key { LocationName },
key { TeamID, LocationFrom },
key { TeamID, LocationTo }
};


Warning: Table '.\dcgforum\watchdog' is marked as crashed and should be repaired query: INSERT INTO watchdog (uid, type, message, severity, link, location, referer, hostname, timestamp) VALUES (0, 'php', '<em>Table &amp;#039;.\\dcgforum\\cache_page&amp;#039; is marked as crashed and should be repaired\nquery: LOCK TABLES cache_page WRITE</em> in <em>C:\\Sites\\DCGForum\\includes\\database.mysql.inc</em> on line <em>172</em>.', 2, '', 'http://forums.dataphor.org/index.php?q=node/15', '', '54.196.190.32', 1542384611) in C:\Sites\DCGForum\includes\database.mysql.inc on line 172

Warning: Table '.\dcgforum\watchdog' is marked as crashed and should be repaired query: INSERT INTO watchdog (uid, type, message, severity, link, location, referer, hostname, timestamp) VALUES (0, 'php', '<em>Table &amp;#039;.\\dcgforum\\cache_page&amp;#039; is marked as crashed and should be repaired\nquery: UPDATE cache_page SET data = &amp;#039;\\0\\0\\0\\0\\0\\0ZrLaL#{J:ر-KJmvڎǖv2X\\0,ŤC}6o\\&amp;#039;\\0C{1m83ׇ\\&amp;#039;~P\\\\&amp;lt;9&amp;gt;09㣓#N&amp;gt;[Mҙ,qqY5!sn|g4&amp;gt;y-RF\\\\cDE^5޽7Fh7t?Re$ڣB7S2#ศ&amp;gt;ʩHciHt 2+2m\\nY&amp;amp;j$*i#iSv/@B9)C]:UɢRȩsD2*_N$k{yV\\nKj\\&amp;#039;2l̨~O\\\\x*\\rxK_O+xS.d[*`k#Q4wYQB5!)ukD 4 0Nq&amp;amp;Ν χ\\0X[eC\\0.Ei~biC& in C:\Sites\DCGForum\includes\database.mysql.inc on line 172

Warning: Table '.\dcgforum\watchdog' is marked as crashed and should be repaired query: INSERT INTO watchdog (uid, type, message, severity, link, location, referer, hostname, timestamp) VALUES (0, 'php', '<em>Incorrect string value: &amp;#039;\\x8B\\x08\\0\\0...&amp;#039; for column &amp;#039;session&amp;#039; at row 1\nquery: INSERT INTO sessions (sid, uid, cache, hostname, session, timestamp) VALUES (&amp;#039;cf23kvl1m7ip080ol7trf0l9h1&amp;#039;, 0, 0, &amp;#039;54.196.190.32&amp;#039;, &amp;#039;messages|a:1:{s:5:\\&amp;quot;error\\&amp;quot;;a:2:{i:0;s:195:\\&amp;quot;user warning: Table &amp;amp;#039;.\\\\dcgforum\\\\cache_page&amp;amp;#039; is marked as crashed and should be repaired\\nquery: LOCK TABLES cache_page WRITE in C:\\\\Sites\\\\DCGForum\\\\includes\\\\database.mysql.inc on line 172.\\&amp;quot;;i:1;s:4560:\\&amp;quot;user warning: Table &amp;amp;#039;.\\\\dcgforum\\\\cache_page&amp;amp;#039; is mark in C:\Sites\DCGForum\includes\database.mysql.inc on line 172