P4DTI issue job000148

TitleReplicator is slow due to reading auxiliary tables many times
Assigned userGareth Rees
DescriptionThe replicator may read auxiliary tables in the defect tracker (like the TeamTrack USERS table) many times in the course of replicating a single job or issue. This slows replication down and increases the possibility of conflicts.
AnalysisWe have only seen this problem at a beta test site, where it resulted from two conditions:
1. The P4DTI was running on a different machine from the TeamTrack server, thus requiring users, states and so on to be accessed across the network.
2. The internal network was very slow (even Perforce was badly affected).

There are two design problems that underlie this:
1. The configuration generator creates one user_translator for each user field, and another for use in translating changes and fixes. Each user_translator has its own idea of what the user map is. This has the potential for inconsistency. There should be one user map per dt_teamtrack object.
2. Every time it misses a user, it re-reads the USERS table in case the user has been added. But missing a user can happen every time something happens, whereas adding a user is rare.

Better strategies can be applied to both problems:
1. Only create one user_translator object in configure_teamtrack.py and to use that in all places where one is needed (this is safe). Similar remarks apply to the other translators, but user_translator is the one where this problem is most obvious.
2. Don't read the USERS table each time a user isn't found, but less frequently. Once per poll would be good.
3. Even once per poll is too much when there are many users and a slow network connection to the TeamTrack server. Since we tell people to restart the replicator when they add a user, it may be best never to refresh the cache.

See also <http://www.ravenbrook.com/project/p4dt...2001-02-22/version-1.0-support/#job-148>.
How foundmanual_test
Evidence<http://www.ravenbrook.com/project/p4dt...2000-12-08/release-0.4.2-test-report-2/>, item 4.
Observed in0.4.2
Introduced in0.4.0
Created byGareth Rees
Created on2000-12-08 19:31:32
Last modified byGareth Rees
Last modified on2001-12-10 19:13:46
History2000-12-08 GDR Created.
2001-02-27 NB Made description more generic, as this also applies to Bugzilla.


Change Effect Date User Description
13708 closed 2001-06-30 17:22:08 Gareth Rees Added flags which indicate whether TeamTrack data has been read and cached this poll, so that we don't read these tables more than once per poll.
10155 open 2001-03-17 14:21:56 Gareth Rees Formatted configure_teamtrack.py as a document. Added links to design. Make only a single user translator to fix job000148.