PERFORCE DEFECT TRACKING INTEGRATION RELEASE NOTES FOR RELEASE 1.4.2 Nick Barnes, Ravenbrook Limited $Date: 2002/04/19 $ CONTENTS 1. Introduction 2. Supported configurations 3. Getting support 4. Project contacts 5. What's new 5.1. What's new in release 1.4.2 5.2. What was new in release 1.4.1 5.3. What was new in release 1.4.0 5.4. What was new in release 1.3.2 5.5. What was new in release 1.3.1 5.6. What was new in release 1.3.0 5.7. What was new in releases up to 1.2.1 A. References B. Document history C. Copyright and licence 1. INTRODUCTION These are the release notes for release 1.4.2 of the Perforce Defect Tracking Integration (P4DTI). The P4DTI connects your defect tracking system to Perforce, so that you don't have to switch between them and enter duplicate information about your work. It also links changes made in Perforce with defect tracker issues, making it easy to find out why a change was made, find the work that was done to resolve an issue, or generate reports relating issues to files or codelines. For instructions on installing the P4DTI, see the product readme (readme.txt). For up-to-date information about releases of the P4DTI, see the product information page . From there you will find links to the latest releases, including reports of defects found. If you want to adapt or extend the P4DTI, please go to the product information page and download the Integration Kit. It contains full source code and documentation to help you. The readership of this document is anyone who wants to download and use the Perforce Defect Tracking Integration. This document is not confidential. 2. SUPPORTED CONFIGURATIONS Perforce versions: - Perforce 2000.2, 2001.1, or 2002.1, on any platform; The TeamTrack integration supports: - TeamTrack 4.5, 5.0, or 5.5 on Windows NT 4 or Windows 2000; - Python 2.0 on Windows NT 4 or Windows 2000; - Running the P4DTI replicator on Windows NT 4 or Windows 2000. The P4DTI does not support a TeamTrack server running on a secure web server, nor does it support the "TeamTrack Online" hosted defect tracking service from TeamShare. The Bugzilla integration supports: - Bugzilla 2.10, 2.12, 2.14, or 2.14.1, on Red Hat Linux 6.2 or Solaris; - MySQL 3.22.19 or later; - Python 1.5.2 or later; - Running the P4DTI replicator on any Unix. Note that although the Bugzilla integration runs on a variety of Unix operating systems, we've only tested it on Red Hat Linux 6.2. Also note that the P4DTI will not be able to run Bugzilla's "processmail" script unless it is running on the same machine as Bugzilla itself. 3. GETTING SUPPORT For problems relating to Perforce or the P4DTI in general, contact Perforce Support by writing to or see the technical support page at for contact information. For problems relating to TeamShare, contact TeamShare Technical Support by writing to or see the support page at for contact information. Bugzilla is not supported by any one person or organization. Consult the documentation that came with Bugzilla, or visit . 4. PROJECT CONTACTS You may want to join the p4dti-discussion mailing list. The goals of the list are: 1. to provide feedback to the project on requirements, design, implementation, etc.; 2. to allow people to exchange information and experience with using and adapting the project; 3. to keep people informed about project progress. To join, send a message with the word "subscribe" in the _body_ to or send the word "help" for general information. Please note that the mailing list will be archived and the archive may be published. 5. WHAT'S NEW This section lists defects that have been fixed. 5.1. WHAT'S NEW IN RELEASE 1.4.2 ESSENTIAL job000512: Bugzilla P4DTI doesn't work with Python 2.2 The Bugzilla P4DTI inadvertently uses a feature of Python that doesn't work in Python 2.2. Running the P4DTI with Python 2.2 gives an error in the use of has_key at line 412 of bugzilla.py. 5.2. WHAT WAS NEW IN RELEASE 1.4.1 CRITICAL job000461: Bugzilla integration doesn't support multiple Perforce servers The Bugzilla integration doesn't work with multiple Perforce servers. The second replicator instance will fail. ESSENTIAL job000437: Jobspec not suitable for creating new jobs The P4DTI creates a Perforce jobspec that's fine for jobs which are replicated from the defect tracker, but not so good for jobs that are created in Perforce. job000478: TeamTrack 5.02 is supported The P4DTI currently supports TeamTrack 4.5, 5.0, 5.02, and 5.5 (see also job000409 and job000433). This adds to the support burden at Perforce and TeamTrack. Perforce have asked that the set of supported platforms be changed. Specifically, TeamTrack 5.01 and 5.02 should _not_ be supported, and the documentation should recommend that customers upgrade to TeamTrack 5.5. job000483: refresh.py comments are out-of-date The comments in refresh.py indicate that the script will "delete all the jobs on the Perforce server" and refresh them from the defect tracker. This is incorrect; refresh.py no longer deletes any jobs. job000491: jobspec has bad defaults for optional user fields In the Bugzilla integration, if the administrator has added a user field (e.g. 'reporter', 'qa_contact') to replicated_fields, creating jobs in Perforce with the default value of this field will fail. job000494: Bugzilla integration fails if qa_contact is in replicated_fields There is a Bugzilla field "qa_contact", which holds the User name of the "QA contact" assigned to a bug. This may be zero, if the Bugzilla parameter "useqacontact" is turned off, or if there is no QA contact assigned (for instance, if there is no initial QA contact for the component against which this bug is reported). If this field is zero, and the P4DTI is configured to replicate this field to Perforce (i.e. if 'qa_contact' is in replicated_fields in config.py) then the P4DTI will fail. job000501: new jobs break TeamTrack integration if LASTMODIFIEDDATE or SUBMITDATE are replicated In the TeamTrack integration, if either of the fields LASTMODIFIEDDATE and SUBMITDATE are replicated (i.e. are in the replicated_fields configuration parameter) and the P4DTI is set up to replicate new jobs (i.e. replicate_job_p is defined to return 1), and a new job is created without filling in these fields, the P4DTI fails. OPTIONAL job000408: Table of Bugzilla fields in AG is incomplete for Bugzilla 2.14 The Administrator's Guide has a table in section 5.1.5 listing the fields in a Bugzilla bug. The set of fields has changed in Bugzilla 2.14, with the addition of four new fields (reporter_accessible, qacontact_accessible, cclist_accessible, assignee_accessible). These fields are not really suitable for replication, but they should be added to the table in any case. job000492: Test suite doesn't cope with 'reporter' in replicated_fields If you add 'reporter' to replicated_fields in test/config_swan.py, the test suite fails in test_p4dti.py:migrate, saying there's no Bugzilla user corresponding to Perforce user ''. This is because the translate_jobspec function provided by that test doesn't provide a 'reporter' field (required by the new jobspec, but not present in the migrated jobs). job000493: Bugzilla migration fails if delta_ts in replicated_fields If the delta_ts field (the timestamp of the most recent change to a Bugzilla bug) is included in the replicated_fields configuration item, and the user's translate_jobspec function doesn't fill it in, migration will fail. job000495: Test suite doesn't cope with 'qa_contact' in replicated_fields If you add 'qa_contact' to replicated_fields in test/config_swan.py, the test suite fails in test_p4dti.py:migrate, saying there's no Bugzilla user corresponding to Perforce user ''. This is because the translate_jobspec function provided by that test doesn't provide a 'QA_Contact' field (required by the new jobspec, but not present in the migrated jobs). job000496: Bugzilla integration breaks migration if numeric fields are replicated Bugzilla bugs include a small number of numeric fields. One is actually a bitset ('groupset'), one is a genuine number ('votes'), and the rest are actually booleans: 'everconfirmed', 'qacontact_accessible', 'cclist_accessible', 'assignee_accessible', 'reporter_accessible'. If one of these fields is in 'replicated_fields', then migrating jobs without a translate_jobspec function to fill in a numeric value for this field will cause the migration to fail. 5.3. WHAT WAS NEW IN RELEASE 1.4.0 CRITICAL job000433: TeamTrack 5.02 not supported The P4DTI is not known to work with TeamTrack 5.02 (build 50207). job000447: Bugzilla 2.14.1 not supported Bugzilla 2.14.1 was released on 2002-01-05. This is a security patch release, fixing various holes in 2.14 (see the release notes -- link below). job000453: Perforce 2002.1 not supported P4DTI does not support Perforce 2002.1. job000459: TeamTrack 5.5 not supported The P4DTI is not known to work with TeamTrack 5.5 (build 55012). job000463: Bugzilla integration fails if mx.DateTime module is installed The Bugzilla integration breaks if the Python mx.DateTime module is installed on the P4DTI server system. ESSENTIAL job000016: Double replication causes many conflicts There is no reliable way for the replicator to distinguish its changes to jobs from changes made by other Perforce users. This means that everything replicated from the DT to Perforce is then replicated back again, and conflicts are very likely when making edits in the DT in quick succession. job000042: Rapid changes in the DT cause conflicts The replicator thinks there's a conflict if a DT issue is rapidly changed more than once. It replicates the issue to the job, then wakes up and replicates it back (see job job000033), but by now it's changed again in the DT, so the replicator believes there's a conflict. job000244: TeamTrack names the wrong user in an error message I was logged into Perforce as user "lmb" and set the state of a job to "assigned"; the corresponding user in TeamTrack didn't have the privilege to make this transition. I was expecting to get the error message "User lmb does not have privilege for transition Assign." but instead I got "User P4DTI-replicator0 does not have privilege for transition Assign." job000245: Unhelpful error from TeamTrack if you can't edit an issue If you edit an issue without permission, you get an error from TeamTrack like this: "User P4DTI-replicator0 does not have privilege for transition ." This isn't very helpful. job000309: Underscores don't work in Perforce jobviews If you have an underscore in your jobview then it doesn't work. For example, the jobview "state=ready_for_test" doesn't select a job even if the State field for that job has the value "ready_for_test". job000343: When using multiple Perforce servers, TeamTrack uses wrong changelist link When the P4DTI is running with multiple Perforce servers, TeamTrack can link to the wrong changelist URL in the fixes section of the issue. That is, it is only picking one changelist URL and using it for all issues, regardless of which Perforce server the issue is replicated to. job000352: Bugzilla emailsuffix parameter not supported Bugzilla has a parameter "emailsuffix" which is appended to all email addresses. P4DTI has no support for this, and will fail to translate users correctly if it is set. job000409: Not clear if TeamTrack 5.01 is supported or not It's not clear from the release notes and documentation whether the P4DTI supports TeamTrack 5.01. job000413: Bugzilla integration does not work with MySQLdb 0.9.0 or later The P4DTI Bugzilla integration uses a Python database library called MySQLdb to access the Bugzilla database. The P4DTI supports release 0.2.2 and 0.3.0 or this library. There are several more recent releases. The most recent releases do not work with the P4DTI. job000426: Migration to TeamTrack fails unless issue ids are zero-filled to 5 digits If you change the number of digits to which issue ids are zero-filled, migration and replication of new issues doesn't work. job000429: Error message in conflict e-mail might be wrong When the P4DTI sends out an e-mail explaining that a job and issue have been edited simultaneously and that it's overwritten the job with the issue, it might include an incorrect error message. job000436: Migration instructions don't help with deleting and restoring databases When you migrate, you'll probably have to delete and restore your defect tracker database several times. The instructions for migration in the Advanced Administrators Guide should explain how to do this. job000442: Can't replicate 'line' fields with hashes in them to Perforce If you have a field with a # in it in the defect tracker, and if that field is replicated to a 'line' field in Perforce, the contents don't get replicated correctly to Perforce because the # is interpreted as the start of a comment. job000445: Bugzilla replicator doesn't like spaces in enum fields The Bugzilla replicator doesn't translate an "enum" field correctly from Bugzilla to Perforce if the values in the field include spaces or other non-word characters. Two enum fields ("bug_status" and "resolution") are handled specially. For other enum fields (i.e. "priority" and "op_sys" in the default Bugzilla schema) there is no special handling and the replicator will fail if given a value with a non-word character such as a space. OPTIONAL job000086: Users can "fix" issues that they don't have permission to change A user can change the status for a job, and the corresponding defect tracker issue will change in status, even if the user doesn't own or have permission to change it. The user has to sneak in an illegal "p4 fix" after a legitimate user has edited the job and then delete the fix. job000276: Can't fix (and then delete the fix) a job owned by user (None) in the TeamTrack integration If an issue has no owner in TeamTrack (and therefore the owner of the corresponding job is (None) in Perforce) then you can't transition the job by fixing and then immediately delete the fix. The replicator is unable to replicate the change to the job and it overwrites the job with the issue. job000289: Users with spaces produce confusing errors A Bugzilla or TeamTrack user name containing spaces causes confusing errors to be produced by the P4DTI, especially when it doesn't correspond to a Perforce user, and perhaps even when it does. job000333: Poor error message from TeamTrack 5.0 when you specify wrong user or password If you specify the wrong user id or password in my P4DTI configuration, you get a misleading error message from TeamTrack. job000427: Can't delete associated filespec in Bugzilla integration In the Bugzilla integration, if you create an associated filespec record in Perforce (by adding a line to the P4DTI-filespecs field) and then delete it, then the deletion can't be replicated to Bugzilla. 5.4. WHAT WAS NEW IN RELEASE 1.3.2 CRITICAL job000422: Refreshing fails after migration if new workflow doesn't match old workflow If you have fixes in Perforce with statuses other than "closed", and if you migrate your jobs from Perforce to the defect tracker and in doing so change the workflow so that some fix statuses disappear, then the P4DTI can't refresh the fixes after the migration. ESSENTIAL job000249: Refresh script fails if you change the closed_state Suppose you start out with closed_state=None and people make fixes in Perforce to the state "resolved". Then you set closed_state="resolved" and run the refresh script. This fails because it tries to make fixes to the state "resolved", which Perforce refuses because there is no Perforce state "resolved" (Perforce now uses "closed" instead). job000421: Migration of jobs to Bugzilla fails if a job is owned by a non-existent user If you have jobs in Perforce that are owned by a non-existent user, or a user who's been deleted, then the P4DTI can't migrate those jobs to Bugzilla. 5.5. WHAT WAS NEW IN RELEASE 1.3.1 CRITICAL job000418: Migration of users to Bugzilla fails for lack of migrated_user_groups If you attempt to migrate users from Perforce to Bugzilla following the instructions in the Advanced Administrator's Guide, section 4.4, the operation fails because there is no value for the configuration parameter migrated_user_groups. job000419: Can't migrate users to Bugzilla 2.14 Migrating users from Perforce to Bugzilla works for Bugzilla 2.10 and 2.12, but in Bugzilla 2.14 it fails with a complaint from MySQL that the field 'password' is unknown. ESSENTIAL job000420: Too time-consuming to find all problems with migration With our current process, migrating jobs involves repeatedly running migrate.py, finding a problem with a particular job not being translated for some reason, adjusting the translate_jobspec and prepare_issue functions to take account of the exceptional case, and running migrate.py again. Since migration starts with the first job each time, this procedure is potentially O(n^2) in the number of jobs. 5.6. WHAT WAS NEW IN RELEASE 1.3.0 CRITICAL job000385: Renumbered changelist causes P4DTI error and deletes fix If a changelist fixing a job is submitted at the same time as a replication of that job is taking place, a race condition may occur which causes the fix record to be deleted and an error message to be sent by email. job000411: Bugzilla integration depends on MySQLdb returning lists A user reported an error from the Bugzilla integration, when the P4DTI was attempting to update a Bugzilla bug which had other bugs blocked on it but which was not blocked on any other bugs. ESSENTIAL job000022: No migration from Perforce jobs to defect tracker There's no way to migrate jobs from Perforce to the defect tracker. We used to have a script and instructions for doing this but it was so difficult to use and so prone to problems [GDR 2001-03-23] that we removed it from the AG. job000046: The replicator process is hard to manage The replicator process is hard to manage, especially on Windows NT, where one has to launch it using a non-trivial command line at a prompt. job000124: Changes by disabled Bugzilla users may be replicated Bugzilla has a mechanism for disabling users. We don't check this as part of checking user permissions. We should. job000145: Deleted TeamTrack states appear in Perforce jobspec If you delete an issue state in TeamTrack, it still appears among the values for the State field in the Perforce jobspec. Similarly for deleted options for a selection field like Priority or Severity. job000306: Bugzilla integration holds locks after errors The Bugzilla integration locks some Bugzilla tables during each replicator poll. If there is an error during the poll, the tables will remain locked until the next poll begins. This may be a long time if the replicator is failing and backing off exponentially. While the tables are locked, Bugzilla is unusable. job000376: In TeamTrack integration, P4Win crashes if you try to create a job If you try to create a job in P4Win when you're using the P4DTI jobspec, then P4Win crashes with the error message "integer division by zero". job000382: "[Errno 5] Input/output error" from Bugzilla integration logger Two separate users running the Bugzilla integration have encountered an "Input/output error" when writing to the log. job000387: TeamTrack 5.0 integration doesn't provide good error messages The TeamTrack 5.0 API introduced a number of new error codes, all of which are reported as "(unknown error)" by the P4DTI. job000398: May get conflicts after running the refresh script If you run the refresh script when there are still outstanding changes to replicate from the defect tracker, then when you start the replicator again after the refresh, you get conflicts. job000399: Some issue titles can't be replicated A customer found that the replicator couldn't replicate a particular issue from TeamTrack to Perforce, because the Perforce server could not update the job. Perforce claims that a required field is missing but in fact the field is present. job000405: Configuration section in AG needs improvement The configuration section in the AG (section 5.1) is hard to understand or hard to make use of in places. job000410: Can't confirm an unconfirmed Bugzilla bug from Perforce Trying to change a Bugzilla bug from UNCONFIRMED to another state by changing the status of the corresponding Perforce job causes an error. job000416: Fixed-point numbers treated as floating-point in TeamTrack integration The TeamTrack integration treats all double-precision numeric fields as floating-point. That means that if you are using a fixed-point field to represent a release number like 16.1 then it will come out in Perforce as 16.100000000000001. OPTIONAL job000036: New jobs in Perforce don't get replicated to the defect tracker New jobs in Perforce don't get replicated to the defect tracker. job000211: Mail can get lost If the replicator tries to send e-mail and fails (typically because of a problem with the SMTP server), then it just drops the mail. The contents are therefore lost. job000317: Error message is unhelpful when you're using an unsupported version of the MySQLdb module If you have an unsupported version of the MySQLdb module (that is, before MySQLdb 0.2.2), then you get an unhelpful error message. job000388: Error reporting is poor if you choose not to receive e-mail If you choose not to receive e-mail (by setting smtp_server to None in config.py) then the Python backtrace is not recorded when an error occurs (the log only records the error itself). job000403: Instructions on how to select issues to replicate are inadequate The AG has an entry for the replicate_p configuration parameter [1], but there's quite a lot more you need to know before you can actually write such a thing. NICE job000404: No startup message if you don't receive e-mail If you turn off e-mail by setting smtp_server or administrator_address to None then the replicator starts without printing anything. This is disconcerting to people who are used to it printing a startup message: it looks as though something's gone wrong. 5.7. WHAT WAS NEW IN RELEASES UP TO 1.2.1 For information about what was new in releases up to and including 1.2.1, see the release notes for that release: A. REFERENCES None. B. DOCUMENT HISTORY 2002-02-01 GDR Noted support for TeamTrack version 5.5. 2002-02-04 GDR Updated for release 1.4.0. 2002-03-14 NB Removed support for TeamTrack 5.02. 2002-04-09 NB Updated for release 1.4.1. 2002-04-19 NB Updated for release 1.4.2. C. COPYRIGHT AND LICENCE This document is copyright (C) 2002 Perforce Software, Inc. All rights reserved. Redistribution and use of this document in any form, with or without modification, is permitted provided that redistributions of this document retain the above copyright notice, this condition and the following disclaimer. THIS DOCUMENT IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. $Id: //info.ravenbrook.com/project/p4dti/release/1.4.2/release-notes.txt#1 $