Perforce Defect Tracking Integration Project


Perforce Defect Tracking Integration Advanced Administrator's Guide

Gareth Rees, Ravenbrook Limited, 2001-11-14

Contents

1. Introduction

This manual is the Perforce Defect Tracking Integration Advanced Administrator's Guide. It describes methods of administering the P4DTI that are outside the scope of the Perforce Defect Tracking Integration Administrator's Guide because they require a higher level of expertise (see "Required experience" below).

This document is intended for P4DTI administrators. Ordinary users of the defect tracker or Perforce should read the Perforce Defect Tracking Integration User's Guide.

This guide does not describe how to install, maintain, or perform ordinary configuration and administration of the P4DTI. Read the Perforce Defect Tracking Integration Administrator's Guide.

Required experience

To do the things described in this guide, you must have the following experience:

2. Select the issues to replicate

The P4DTI selects the issues that get replicated from the defect tracker using the replicate_p function. To select the issues that get replicated, edit this function in the file config.py in the installation directory.

Normally, the P4DTI replicates all issues created or modified after the start_date, but you can modify this function to further restrict the issues. See section 5.1, "P4DTI configuration", of the Perforce Defect Tracking Integration Administrator's Guide for more details of the start_date parameter.

Note: after an issue starts being replicated, it continues to be replicated, even if it no longer matches the replicate_p criteria.

The function is passed one argument, the issue, in the form of a Python dictionary that maps field names (strings) to field values. If the function returns 1, the issue is replicated. If it returns 0, the issue is not.

The replicate_p function does not have to check the start_date configuration parameter. The start_date is always honored by the P4DTI, in addition to any test made by replicate_p.

2.1. Bugzilla examples

This example restricts replication to unresolved issues in the "horseshoe" product:

def replicate_p(issue):
    return (issue["product"] == "horseshoe"
            and issue["resolution"] == "")

This example restricts replication to issues with the "nail" component:

def replicate_p(issue):
    return (issue["component"] == "nail")

This example restricts replication to all issues in the "horseshoe" and "nail" components, and additionally unresolved issues for the "harness" product with the "harvest fair" target milestone). Note the use of parentheses:

def replicate_p(issue):
    return (issue["component"] in ["horseshoe", "nail"] or
            (issue["product"] == "harness" and
             issue["resolution"] == "" and
             issue["target_milestone" == "harvest fair"))

3. Allowing users to create issues in Perforce

By default, the P4DTI requires users to create issues using the defect tracker. It ignores jobs created in Perforce. The P4DTI can be configured to transform new jobs created in Perforce into new issues in the defect tracker.

To cause the P4DTI to replicate jobs created in Perforce, add two functions to the end of config.py: replicate_job_p and prepare_issue.

3.1. The replicate_job_p function

The replicate_job_p function chooses the jobs in Perforce that get added to the defect tracker. The function takes the newly-created job as its argument, in the form of a Python dictionary that maps field names (strings) to field values (strings). If it returns 1, the job is added to the defect tracker and replicated thereafter. If it retuns 0, the job is not replicated.

Note: after a job starts being replicated it, continues to be replicated, even if it no longer matches the replicate_job_p criteria.

If you want all newly-created jobs to be translated and added to the defect tracker, write a replicate_job_p function as follows:

def replicate_job_p(job):
    return 1

3.1.1. Bugzilla example

To replicate newly-created jobs for products called "horseshoe" and "nail", add the product field to the replicated_fields configuration parameter. This field gets replicated to the job field "Product" in Perforce, so write a replicate_job_p function as follows:

def replicate_job_p(job):
    return job["Product"] in ["horseshoe", "nail"]

3.2. The prepare_issue function

The P4DTI does its best to build the new issue, but the defect tracker might require fields that can't be translated automatically from fields in the job. So the P4DTI calls the prepare_issue function to complete the transformation of a new job in Perforce to an issue in the defect tracker, by supplying values for fields that can't be translated automatically.

The function takes two arguments (issue, job):

issue

The partly-constructed issue, in the form of a Python dictionary that maps field names (strings) to field values. For example, in Bugzilla this might be:

{
    "bug_status":  "CLOSED",
    "resolution":  "FIXED",
    "short_desc":  "The widgets are specified in imperial units (ft/lb/s).",
    "longdesc":    "But 80% of customers use ISO 31, so we should use m/kg/s.\n"
    "priority":    "P1",
    "product":     "widget",
    "assigned_to": 32,
}
job

The job, in the form of a Python dictionary that maps field names (strings) to a field values (strings). For example:

{
    "Job": "job000123",
    "Status": "closed",
    "Assigned_To": "gdr",
    "Date": "2001/10/11",
    "Summary": "The widgets are specified in imperial units (ft/lb/s).",
    "Description": "But 80% of customers use ISO 31, so we should use m/kg/s.\n",
    "Priority": "P1",
    "Product": "widget",
}

The function must modify issue so that it is valid for the defect tracker. The following section explains what "valid" means for Bugzilla:

3.2.1. Preparing issues for Bugzilla

In the Bugzilla integration, new issues must have a valid value for the following fields:

product

The the name of a product in the Bugzilla products table.

If the product field is not one of the replicated_fields, then the P4DTI picks the empty string for issue["product"] (except when there's only one product, in which case the P4DTI picks that product).

component

The name of a component of the product to which the issue belongs.

If the component field is not one of the replicated_fields, then the P4DTI picks the empty string for issue["component"] (except when there's only one component of the product, in which case the P4DTI picks that component).

version

The name of a version of the product to which the issue belongs.

If the version field is not one of the replicated_fields, then the P4DTI picks the empty string for issue["version"] (except when there's only one version of the product, in which case the P4DTI picks that version).

You can use the prepare_issue function to supply default values for other fields in Bugzilla. See the Bugzilla database schema [NB 2000-11-14a] for details.

3.2.2. Bugzilla example

The following function supplies a default product, component and version if necessary:

def prepare_issue(issue, job):
    default_component = {
        "horseshoe": "user interface",
        "nail": "manufacturing",
        }
    default_version = {
        "horseshoe": "1.7",
        "nail": "2.3beta",
        }
    if issue["product"] == "":
        issue["product"] = "horseshoe"
    product = issue["product"]
    if issue["component"] == "":
        issue["component"] = default_component[product]
    if issue["version"] == "":
        issue["version"] = default_version[product]

4. Migrating to the defect tracker from Perforce jobs

Migration is the process of loading your Perforce jobs into the defect tracker.

4.1. Planning for migration

4.1.1. Configuring and testing the P4DTI and migration

  1. Create a Perforce server for testing.

  2. Create a defect tracker for testing. If you have existing defect tracking data, make a copy that includes your workflow and a representative sample of your existing issues.

  3. Set up and test the P4DTI on the test Perforce server and test defect tracker, following the instructions in the Perforce Defect Tracking Integration Administrator's Guide. Test the integrated system and verify that the workflow is correct.

  4. Record the jobspec in your test Perforce server, either created by the P4DTI or (using keep_jopspec) installed by you and extended by extend_jobspec.py (see section 5.2.3, "Handle Perforce jobs and jobspec", of the Perforce Defect Tracking Integration Administrator's Guide). For example run a command like p4 jobspec -o > saved-jobspec, or print out the jobspec. This jobspec is used in configuring the translation step of the migrator. (See section 4.7, "Translating to the new jobspec".)

  5. Replace the test Perforce server with a server that has a copy of all your jobs and fixes. (Create a checkpoint of your main Perforce server and restore it onto a new empty Perforce server. See chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.)

  6. Set up and test migration from the test Perforce server to the test defect tracker, as described in section 4.2, "How to migrate".

  7. Apply any workflow changes you made on your test defect tracker to your main defect tracker server.

When you are confident that you have the correct configuration, proceed to section 4.1.2, "Going live with the P4DTI with migration".

4.1.2. Going live with the migrated data

After you have configured and tested the P4DTI with migration, follow these steps to make the system available to your users:

  1. Ask your users to stop using the defect tracker and Perforce server.

  2. Make a backup up of your Perforce repository and your defect tracker database, in case there are any serious problems.

  3. Edit your config.py file to point to your main Perforce and defect tracker servers.

  4. Run the migration and check the results (4.10).

  5. Start the P4DTI.

4.2. How to migrate

Migration cannot be done automatically by the P4DTI, because the data in Perforce is in a very different format from the defect tracker, and the way your workflow is implemented in Perforce is likely to be different from the way it's implemented in the defect tracker.

Here is an overview of migration. The details of each step are covered in the following sections.

  1. Prepare the Perforce data for migration:

    1. Ensure that users in Perforce match users in the defect tracker (4.3 and 4.4).

    2. Make Perforce jobs consistent with the current jobspec so the P4DTI can transform them correctly (4.5).

  2. Write Python functions in your config.py to do the following:

    1. Select the jobs to migrate (4.6).

    2. Translate these jobs so they match the jobspec you plan to use (4.7).

    3. Satisfy any defect tracker constraints on submitted issues (4.8).

  3. Migrate the selected jobs to the defect tracker (4.10).

4.3. Provide names and e-mail addresses for Perforce users

Give every Perforce user a name and e-mail address, to enable the P4DTI to automatically make user accounts in Bugzilla for each Perforce user (see section 4.4, "Making defect tracker user accounts").

4.4. Create defect tracker user accounts

In your defect tracker, create a user account for every Perforce user who originates or works on issues.

For Bugzilla, follow these steps to create the users automatically:

  1. Specify dbms_host, dbms_port, dbms_user and dbms_password in config.py if you haven't already done so.

  2. Specify these two configuration parameters in config.py (unless you're happy with the default values):

    • migrated_user_groups is a list of Bugzilla groups. Automatically created users in Bugzilla belong to all the groups in this list. For example, if migrated_user_groups = ["editbugs", "canconfirm"] then users can edit and confirm bugs. The default value is [] (that is, users belong to no groups).

      To see the Bugzilla groups and their meaning, log into Bugzilla as an administrator and select "edit groups" from the options at the bottom right.

    • migrated_user_password is a string. Automatically created users have this password in Bugzilla. For example, if migrated_user_password = "spong" then users have password "spong". The default value is "password".

  3. Go to the P4DTI installation directory.

  4. Run the command python migrate_users.py.

4.5. Make Perforce jobs consistent with the jobspec

If you change the jobspec, Perforce doesn't update its existing jobs to match, so some jobs might be inconsistent with the revised jobspec. Examples: a "required" field that is missing from some jobs; a "select" field for which some jobs have an illegal value. These problems prevent the P4DTI from reading the jobs.

To find inconsistencies, go to the P4DTI installation directory and run the command python check_jobs.py (it takes about a minute for every thousand jobs on a typical system). For example:

$ python check_jobs.py
(P4DTI-10081)  Job 'job000024' doesn't match the jobspec:
(P4DTI-7087)  Error detected at line 11.
Value for field 'Status' must be one of open/suspended/closed.

Fix these jobs so that they match the jobspec. When this is done, you'll get the following output:

$ python check_jobs.py
(P4DTI-10092)  All jobs match the jobspec.

4.6. Select the jobs to migrate

To select the jobs to migrate, add the function migrate_p to the end of config.py.

The migrate_p(job) function determines whether to migrate a job to the defect tracker. The job argument is the job being considered for migration (in the old jobspec), in the form of a dictionary that maps field names (strings) to field values (strings). If it returns 1, the job is migrated. If it returns 0, the job is not migrated.

4.6.1. Examples of migrate_p

Example 1. To migrate all jobs:

def migrate_p(job):
    return 1

Example 2. To migrate only open jobs:

def migrate_p(job):
    return job["Status"] == "open"

Example 3. To migrate only jobs in the "horseshoe" and "nail" projects:

def migrate_p(job):
    return job["Project"] in ["horseshoe", "nail"]

4.7. Translate jobs to the new jobspec

Add the function translate_jobspec to the end of config.py.

The translate_jobspec function translates a job from the old jobspec (before migration) to the new jobspec (after migration). Note that the jobspecs might be very similar, especially if you use keep_jopspec, so this function might be very simple. The old job is passed as the single argument, in the form of a dictionary that maps field names (strings) to field values (strings). The new job must be returned in the same form (it's OK to update and return the argument).

4.7.1. Bugzilla example

Example Perforce job in the old jobspec:

Job: job000123
Status: closed
User: gdr
Date: 2001/10/11
Description:
        The widgets are specified in imperial units (ft/lb/s).
        But 80% of customers use ISO 31, so we should use m/kg/s.
Priority: high
Project: widget

You plan to set replicated_fields=["resolution", "short_desc", "priority", "product"], so jobs look like this in the new jobspec:

Job: job000123
Status: closed
Assigned_To: gdr
Date: 2001/10/11
Summary:
        The widgets are specified in imperial units (ft/lb/s).
Description:
        But 80% of customers use ISO 31, so we should use m/kg/s.
Resolution: FIXED
Priority: P1
Product: widget

So these changes are needed:

  1. The old jobspec has a single "Description" field, but the new jobspec has both a "Summary" (one line) and a "Description" (many lines). So split the old description so that its first line becomes the new summary.

  2. Rename the "User" field as "Assigned_To".

  3. In the old jobspec, the status is "open", "closed" or "suspended". In the new jobspec, the status is represented by a combination of "Status" and "Resolution" fields. Translate status as shown in the table below.

    Old status New status Resolution
    "open" "assigned" ""
    "closed" "closed" "FIXED"
    "suspended" "closed" "LATER"

Here's a translate_jobspec function that implements all the changes discussed above:

def translate_jobspec(job):
    # Description -> Summary + Description
    import string
    desc = job.get("Description", "")
    newline = string.find(desc, "\n")
    job["Summary"] = desc[:newline]
    job["Description"] = desc[newline+1:]

    # User -> Assigned_To
    job["Assigned_To"] = job.get("User", "")

    # Translate Status -> Status + Resolution
    status_map = {
        "open": ("assigned", ""),
        "closed": ("closed", "FIXED"),
        "suspended": ("closed", "LATER"),
        }
    (status, resolution) = status_map[job.get("Status", "open")]
    job["Status"] = status
    job["Resolution"] = resolution

    return job

4.8. Transform jobs to issues

Jobs translated to the new jobspec might not be suitable for submission to the defect tracker. The defect tracker might have additional constraints, or require fields that can't be translated from fields in the job. You must write a prepare_issue function as described in section 3.2, "The prepare_issue function".

4.9. Summary of translation, transformation, and preparation steps

Figure 1 shows a summary of the translate, transform, and preparation steps of the migration process:

  1. Translate the job from the old jobspec to the new jobspec using the translate_jobspec function.

  2. The P4DTI transforms the job to an issue by converting each field in replicated_fields and any system fields.

  3. Prepare for submission to the defect tracker using the prepare_issue function.

Figure 1. The translate, transform, and prepare steps. The example data in the figure is for Bugzilla.

Diagram of the translate, transform, and prepare steps during migration Step 1:
translate from old jobspec to new jobspec Step 2:
translate job to issue. Step 3:
prepare issue for submission to the defect tracker.

4.10. Migrate

  1. Make a checkpoint of the Perforce server as described in chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.

  2. Go to the P4DTI installation directory and run the command python migrate.py. If it succeeds, it prints output like this. (If you see no output, then your migrate_p function isn't matching any jobs.)

    2001-11-14 12:34:13  (P4DTI-8920)  Migrated job 'job000001' to issue 'BUG00001'.
    2001-11-14 12:34:13  (P4DTI-8920)  Migrated job 'job000002' to issue 'BUG00002'.
    2001-11-14 12:34:13  (P4DTI-8206)  -- Added fix for change 3326 with status closed.
    ...
    2001-11-14 12:35:02  (P4DTI-8953)  Migration completed.

    Notes for Bugzilla users:

    • The P4DTI locks the Bugzilla database while migrating, so no-one can use Bugzilla until migration completes.

    • If you've specified a value for bugzilla_directory, you'll see the message "Running processmail for %d bugs..." after the last job is migrated and before the "Migration completed" message appears. This is because Bugzilla's processmail script is being run for each migrated issue. (If you don't want notifications to be sent when migrating, set bugzilla_directory=None while migrating and reset it afterwards.)

  3. Handle errors (see section 4.11). When you've fixed the problem, run python migrate.py again. Provide the command-line option -s JOBNAME to start migrating at a particular job. For example, if migration failed because of a problem job000123, then continue migrating with python migrate.py -s job000123.

  4. In the defect tracker, verify that all the Perforce jobs you want to replicate are present and that the values in all fields are correct. If not, fix the problems by hand, or edit the migrate_p, translate_jobspec and prepare_issue functions to correct the problem, and run migration again.

  5. If using keep_jopspec, update the jobspec to support the P4DTI, for instance by running python extend_jobspec.py. See section 5.2.3, "Handle Perforce jobs and jobspec", of the Perforce Defect Tracking Integration Administrator's Guide.

  6. Replace the existing jobs in Perforce with jobs replicated from the defect tracker's database by refreshing them as described in section 9.2, "Refreshing jobs in Perforce", of the Perforce Defect Tracking Integration Administrator's Guide.

  7. Check the consistency of the replicated data by running the consistency checker as described in section 7.3, "Checking data consistency", of the Perforce Defect Tracking Integration Administrator's Guide. If this consistency check fails, fix the problems by hand, or restore your Perforce server from the checkpoint you made earlier, edit the migrate_p, translate_jobspec and prepare_issue functions to correct the problem and run migration again.

  8. Start the P4DTI as described in section 5.5, "Starting and stopping the replicator manually", of the Perforce Defect Tracking Integration Administrator's Guide.

4.11. Handle migration errors

These kinds of error are likely to be encountered during migration:

5. Integrating with multiple Perforce servers

The P4DTI can be set up to replicate defects from a single defect tracker to multiple Perforce servers. For example, if you have several development groups, each with their own Perforce server, you can set up the P4DTI to replicate the issues for each group to the appropriate Perforce server.

Note: the P4DTI doesn't support integration between one Perforce server and multiple defect tracking servers.

To replicate to multiple Perforce servers:

  1. Install a copy of the P4DTI for each Perforce server, as described in readme.txt.

  2. Configure each copy of the P4DTI. The following configuration parameters must be different for each copy:

  3. Start and manage each P4DTI as described in the Perforce Defect Tracking Integration Administrator's Guide.

Note that there's no way to move issues from one Perforce server to another. Once an issue is replicated to one Perforce server then continues to be replicated to that server, even if it later matches the replicate_p function for another replicator. If it's doubtful which development group is going to work on an issue, then you must design your workflow so that issues don't get replicated until it's clear who's going to work on them.

For example, you might have a replicate_p function like this:

def replicate_p(issue):
    return (issue["product"] in ["horseshoe", "nail"] and
            issue["status"] == "ASSIGNED"))

so that it will not get replicated while it has status "UNCONFIRMED" or "NEW". In this way you get an opportunity to take a decision on the issue and possibly move it to a different project.

Multiple Perforce server example

You have Perforce servers for your development groups in France and Germany and a single Bugzilla server. products "horseshoe", "nail", and "bit" are worked on in France and projects "plough", "poker", "buckle" are worked on in Germany. So set up one P4DTI copy with the following in config.py:

rid = "replicator_fr"
sid = "france"
p4_port = "perforce.company.fr:1666"
p4_server_description = "French development group"
changelist_url = "http://perforce.company.fr/cgi-bin/perfbrowse?@describe+%d"
job_url = "http://perforce-company.fr/cgi-bin/perfbrowse?@job+%s"
def replicate_p(issue):
    return issue["product"] in ["horseshoe", "nail", "bit"]

and the other with the following:

rid = "replicator_de"
sid = "germany"
p4_port = "perforce.company.de:1666"
p4_server_description = "German development group"
changelist_url = "http://perforce.company.de:8080/%d?ac=10"
job_url = None
def replicate_p(issue):
    return issue["product"] in ["plough", "poker", "buckle"]

A. References

[GDR 2000-10-16] "Perforce Defect Tracking Integration Integrator's Guide"; Gareth Rees; Ravenbrook Limited; 2000-10-16.
[MySQL 2001] "MySQL Manual"; MySQL; 2001.
[NB 2000-11-14a] "Bugzilla database schema"; Nick Barnes; Ravenbrook Limited; 2000-11-14.
[Perforce 2001-06-18b] "Perforce 2001.1 System Administrator's Guide"; Perforce Software; 2001-06-18; <http://www.perforce.com/perforce/doc.011/manuals/p4sag/>, <ftp://ftp.perforce.com/pub/perforce/r01.1/doc/manuals/p4sag/p4sag.pdf>.
[RB 2000-08-10a] "Perforce Defect Tracking Integration Administrator's Guide"; Richard Brooksby; Ravenbrook Limited; 2000-08-10.

B. Document History

2001-11-14 GDR Branched from Administrator's Guide.
2001-11-21 GDR Note the password for migrated Bugzilla users.
2001-11-22 GDR Documented the need to refresh Perforce jobs after migrating if you want to move to an integrated system.
2001-11-23 RB Substantial revision, editing for clarity and adding details where necessary. Devised and added a section giving a plan for installing with migration.
2001-11-25 GDR Added material on replicating to multiple Perforce servers and on querying the TeamTrack database.
2001-11-27 GDR Added instructions for setting migrated_user_groups and migrated_user_password. Documented the -s option to the migration script.
2002-01-10 GDR Noted non-availability of some advanced features in old TeamTrack releases.
2002-02-04 GDR Explain how to back up and restore databases. Refer to TeamTrack 5.5 documentation.
2003-05-21 NB de-TeamTrack.

This document is copyright © 2001 Perforce Software, Inc. All rights reserved.

Redistribution and use of this document in any form, with or without modification, is permitted provided that redistributions of this document retain the above copyright notice, this condition and the following disclaimer.

This document is provided by the copyright holders and contributors "as is" and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the copyright holders and contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this document, even if advised of the possibility of such damage.