Emacs/Perforce integration: a retrospective

,

I’ve been maintaining the Perforce/Emacs integration for a couple of years now, so it’s time for a retrospective. The first few months of work, described here, were spent on three areas needing immediate attention:

  1. Fixing bugs. You no longer lose unsaved changes in a buffer when you run the p4-edit command. The ‘mode check’ runs in the background, meaning that if you are offline, or the Perforce server is otherwise unavailable, then Emacs no longer hangs each time you open a file. If you are logged out of Perforce, issuing any Perforce command automatically prompts you for your password.
  2. Supporting modern features of Perforce. I added interfaces to many commands, most importantly annotate, grep, move, reconcile, status, shelve, and unshelve.
  3. Supporting modern features of Emacs. Diffs are now displayed using the same colours and key bindings as diff-mode. Unicode characters are decoded when received and encoded when sent. The global-font-lock-mode setting is respected instead of rolling our own.

Having caught up, the work has been driven largely by my own work practices, a mode of development known as ‘eating your own dog food’. This has some big advantages: Perforce and Emacs are tools that I use all day every day, which means that small inconveniences in the integration stand out to me and I can see clearly cases where I have a frequently performed task that deserves some extra automation or assistance. I’m also in a position to check quickly that my change really fixes the problem. Here are examples of improvements I’ve made based on this approach:

  1. Several of the objects managed by Perforce (branch specifications, change descriptions, client specifications, and jobs) are represented as collection of key/value pairs, and these can be printed and edited using the form representation, which looks like this:

    Job:    job003890
    
    Status: closed
    
    User:   gdr
    
    Title:  No error if you create a pool using a format from another arena
    
    Description:
            You can create a pool in one arena using a format from another
            arena, and the MPS does not detect this or complain.
    

    Having edited this form and committed it to the Perforce server, it’s natural to want to make further changes to the form and commit those too. This used to be inconvenient: you had to run the p4-job command (or p4-branch, p4-change, and so on) again to request a new copy of the form from the server, and edit that. So I added the ability to edit a committed form and commit it again to the server.

    This raised a problem: when a form is committed, the Perforce server may rewrite it. For example, in order to create a new job, you put the special value new in the Job field. When the form is committed to the Perforce server it allocates a new unique job identifier and rewrites the form. (There’s a similar feature for changelists.) So I added code to parse the new job identfier or changelist number out of the server’s response, and rewrite the form accordingly, so that you can continue to edit it.

  2. Now that forms can be easily edited and committed, I found myself accidentally typing C-x C-s, the standard Emacs keybinding for save-buffer, instead of C-c C-c, the keybinding for p4-form-commit. The similarity of forms to files was strong enough to activate my muscle memory! So I went with the flow and added a binding from C-x C-s to p4-form-commit.

  3. The output of the p4-help command often contains cross-references to other help topics, for example the output of p4 help usage includes the text:

    The -C flag specifies the client's character set, overriding the
    value of $P4CHARSET in the environment.  See 'p4 help charset'
    for more information.
    

    It used to be the case that following the cross-reference to p4 help charset was inconvenient: you had to type C-x p h charset RET. So I arranged to parse the output of p4-help and make clickable links whenever the patterns p4 \w+ and p4 help \w+ (among others) appear.

  4. When you have files opened in numbered pending changelists, the output of the p4-opened command looks like this:

    //depot/mps/master/code/poolamc.c#86 - edit change 187778 (ktext)
    //depot/mps/master/design/poolamc.txt#12 - edit change 187778 (ktext)
    

    It used to be the case that if you were looking at this buffer, viewing or editing the numbered pending change was inconvenient: you had to type C-u M-x p4-change RET 187778 RET. So I arranged to make clickable links for pending change numbers.

  5. It used to be inconvenient to change the type of a file. To set the file type to ktext, you had to find the file, and then type C-u M-x p4-reopen RET C-a -t ktext RET. So I added a new command p4-opened-list-type, bound to t in the *P4 opened* buffer. Now, to set the file type to ktext, you find the file in the *P4 opened* buffer and type t ktext RET.

  6. Similarly, it used to be inconvenient to move an opened file from one pending changelist to another. You had to find the file and then type something like C-u M-x p4-reopen RET C-a -c 187778 RET. So I added a new command p4-opened-list-change, bound to c in the *P4 opened* buffer. Now, you find the file in the *P4 opened* buffer and type c TAB which shows you a list of your pending changelists, and you click on the one you want.

  7. Being automatically prompted for your password when logged out of the Perforce server is a big improvement on the previous state of affairs, but it’s not really compatible with modern password management practices, in which your keychain manager remembers your password, but you don’t. Now, your password can be retrieved from the system keychain on OS X or Linux (if you customize p4-password-source appropriately).1

  8. When the Perforce server’s public key changed, it used to be the case that all Perforce operations would stop working until you ran p4 trust again. Now, you are shown the fingerprint of the new public key and prompted to say whether it should be trusted.

    I discovered this problem as a lucky consequence of the otherwise very unlucky Heartbleed bug. Perforce’s servers contained the vulnerable OpenSSL code and so after installing the patched binaries we regenerated our keys.2

  9. When I first added support for the shelve and unshelve commands, they were minimal wrappers around the Perforce command line. But I’ve found myself making a fair bit of use of shelves over the last year or so, and the awkwardness of unshelving eventually stood out as a problem. You had to find the shelved changelist by running p4 changes -s shelved -c $P4CLIENT and then run p4 unshelve -s 187779, and I found that I often mistakenly used the -c option (which specifies the destination changelist for unshelving the file into) rather than -s (which specifies the shelved changelist). So I changed the default behaviour of the p4-unshelve command: now it prompts for the shelved changelist, with TAB-completion.

  10. The output of the p4-fixes command looks like this:

    job003908 fixed by change 187660 on 2015/02/04 by gdr@gdr-peewit (closed)
    job003912 fixed by change 187662 on 2015/02/04 by gdr@root-raven (closed)
    job003913 fixed by change 187681 on 2015/02/06 by gdr@gdr-peewit (closed)
    

    It used to be inconvenient to navigate from this buffer to the jobs and changes; now these are clickable links.

  11. It used to be inconvenient to find the fixes associated with a job. You had to type C-u M-x p4-fixes RET -j job003913 RET. Now, if you are looking a job, then the command p4-job-form-fixes, bound to C-c C-f, shows the fixes.

So those are the kinds of improvement that it’s easy to make using the ‘dogfooding’ approach: I notice a small inconvenience in my own work practice, and make a small change that automates a step or two.

On the other hand, both Emacs and Perforce have mature, highly complex interfaces that support many kinds of work practice. The way I use them is just one of many possible approaches, and when it comes to the other approaches, I don’t have any insight into the extent to which the integration helps or hinders those work practices.

When I worked on video games we spent a lot of time trying to figure out whether our games were playable: whether the goals were clear, the inputs discoverable, the puzzles challenging but solvable. We did this by watching people play: informally, over their shoulder, or formally, by video-recording a session of play. It was necessary to do this because we were trying to discover what people had understood about the game and what they hadn’t. Few people can explain the former and hardly anyone can explain the latter.

Here are three examples, in increasing order of success.

  1. I’ve never made any serious use of Perforce’s streams feature, so I have no idea what the use cases are, or how the feature gets used in the course of a normal day’s development. So it would be a lot of work for me to add support for streams: I’d have to do a lot of research to figure out how other people are using them: work that, realistically, I’m not going to do on my own initiative. What I’m missing is someone who uses streams all the time, can clearly describe their use cases to me, is willing to try out my fixes and report back on whether they work and whether features and keybindings are discoverable. People with the skill and time to do that are very rare.

  2. I use Perforce’s shelves from time to time, but I have only two use cases for them:

    1. Branching work in progress, by running p4 shelve on the source branch, creating a branch specification, running p4 integ -b branch to create the target branch, and then running p4 unshelve -b branch to map the shelved files through the branch specification prior to being unshelved.
    2. Moving work in progress from one machine to another, by running shelve on one machine and unshelve on the other.

    These use cases are too specific for me to want to provide special-case user interfaces for. To actually save effort over just running the commands would require knowledge of the branch layout in case (a), and I don’t see any way to do it at all in case (b). So my use cases don’t provide me with any guidance as to how to make generally useful interfaces to the shelve and unshelve commands.

  3. Some users like to manage their work by maintaining multiple numbered pending changelists, and would like to be prompted, each time they open a file for add, delete, edit, or integrate, which numbered pending changelist to open that file under. This is a use case that simply never would have occurred to me. I prefer to work on one thing at a time, and submit it as soon as possible, for the following reasons:

    1. working on multiples change at the same time makes it hard for me to remember what I’m doing;
    2. multiple pending changes providing no isolation of changes from each other (branches are better in this respect);
    3. keeping changes pending on your client runs a risk of loss if your hard drive failed (shelves would be better as at least the work in progress is stored on the server, and branches are best because the work can be submitted regularly as you go along);
    4. you can't transfer pending changes between machines (shelves are better and branches are best);
    5. pending changes don't get tested by automatic build and test infrastructure;
    6. pending changes don't get used by other developers, and so result in more complex merges when you do come to submit them;
    7. pending changes are a burden for other developers if you are ill or quit (they have to grub around on your disk to recover your work, instead of being able to get it out of the repository).

    Despite these reservations, I believe that these users have a genuine case, and so I provided a new customization option, p4-open-in-changelist: if non-NIL, this causes Emacs to prompt for a numbered pending changelist when opening files.3


  1.  I still haven’t figured out how to do this on Windows. In fact, even after reading the answers to this Stack Overflow question I still haven’t figured out where the Windows system keychain is. If you have any idea about what to do on Windows, let me know!

  2.  The Perforce command line has its own inconvenience in this situation. When the server is untrusted, you can’t get help on the command that you need to run to trust the server again:

    $ p4 help trust
    ******* WARNING P4PORT IDENTIFICATION HAS CHANGED! *******
    It is possible that someone is intercepting your connection
    to the Perforce P4PORT '123.45.67.89:1666'
    If this is not a scheduled key change, then you should contact
    your Perforce administrator.
    The fingerprint for the mismatched key sent to your client is
    B7:3D:56:4F:13:90:A1:48:26:EB:6B:F5:97:63:35:E8:81:F3:7A:57
    To allow connection use the 'p4 trust' command.
    

    In fact, Perforce have thought about this problem and you can run p4 trust -h at this point, but how are you supposed to discover this? If only you could run p4 help trust, then you’d see:

    trust -- Establish trust of an SSL connection
    
        p4 trust is a client side command.  For detailed help run
        p4 trust -h
    

    But the only circumstance in which you need to know this is exactly the circumstance in which you’re not allowed to see it!

  3.  But before I could provide this, I had to solve the problem that completion on pending changelists just showed meaningless numbers, and this led me to discover Emacs’ completion annotations mechanism.