garethrees.org

Circular logic

Wed, 15 May 2013 00:00:00 GMT

How many 6-input binary truth tables, $τ$, satisfy the formula $$ τ(a, b, c, d, e, f) ∧ τ(b, c, d, e, f, a ⊕ (b ∧ c)) = 0 $$ for all 6-bit inputs $(a, b, c, d, e, f)$?

Spoilers below.

★

The formulae are all of the form $τ(A) ∧ τ(B) = 0$ for two truth table rows $A$ and $B$. In other words, any truth table satisfying these formulae must assign true to at most one of $τ(A)$ and $τ(B)$. Each formula thus specifies a constraint joining two rows of the truth table. What happens if we plot all 64 truth table rows as nodes in a graph, with the constraints as edges? Each satisfying truth table would correspond to a colouring of the nodes of the graph with the colours black (representing true) and white (representing false), such that no two black nodes are adjacent.

The web site chart.ravenbrook.com provides a handy way to draw this graph:

def chart_url_209():
    """Return a Ravenbrook Chart URL for the graph generated from Project
    Euler problem 209.

    """
    edges = []
    for i in range(1 << 6):
        a, b, c = i >> 5 & 1, i >> 4 & 1, i >> 3 & 1
        j = (i << 1 & 63) + a ^ (b & c)
        if i != j: # Limitation of chart.ravenbrook.com (no self edges)
            edges.append('{},{}'.format(i, j))
    return ''.join(['http://chart.ravenbrook.com/chart?chs=1000x1000&chl=',
                    '|'.join('{:06b}'.format(i) for i in range(1 << 6)),
                    '&che=', '|'.join(edges)])

And here’s the graph:

In order to count the number of truth tables in the original problem, we need to count the ways of two-colouring the nodes of this graph such that no two black nodes are adjacent. The graph is made up of six components, and each component can be coloured independently. So the total number of colourings is the product of the number of colourings of each component. You’ll observe that the graph is cyclic, with cycles of length 1, 2, 3, 6, 6, and 46. So the problem reduces to counting the number of ways there are to two-colour a cycle of length $n$ such that no two black nodes are adjacent.

How to do that? Well, suppose we want to pick $k$ non-adjacent nodes from a cycle of length $n$. (Where $0 \le k \le \lfloor {n \over 2} \rfloor$.) Start by placing the $k$ black nodes with another $k$ white nodes between them, to make gaps that keep them from being adjacent. The remaining $n − 2k$ white nodes can be placed in any of the $k$ gaps: there are $C(n−k−1, k−1)$ ways to do this. Then multiply by $n$ (because the first black node may go anywhere in the cycle) and divide by $k$ (because the black nodes are indistinguishable).

def combinations(n, k):
    """Return C(n, k), the number of combinations of k out of n."""
    c = 1
    k = min(k, n - k)
    for i in range(1, k + 1):
        c *= n - k + i
        c //= i
    return c

def cyclic_nonadjacent_combinations(n, k):
    """Return the number of ways to choose k elements from a cycle of
    length n, such that none of the chosen elements are adjacent.

    """
    if n < 0 or k < 0 or n - k < k: return 0
    if k == 0: return 1
    return n * combinations(n - k - 1, k - 1) // k

def cyclic_nonadjacent_two_colourings(n):
    """Return the number of ways to two-colour a cycle of length n such
    that no black nodes are adjacent.

    """
    return sum(cyclic_nonadjacent_combinations(n, k) for k in range(n // 2 + 1))

Let’s check that for some small values of $n$:

>>> [cyclic_nonadjacent_two_colourings(n) for n in (2, 3, 4)]
[3, 4, 7]

This solves the problem in 125 µs:

def problem209():
    c = cyclic_nonadjacent_two_colourings
    return c(1) * c(2) * c(3) * c(6) * c(6) * c(46)

>>> from timeit import timeit
>>> timeit(problem209)
125.88723683357239

But is there more to these counts of cyclic nonadjacent two-colourings? Let’s compute a few more values and look them up in the On-Line Encyclopedia of Integer Sequences.

>>> [cyclic_nonadjacent_two_colourings(n) for n in range(1, 16)]
[1, 3, 4, 7, 11, 18, 29, 47, 76, 123, 199, 322, 521, 843, 1364]

These are are the Lucas numbers starting with 1 and 3 (A000204): in particular, each number is the sum of the previous two. This is because we can adopt the following procedure to generate the cycles of length $n$. We take each cycle of length $n-1$, cut it at a distinguished edge, and insert a white node. In these cases the neighbours of the inserted node are not both black (since they were adjacent in the smaller cycle). We also take each cycle of length $n-2$, duplicate a distinguished node, and insert a node of the opposite colour between the duplicates. In these cases either the inserted node is black, or else it’s white with both neighbours black. (Thus no duplicates arise in the procedure.) The figure below shows how this works in the case of $n = 4$.

Language-oriented programming

Thu, 09 May 2013 00:00:00 GMT

There’s a conventional piece of software development wisdom, to the effect that if writing your own programming language is an absurd and wasteful activity, because developing a programming language is a hideous time-sink, and because some existing language is almost certain to satisfy your needs perfectly well.

There’s a good deal of truth in this advice, in the case where you find yourself trying to develop a general-purpose programming language, and high performance is critical. The vast range of functionality needed to be general purpose overwhelms the capacities of a small team, and it is well known that making an optimizing compiler targeting general-purpose hardware requires an investment of many years of effort.

But if you’re not facing that situation, then I think that writing your own language has a lot to be said for it:

Languages are not just for describing algorithms to computers: they are for communication with other people. Other programmers, of course, but also non-programmers (or less expert programmers) like designers and managers. And your future self, who has forgotten the details of the implementation. If the syntactic form of the language is familiar to its readers, they have a better chance of being able to understand the code and review it for correctness (even if they cannot necessarily write it).

For example, in Mathematica, mathematical expressions are basic objects: thus, the polynomial x² + 3x + 2 is represented by the code:

x^2 + 3 x + 2

Compare, for example, with the corresponding representation in Numpy:

Polynomial([2, 3, 1])

In the Mathematica case it’s possible for someone completely unfamiliar with the language to verify that the correct polynomial has been entered, but this is not the case in Numpy: for example, which number is the units coefficient?¹

Or consider the meaning of the following snippet of code in Inform 7. Even if you don’t know the first thing about Inform 7, you can probably guess what it means:

The player is in Longwall Street. The player knows detect trap, memorise, fashion staff, know nature and exorcise undead. The player wears a leather jerkin. The player carries a dagger.
Implementing your own language allows you to impose modelling constraints on programs, making it impossible to express certain kinds of common errors, like dereferencing of null pointers, out-of-bounds array access, or the addition of numbers representing measurements in different units.
You can develop an execution model that’s completely different from the execution model of the language that it’s embedded in. It’s often much clearer and more convenient to be able to express concurrent programs in coroutine style, rather than in the form of state machines embedded in a single-threaded program (even if that’s actually how they are implemented).
A higher-level language aids portability by abstracting away details of the underlying platform. You can port to new platforms by reimplementing parts of the runtime.
It would be a shame and a waste to have studied lexers and parsers and code generators without actually going on to make use of it!

The name language-oriented programming is sometimes given to the technique of designing a language as the first step in modelling a problem domain. The term was coined by Martin Ward in his 1994 paper “Language Oriented Programming”. Ward made some quite expansive claims for the technique:

Our experience with FermaT, and the experiences from other projects, indicate that a system implemented using the language oriented method, as a series of language levels, ends up much smaller than an bottom up or top down implementation of the same system. This is due to the fact that with a problem-specific very high level language, a few lines of code are suffcient to implement highly complex functions. The implementation of the language is also kept small since only those features which are relevant to the particular problem domain need to be implemented.

The small size of the final system means that the total amount of development work required is reduced, without increasing the complexity of the system, and for the same or higher functionality. This leads to improved maintainability, fewer bugs, and improved adaptability

★

To demonstrate the power of the technique, the program below is written in a language that I’ve just invented but which you most likely already know how to read, because, like me, in your mispent youth you read The Warlock of Firetop Mountain and similar choose-your-own-adventure books.

                                    PAGE 1

It is only the autumn night that makes you shiver, you tell yourself, but you
know better.

The castle stands on the hill in the gathering dusk. Wrapped in your grey cloak
against the cold, you crouch on the edge of the counterscarp. Eyes peer down
from the battlements, but you need not fear discovery: they are only the empty
sockets in the skulls of the adventurers who have come before you to harrow the
fortress of the wizard Gwydion.

Soon it will be dark enough to make your move.

You have boots.

You have a cloak.

You have a dagger.

To scale the wall, turn to page 27.

To swim the moat, turn to page 24.


                                    PAGE 2

The dagger makes no impression on the chain. “Ah, you wish to free me,” says
the monk. “You have a good heart, but no blade forged of iron can cut a chain
forged with magic.”

Suddenly he stands up. “Only Gwidion’s death can undo his magic.  But if you
wish to defeat him, take this.” He selects a tattered scroll from the shelf
above the desk and hands it to you.

Defeat Gwydion? You only meant to rob him. But you nod and take the scroll.

You have a tattered scroll.

To leave the monk and follow the stone passageway, turn to page 13.


                                    PAGE 3

You are in a long stone passageway.

To go through the doorway on the right, turn to page 33.

To follow the passageway, turn to page 13.


                                    PAGE 4

You push the door. It opens noiselessly on well-oiled hinges and you peer
through the gap. What you see inside takes your breath away. Golden coins in
heaps! Tapestries in silk and damask! Kingly crowns studded with polished
stones. This is what you came for.

To fill your pockets and make a run for it, turn to page 34.

To try the other door instead, turn to page 10.


                                    PAGE 5

To tag onto the back of the troop, turn to page 6.

To stay hidden until they pass, and then try your key in the door of the keep,
turn to page 40.


                                    PAGE 6

You follow the troop, imitating their shambolic walk as best you can. No one
seems to notice your presence. The door to the keep swings open, squealing on
its rusty hinges like a stuck pig, and you pass through one by one. The massive
door swings shut behind you.

If you have a cloak, turn to page 36.

Test your luck. If you succeed, turn to page 23.

Otherwise, turn to page 21.


                                    PAGE 7

The courtyard is quiet in the moonlight, but you remain on your guard. Who
knows what traps Gwydion has set for unwary intruders?

To approach the keep, turn to page 41.


                                    PAGE 8

Your boots echo loudly on the tiles. Too loud. You freeze, but the sound of
footsteps does not stop. You turn to see an armoured knight step emerging from
an alcove. Its visor is down, but somehow you doubt that there is a face behind
it.

        suit of armour (the, its) SKILL 10 STAMINA 10

If you win, turn to page 30.


                                    PAGE 9

It looks as if the goblin dropped a key in the struggle. You pick it up.

You gained a key.

To leave the scullery by the door, turn to page 7.


                                    PAGE 10

You push the door. It opens noiselessly on well-oiled hinges and you peer
through the gap. Inside is a study lined with tapestries, and at a desk sits a
tall white-haired man, bent in concentration over a grimoire. It is the wizard
Gwydion. His staff rests at his side.

If you have a tattered scroll, turn to page 47.

To attack Gwydion, turn to page 45.

To try the other door instead, turn to page 4.


                                    PAGE 11

You leap through the window.

Test your luck. If you succeed, turn to page 12.

Otherwise, turn to page 32.


                                    PAGE 12

It is a long fall, but the moat is deep. You pull yourself out and slink away
empty-handed into the night. Is that distant laughter you can hear? No
matter. You will be back.


                                    PAGE 13

The dark passageway enters a large hall lit by a candelabra. Suits of armour
stand silently in alcoves. The floor is tiled in a black and white checkerboard
pattern, and at the far side a wide staircase ascends. Something about the room
makes you suspicious.

To cross the hall, stepping only on the white tiles, turn to page 20.

To cross the hall, stepping only on the black tiles, turn to page 20.


                                    PAGE 14

You dart into the room and snatch the staff from the wizard’s side. He snaps
awake, and makes a grab for it, but you draw back from his reach.

“Very good, {player.name}!” he says. “But with or without my staff, I am still
the wizard Gwydion!”

To fight him, turn to page 29.

To run away, turn to page 25.


                                    PAGE 15

Two steps are all it takes to cover the distance, and then your dagger goes in
between his cervical vertebrae. The monk barely make a sound as he collapses
forward onto the desk, bleeding onto his half-copied page.

You search his cassock efficiently but find nothing. It is only then that you
notice the heavy iron chain from his leg to the desk. He was a prisoner here,
not an enemy.

To go back and follow the stone passageway, turn to page 13.


                                    PAGE 16

You place your foot on a loose cobble and it gives way. For a moment you hang
from your fingertips, but with lightning speed you find a new foothold. Soon
you swing up through a machicolation and onto the parapet walk.

To quickly descend the stairs to the courtyard, turn to page 37.


                                    PAGE 17

You take a ladleful and sip. Pfaughh! This isn’t soup, it’s laundry! There are
underclothes boiling here… and not overly clean ones.

A scraping from behind you makes you drop the ladle in alarm.  Someone is
coming through the door.

To try to sneak out past the newcomer, turn to page 22.

To stand your ground, turn to page 26.

To hide, turn to page 49.


                                    PAGE 18

You cough. The monk turns his head. “It’s nearly done,” he says, “I’m writing
as fast as I can. But with this light…” He turns back to his work. It is only
then you notice the heavy iron chain linking his leg to the desk. He is a
prisoner here!

To cut the chain with your dagger, turn to page 2.

To leave the monk to his fate and follow the stone passageway, turn to page 13.


                                    PAGE 19

You place your foot on a loose cobble and it gives way. For a moment you hang
from your fingertips, cursing the mason. And then you are gone.


                                    PAGE 20

If you have boots, turn to page 8.

Otherwise, turn to page 30.


                                    PAGE 21

You are still wet with water from your swim, and in the silence a drip echoes
in the stone passageway. The three shambling figures turn to look at you. You
wish that you had never seen the rotting flesh beneath their hoods.

          first zombie (the, its) SKILL 4 STAMINA 6

The second zombie is even more hideous than the first. It reaches for you with
the claws on its one good hand.

         second zombie (the, its) SKILL 5 STAMINA 6

The third zombie is more horrible than the first two put together.  It glares
at you with one eye dangling from its socket.

         third zombie (the, its) SKILL 6 STAMINA 6

If you win, turn to page 3.


                                    PAGE 22

Test your luck. If you succeed, turn to page 42.

Otherwise, turn to page 26.


                                    PAGE 23

The troop continues down a long stone passageway, but you scuttle through a
doorway on the right, glad to be rid of their unsettling company.

Turn to page 33.


                                    PAGE 24

You wrap your boots in your cloak and slip silently into the water. There is a
water gate, barred with iron, but the iron is rusted and crumbling below the
waterline. You take a deep breath and squeeze through into a dark underwater
passage.

Test your stamina. If you succeed, turn to page 35.

Otherwise, turn to page 46.


                                    PAGE 25

Clutching the wizard’s staff, you run out onto the landing.  Gwydion
follows. You run down the stairs. Gwydion follows. You run across the tiled
floor. Gwydion follows, his hobnailed boots ringing on the stone.

A suit of armour steps down from an alcove and swings its sword at Gwydion. He
shatters it with a blast of lightning, but a second suit of armour is at the
wizard’s back, swinging its sword. A third and a fourth step into the fray
until you can no longer see the wizard. Caught in his own trap!

Turn to page 50.


                                    PAGE 26

The newcomer is green-skinned and ugly as sin. He lunges at you with a
staff. You have no choice but to defend yourself.

            goblin (the, his) SKILL 7 STAMINA 6

If you win, turn to page 9.


                                    PAGE 27

The masonry has not been pointed in many years, and the stones are rough. You
climb swiftly and silently: an owl, perched in an arrow slit, is not disturbed
as you pass. But the wall is high.

Test your skill. If you succeed, turn to page 16.

Otherwise, turn to page 19.


                                    PAGE 28

You flourish your dagger, but Gwydion taps you with his staff and you find
yourself unable to move.

Turn to page 44.


                                    PAGE 29

Gwydion raises his hands, lightning flickering from his fingers.

            Gwydion (-, his) SKILL 10 STAMINA 12

If you win, turn to page 50.


                                    PAGE 30

You cross the hall and ascend the stairs. On the landing there are two
doors. One has a stuffed owl on the lintel, the other a lizard in a jar.

To enter the door with the stuffed owl, turn to page 4.

To enter the door with the lizard, turn to page 10.


                                    PAGE 31

This page intentionally left blank.


                                    PAGE 32

It is a long fall, and the ground is hard.


                                    PAGE 33

This is a scriptorium, with shelves stacked haphazardly with scrolls and
codices. By the light of a candle, a tonsured man sits at a writing desk,
copying a manuscript. His back is turned to you and he does not appear to have
heard him enter.

To slip back out again and follow the stone passageway, turn to page 13.

To kill the monk, turn to page 15.

To talk to the monk, turn to page 18.


                                    PAGE 34

Just one of the crowns would set you up for life. But you close your fingers on
it and it vanishes.

You hear a mocking laugh behind you, and whirl around to see the wizard Gwydion
standing there.

“Your head will make a fair adornment for my battlements, {player.name},” he
says.

If you have a tattered scroll, turn to page 48.

To jump out of the window, turn to page 11.

To attack Gwydion, turn to page 28.


                                    PAGE 35

You feel your way along the passage in the dark. Nothing but cold stone. Your
lungs are bursting: you must find a way out or drown.  With a last desperate
burst of energy you kick upwards and surface. Blessed air! You breathe it deep
into your lungs. You seem to have left your cloak and boots in the passage, but
at least you are alive.

You lost your boots.

You lost your cloak.

To haul yourself out of the water and look around, turn to page 38.


                                    PAGE 36

Are you scrutinized by unseen eyes as you pass the threshold? With your grey
cloak pulled over your head, you cannot tell.

Your luck went up by 1.

Turn to page 23.


                                    PAGE 37

The courtyard is quiet in the moonlight, but you remain on your guard. Who
knows what traps Gwydion has set for unwary intruders?

To enter a low stone building in a corner of the curtain wall, turn to page 38.

To approach the keep, turn to page 41.


                                    PAGE 38

By the flickering light of a fire, you can see that this is a scullery. Buckets
of dirty dishes stand by the water’s edge, and a giant cauldron bubbles over
the flame. There is a door in the west wall.

To drink from the cauldron, turn to page 17.

To leave the scullery by the door, turn to page 7.


                                    PAGE 39

You start to unroll the scroll, but Gwydion dashes it out of your hand with his
staff.

Turn to page 28.


                                    PAGE 40

The unsettling shambolic figures disappear into the keep and the door swings
shut behind them. You stay hidden for a quarter of an hour. Nothing moves. You
step silently across the drawbridge and approach the door.

The key grates in the lock but turns. You pull on the massive brass ring, and
the door opens, squealing on its rusty hinges like a stuck pig. You freeze, but
no one seems to have heard: or maybe they are used to the noise. You step
through and leave the door ajar.

Turn to page 3.


                                    PAGE 41

Massive blocks of well-fitted grey stone make up the walls of the keep: you can
see no way to scale the walls. Nor is there a moat with unguarded water gate:
just a ditch filled with chevaux de frise.

But what’s this? You crouch behind a pigsty as a troop of guards marches
towards the keep. Though maybe “marches” is the wrong word: the first one is
limping, the second has a hunched back, and the third drags his foot along the
ground.

If you have a key, turn to page 5.

This could be your only chance. To tag onto the back of the troop, turn to page
6.


                                    PAGE 42

You flatten yourself against the wall beside the door, and hold your breath as
the newcomer enters. He is green-skinned, ugly as sin, and carrying a staff. He
approaches the cauldron and gives it a stir, sniffing the fumes.

To sneak out while his back is turned, turn to page 7.


                                    PAGE 43

Quietly, so as not to disturb the wizard, you unroll the scroll and whisper the
words. Gradually the wizard’s head nods forward until his beard is on the desk
and you can hear him snoring.

To run in and stab him, turn to page 45.

To steal his staff, turn to page 14.


                                    PAGE 44

The wizard studies your face as you stand there paralyzed. “A most respectable
visage. I think it will look best on the western wall.”


                                    PAGE 45

Two swift steps and you plunge your dagger into his back, only for it to
shatter into pieces. Gwydion jumps up from the desk with staff in hand and taps
you with it. Suddenly you find yourself unable to move.

Turn to page 44.


                                    PAGE 46

You feel your way along the passage in the dark. Nothing but cold stone as far
as you can swim. Your lungs are bursting: you must turn back or drown. Back to
the water gate. Where is the gap? In panic you wrench at the bars, but it is
too late….


                                    PAGE 47

To read your scroll, turn to page 43.

To attack Gwydion, turn to page 45.

To try the other door instead, turn to page 4.


                                    PAGE 48

To read your scroll, turn to page 39.

To jump out of the window, turn to page 11.

To attack Gwydion, turn to page 28.


                                    PAGE 49

You crouch down behind the cauldron and watch as a goblin enters the laundry
room. He is green-skinned, ugly as sin, and carrying a staff. He approaches the
cauldron and gives it a stir, sniffing the fumes.

You leap up from your hiding place and push the goblin into the cauldron, where
he expires horribly in the boiling water and a tangle of dirty clothing.

Turn to page 9.


                                    PAGE 50

Gwydion is dead. His staff and his books are in your hands. All you have to do
now is find his gold, subdue his minions, free his slaves, and get the treasure
safely away. It should be easy now…

And here's an interpreter for this language, written in Python.

# -*- coding: utf-8 -*-

import re
import random
import sys
import textwrap

# For compatibility between Python 2 & 3.
if sys.hexversion > 0x03000000:
    raw_input = input

class GameOver(Exception):
    pass

class Roll(object):
    """The object `Roll('2d6+1)` represents a roll of 2 6-sided dice,
    plus 1. When converted to a string it describes the roll in
    English.

    """
    DICE_RE = re.compile(r'([1-9]\d*)d([1-9]\d*)([+-]\d+)?$')

    def __init__(self, dice):
        m = self.DICE_RE.match(dice)
        if not m: raise ValueError('bad dice description: {}'.format(dice))
        n, k, bonus = int(m.group(1)), int(m.group(2)), int(m.group(3) or '0')
        self.roll = [random.randint(1, k) for _ in range(n)]
        self.bonus = bonus
        self.total = sum(self.roll) + self.bonus

    def __str__(self):
        if len(self.roll) == 1:
            s = '{}'.format(self.roll[0])
        elif len(self.roll) == 2:
            s = '{} and {}'.format(*self.roll)
        else:
            s = '{}, and {}'.format(', '.join(self.roll[:-1]), self.roll[-1])
        if self.bonus:
            s += ', plus {}'.format(self.bonus)
        if len(self.roll) > 1 or self.bonus:
            s += ', making {}'.format(self.total)
        return s

class Player(object):
    STATS = [('skill', '1d6+6'), ('stamina', '1d6+6'), ('luck', '1d6+6')]

    def __init__(self, game):
        self.game = game
        self.name = self.game.input("What is your name? ")
        self.equipment = set()
        self.initial_stats = dict()
        self.w = lambda s: game.write(s, 4)

    def rollup(self):
        self.w("Welcome, {player.name}. Let’s roll up your character.")
        for stat, dice in self.STATS:
            r = Roll(dice)
            self.initial_stats[stat] = r.total
            setattr(self, stat, r.total)
            self.w("{}: you rolled {}.".format(stat.capitalize(), r))

    def test_stat(self, stat):
        w = self.w
        value = getattr(self, stat)
        w("Testing your {} ({}).".format(stat, value))
        r = Roll('2d6')
        if r.total > value:
            w("You rolled {}. You failed the test.".format(r))
            return False
        elif r.total == value:
            w("You rolled {}. You just made it.".format(r))
        else:
            w("You rolled {}. You passed the test.".format(r))
        if stat == 'luck':
            self.adjust_stat(stat, -1)
        return True

    def adjust_stat(self, stat, change, report = True):
        old_value = getattr(self, stat)
        new_value = max(0, min(self.initial_stats[stat], old_value + change))
        change = new_value - old_value
        setattr(self, stat, new_value)
        if report and change != 0:
            self.w("Your {} just went {} by {}. It is now {}."
                   .format(stat, 'down' if change < 0 else 'up',
                           abs(change), new_value))

class Paragraph(object):
    """`Paragraph(game, **kwargs)` represents a paragraph of instructions
    on a page. Call it to execute the instructions. The class property
    `re` is a regular expression matching all paragraphs of this type.
    The `kwargs` dictionary passed to the constructor must contain all
    the named groups in `re`.

    """
    def __init__(self, game, **kwargs):
        self.game = game
        self.w = lambda s: self.game.write(s, 4)
        for k, v in kwargs.items():
            setattr(self, k, v)

    group_re = re.compile(r'\(?P<(\w+)>')

    @classmethod
    def group(cls):
        """Return the name of the first named group in the `re` property."""
        return cls.group_re.search(cls.re).group(1)

class Fight(Paragraph):
    re = (r'(?P<monster_name>\S.*\S) \((?P<monster_article>the|an?|-),'
          r' (?P<monster_pronoun>their|his|her|its)\)'
          r' +SKILL +(?P<monster_skill>\d+) +STAMINA +(?P<monster_stamina>\d+)')

    def __call__(self):
        w = self.w
        player = self.game.player
        name = self.monster_name
        article = self.monster_article
        if self.monster_article == '-':
            fullname = name
        else:
            fullname = '{} {}'.format(self.monster_article, name)
        pronoun = self.monster_pronoun
        skill = int(self.monster_skill)
        stamina = int(self.monster_stamina)
        w('{0.name}: SKILL {0.skill} STAMINA {0.stamina}'.format(player))
        w('{}: SKILL {} STAMINA {}'.format(name.capitalize(), skill, stamina))
        while True:
            player_roll = Roll('2d6+{}'.format(player.skill))
            w("You rolled {}.".format(player_roll))
            monster_roll = Roll('2d6+{}'.format(skill))
            w("{} rolled {}.".format(fullname.capitalize(), monster_roll))
            if monster_roll.total == player_roll.total:
                w("You skirmish without damage on either side.")
                continue
            luck_adjustment = 0
            if self.game.input("Test your luck? ").lower() in ('y', 'yes'):
                luck_adjustment = 1 if player.test_stat('luck') else -1
            if monster_roll.total > player_roll.total:
                player.adjust_stat('stamina', -2 + luck_adjustment, False)
                w("{} hits you! Your stamina is now {}."
                  .format(fullname.capitalize(), player.stamina))
                if player.stamina <= 0:
                    w("You die.")
                    raise GameOver()
            else:
                stamina = max(0, stamina - (2 ** (luck_adjustment + 1)))
                w("You hit {}! {} stamina is now {}."
                  .format(fullname, pronoun.capitalize(), stamina))
                if stamina <= 0:
                    w("You killed {}.".format(fullname))
                    return

class NextPage(Paragraph):
    re = r'(?:If you win, t|Otherwise, t|T)urn to page (?P<next_page>\S+)\.'

    def __call__(self):
        return self.next_page

class TestEquipment(Paragraph):
    re = (r'If you have(?: (?:an?|your|the|some))? (?P<equip_test>\S.*\S),'
          r' turn to page (?P<equip_page>\S+)\.')

    def __call__(self):
        if self.equip_test in self.game.player.equipment:
            return self.equip_page

class TestStat(Paragraph):
    re = (r'Test your (?P<test_stat>\S+)\. If you succeed,'
          r' turn to page (?P<test_page>\S+)\.')

    def __call__(self):
        if self.game.player.test_stat(self.test_stat):
            return self.test_page

class ChangeStat(Paragraph):
    re = (r'Your (?P<stat_change>\S+) went (?P<stat_dir>up|down) by'
          r' (?P<stat_amount>\d+)\.')

    def __call__(self):
        amount = int(self.stat_amount) * (-1 if self.stat_dir == 'down' else 1)
        self.game.player.adjust_stat(self.stat_change, amount)

class GainEquipment(Paragraph):
    re = (r'You (?:gained|have)(?: (?:an?|your|the|some))?'
          r' (?P<gain_equipment>\S.*\S)\.')

    def __call__(self):
        self.game.player.equipment.add(self.gain_equipment)

class LoseEquipment(Paragraph):
    re = r'You lost(?: (?:an?|your|the|some))? (?P<lose_equipment>\S.*\S)\.'

    def __call__(self):
        self.game.player.equipment.remove(self.lose_equipment)

class Choice(Paragraph):
    re = r'(?P<choice>\S.*\S), turn to page (?P<choice_page>\S+)\.'

    def __call__(self):
        n_choices = len(self.choices)
        if n_choices == 1:
            self.w("{}, press return.".format(self.choices[0]['choice']))
        else:
            for i, c in enumerate(self.choices):
                self.w("{}, choose {}.".format(c['choice'], i + 1))
        prompt = "? "
        while True:
            inp = self.game.input(prompt)
            if n_choices == 1:
                return self.choices[0]['choice_page']
            try:
                choice = int(inp)
                if 1 <= choice and choice <= n_choices:
                    return self.choices[choice - 1]['choice_page']
            except ValueError:
                pass
            prompt = "Please enter a choice from 1 to {}> ".format(n_choices)

class Description(Paragraph):
    re = r'(?P<description>\S.*\S)'

    def __call__(self):
        self.game.write(self.description)

class Page(object):
    def __init__(self, game):
        self.game = game
        self.paras = []
        self.choice = None

    def add(self, cls, **kwargs):
        if cls == Choice:
            if not self.choice:
                self.choice = Choice(self.game, choices = [])
                self.paras.append(self.choice)
            self.choice.choices.append(kwargs)
        else:
            self.paras.append(cls(self.game, **kwargs))

    def visit(self):
        for para in self.paras:
            page = para()
            if page:
                return page
        raise GameOver()

class Game(object):
    width = 72
    para_re = re.compile(
        r'^(?:PAGE (?P<page>\S+)|{})$'
        .format('|'.join(cls.re for cls in Paragraph.__subclasses__())))

    def __init__(self, text):
        self.pages = dict()
        page = None
        for para in re.split(r'\n(?:[ \t]*\n)+', text.strip()):
            para = textwrap.dedent(para).strip().replace('\n', ' ')
            m = self.para_re.match(para).groupdict()
            if m['page']:
                if page is None: self.initial_page = m['page']
                page = self.pages[m['page']] = Page(self)
                continue
            for cls in Paragraph.__subclasses__():
                if m[cls.group()]:
                    page.add(cls, **m)
                    break

    def input(self, prompt):
        answer = raw_input(prompt)
        print('')
        if answer.lower() in ('q', 'quit'):
            raise GameOver()
        return answer

    def write(self, para, left_margin = 0):
        indent = ' ' * left_margin
        for line in textwrap.wrap(para.format(player = self.player),
                                  self.width, initial_indent = indent,
                                  subsequent_indent = indent):
            print(line)
        print('')

    def centre(self, text):
        self.write(text, (self.width - len(text)) // 2)

    def play(self):
        self.player = Player(self)
        self.player.rollup()
        self.centre('*')
        page = self.initial_page
        try:
            while True:
                page = self.pages[page].visit()
        except GameOver:
            pass

if __name__ == '__main__':
    Game(open(sys.argv[1], encoding='utf-8').read()).play()

^ Thus SymPy would be a better choice if you needed to manipulate polynomial expressions in Python.

Emacs/Perforce integration: back from the dead

Wed, 01 May 2013 00:00:00 GMT

p4.el, the Emacs/Perforce Integration, provides an interface to the Perforce software version management system from the Emacs text editor. Written originally by Eric Promislow, with important contributions from Rajesh Vaidheeswarran, Peter Österlund, and many others, it’s a piece of software that I have used more or less every day for the last thirteen years.

However, the SourceForge repository hasn’t seen any development since 2005¹, and it has been looking a bit ragged around the edges for a few years now. Perforce has acquired many useful new features in the last eight years: annotate -i in 2005.2, move in 2009.1, shelve and unshelve in 2009.2, grep in 2010.1, annotate -I and copy in 2010.2, sync -s and streams in 2011.1, and status, reconcile and populate in 2012.1. But more importantly, we’re now making much more serious use of the Internet for software development than we were ten years ago. When p4.el was first written back in 1996 or so, it was reasonable to assume that the Perforce server was on a local network, and therefore that it would be fine to perform frequent operations, like checking to see if a visited file is under the control of Perforce, synchronously with respect to the Emacs user interface. Sure, Emacs will hang if the Perforce server is unavailable (at least until the network connection times out after thirty seconds or so), but how often does that happen? The Perforce server is very reliable, and hardly ever goes down.

But nowadays, your Perforce server is on the other side of the world, and you’re accessing it via a VPN from your laptop on the train over a mobile data connection that drops every time the train goes into a tunnel. Even though these circumstances are admittedly somewhat trying for networking applications, it’s still completely unacceptable for Emacs to hang for thirty seconds every time you open a file.²

So finally I lost patience with this state of affairs, and I’ve forked p4.el on github, and I’m gradually fixing it. At this stage I think it’s too unstable to recommend for everyone, but if you’re a p4.el user who’s frustrated with the hangs and missing features, and you’re expert enough to cope with it being a bit buggy, then I’d be grateful if you could download a copy, try it out, and report problems (or better still, fork the repository and send me pull requests for your bug fixes). I use GNU Emacs 24.3 on OS X, Windows 7, FreeBSD and Ubuntu, but I don’t propose to put any work into ensuring that p4.el runs on any other versions of Emacs or any other operating systems. In particular, I’m likely to break XEmacs compatibility unless someone keeps me honest.

★

I’ve made many improvements to the code, but the most important are as follows:

You no longer lose unsaved changes in a buffer when you run the p4-edit command. Instead, you are prompted as to whether you want to revert the buffer.

A warning about this bug was right there near the top of p4.el for at least the last ten years:
```
;; WARNING:
;; --------
;;
;;    % p4 edit foo.c
;;    ... make changes to foo.c in emacs
;;    % p4 submit
;;     ... keep the writable copy of foo.c in emacs.  Start making changes
;;     to it.  Discover that you can't save it.  If you do M-x:p4-edit,
;;     you'll lose your changes.  You need to do a 'p4 edit' at the
;;     command-line.
```
This was a horrible misfeature, because in situations where you have the “writable on client” flag turned on your Perforce client workspace, it’s completely natural to start editing the file, and only later remember that you should have opened it for edit. If you forgot to save before running p4-edit, then you lost your work.
If you are logged out of Perforce, then nearly all³ Emacs commands that run a Perforce command will prompt you to log in, and then automatically re-run the Perforce command that failed.
The “mode check”—that is, the check, when you open a file, to see whether that file is under control of Perforce, so that the mode line can be updated, and menu options can be enabled or disabled—now runs quietly (that is, it never bothers you, even if there are errors) and asynchronously with respect to the Emacs user interface. This means that if you are offline, or the Perforce server is otherwise unavailable, then Emacs no longer hangs each time you open a file, waiting for a network timeout to the Perforce server. And if you are logged out, then p4.el does not bother you for your Perforce password just for opening a file.

(There is a user option, p4-do-find-file, that disables the mode check. I've left this user option in place because if you don't use the menu, or don't care about the mode line and the menu options being correctly updated, then you can avoid these queries to the Perforce server by disabling the option.)
Many more user commands now run Perforce in the background.⁴
When you revert a file with changes, you get shown the diffs that you are about to revert. (I got this excellent idea from the vc-revert command.)
When you submit a changelist that includes a file that is open for edit but with no changes, p4.el prompts you “File with empty diff opened for edit. Submit anyway?” The file or files in question are now shown to you in a buffer, to avoid the awkward pause where you have to go away and try to figure out which file or files is being complained about. (This feature also now uses p4 diff -sr instead of plain p4 diff, so the check for empty diffs is much faster in situations when you have made many changes.)
Errors from Perforce commands are shown to the user more reliably. There should be no need any more to switch to the *P4* buffer to try to figure out why your command didn't work.
The p4-blame command now uses p4 annotate -i -c if the server supports this, instead of working out the annotations by starting with the base revision, and patching it with each diff. It also caches the annotation strings instead of rebuilding them for each line. This makes it much faster on files with many revisions. For users on older servers the annotate-by-patching code still works, and now there are now progress messages to reassure you that the operation has not hung.

^ Fujii Hironori worked on it in 2009, adding major modes for the output buffers and modernizing the syntax highlighting and menus.
^ Sure, you can turn off the user option p4-do-find-file when you’re on the train and turn it back on again when you get back to the office. Or you can type C-g to abort the operation and then revisit the file. But life would be so much better if things like this worked smoothly out of the box without you having to nurse them along like this.
^ The two outstanding exceptions are committing a form and p4-resolve.
^ Caveat: I’m not entirely certain if it’s wise for all user operations to run in the background. Initially I thought it was nice to be able to issue a big p4-describe or whatever and then immediately get on with some other work while the server chugged away processing my query. But then I found myself worrying about whether in fact it had succeeded, and going off to check the relevant buffer to see if it had failed for some reason. This despite having programmed it to always show me error messages, if there are any. So if you try out the new code, let me know what you think about this. Maybe there needs to be some kind of dashboard, if only to reassure you that nothing’s gone wrong?

Bogus foreign keys in Django

Sun, 24 Mar 2013 00:00:00 GMT

This is a problem that had me puzzled for a while, but has a really simple solution, so I’m writing it up here in case it might be useful to other developers who find themselves in this slightly bizarre situation.

1. The background

There’s quite a lot of background to get through before I can introduce the problem. Imagine that you are developing a web site using the Django framework. The underlying database has been in production for some years, so that the ideal mode of operation (where you describe your data model to Django and then it creates the appropriate database tables) is not available. Instead you have to reverse-engineer the database and translate it into a form that Django understands. So you have a look at the database and it looks like this:¹

mysql> select * from user limit 3;
+--------+------------------+----------------+
| userid | name             | (other fields) |
+--------+------------------+----------------+
|      1 | Alarico De Luca  | ...            |
|      2 | Blanka Zielinska | ...            |
|      3 | Cristin Åberg    | ...            |
+--------+------------------+----------------+
3 rows in set (0.00 sec)

mysql> select * from task limit 3;
+--------+--------+----------------+
| taskid | userid | (other fields) |
+--------+--------+----------------+
|      1 | 3      | ...            |
|      2 | 1      | ...            |
|      3 | 2      | ...            |
+--------+--------+----------------+
3 rows in set (0.00 sec)

So there are tasks and users, and each task is assigned to a user. So let’s translate that into a couple of Django models:

from django.db import models

class User(models.Model):
    id = models.IntegerField(primary_key=True, db_column='userid')
    name = models.CharField()

class Task(models.Model):
    id = models.IntegerField(primary_key=True, db_column='taskid')
    user = models.ForeignKey(User, null=True, db_column='userid')

This all looks good from the Django shell:

>>> from myapp.models import *
>>> User.objects.values('name')[0]
{'name': u'Alarico De Luca'}
>>> Task.objects.get(id = 1).user.name
u'Cristin Åberg'

So you write a template to present some query results in a table:

<table>
  <thead>
    <tr>
      <td>#</td>
      <td>Assigned to</td>
    </tr>
  </thead>
  <tbody>
    {% for task in tasks %}
      <tr>
        <td>{{ task.id }}</td>
        <td>{{ task.user.name }}</td>
      </tr>
    {% endfor %}
  </tbody>
</table>

But when you run some sample queries, the Django server falls over with the following error while expanding {{ task.user.name }} in the template:

Traceback (most recent call last):
  [...]
  File "[...]/django/db/models/fields/__init__.py", line 537, in get_prep_value
    return int(value)
ValueError: invalid literal for int() with base 10: 'John Smith'

What’s going on here? You dig into the backtrace and discover that the offending task is number 123.

mysql> select * from task where taskid=123;
+--------+------------+----------------+
| taskid | userid     | (other fields) |
+--------+------------+----------------+
|    123 | John Smith | ...            |
+--------+------------+----------------+
1 row in set (0.00 sec)

This explains why Django is trying to convert a string into an integer when trying to get the user for this task. The userid field is a string field! This was in fact deducible from the way the numbers in this field were left-aligned in the output of the query select * from task, but that was easy to miss. Or it would have been obvious if you had done the sensible thing and looked at the table schema as well as the first few rows:

mysql> describe task;
+----------+--------------+------+-----+---------+----------------+
| Field    | Type         | Null | Key | Default | Extra          |
+----------+--------------+------+-----+---------+----------------+
| taskid   | int(11)      | NO   | PRI | NULL    | auto_increment |
| userid   | varchar(100) | NO   |     | NULL    |                |
| ...      | ...          | ...  | ... | ...     | ...            |
+----------+--------------+------+-----+---------+----------------+
11 rows in set (0.38 sec)

You can guess how this might have happened: perhaps originally there was no user table: instead, the user’s name was written directly into a field in the task table. Later, the user table was added, and instead of adding another field to the task table containing a foreign key on the user table, the foreign key was placed in the existing userid field. MySQL is quite happy to join a string field against an integer field (performing an implicit conversion), so it would have been easy for an inexpert or hurried database administrator to make the change this way. Now of course, with several applications deployed against this database, it is not worth the work to fix the schema.

2. The problem

Right, so now you know that userid is a string field, but how do you fix the Django model? In particular, if you make this field a models.CharField, how do you get the associated user? Here’s a sequence of plans that you might try:

Leave the field as a models.ForeignKey field, but add a user() method that catches the ValueError:

class Task(models.Model):
    id = models.IntegerField(primary_key=True, db_column='taskid')
    userid = models.ForeignKey(User, null=True)

    def user(self):
        try:
            return self.userid
        except ValueError:
            return None

This works, but how do you get at the user name in the exceptional cases where it’s the name (rather than the foreign key) that’s stored in the record’s userid field?

Looking at the implementation of foreign key fields in Django (see django.db.models.fields.related.py), it seems that you can get the actual value of a foreign key field F (rather than the related object) like this:

getattr(instance, type(instance).F.field.attname)

So you could add a user_name() method like this:

class Task(models.Model):
    id = models.IntegerField(primary_key=True, db_column='taskid')
    userid = models.ForeignKey(User, null=True)

    def user(self):
        try:
            return self.userid
        except ValueError:
            return None

    def user_name(self):
        try:
            return self.userid.name
        except ValueError:
            return getattr(self, Task.userid.field.attname)

The trouble with this is that field.attname isn’t documented, so it’s not supported. This area of the Django implementation changed significantly between 1.3 and 1.5 (though without breaking field.attname, as it happens). So this is risky: you would prefer to have a solution that is supported.

Represent the userid field in Django as a models.CharField (so that you can get the name in a supported way), and look up the related object in Python:

class Task(models.Model):
    id = models.IntegerField(primary_key=True, db_column='taskid')
    userid = models.CharField()

    def user(self):
        try:
            return User.objects.get(id = int(self.userid))
        except ValueError:
            return None
        except User.DoesNotExist:
            return None

    def user_name(self):
        try:
            return self.user().name
        except AttributeError:
            return self.userid

This is a non-starter: it means a whole extra database query each time a user name is looked up, and kills performance. So on to:

Fetch all the users from the database, store them using Django’s cache framework, and use the cache to look up the user as needed?

from django.core.cache import cache

class Task(models.Model):
    id = models.IntegerField(primary_key=True, db_column='taskid')
    userid = models.CharField()

    def user(self):
        users = cache.get('users')
        if users is None:
            users = {user.id: user for user in User.objects.all()}
            cache.set('users', users)
        try:
            return users[int(self.userid)]
        except ValueError:
            return None
        except KeyError:
            return None

    def user_name(self):
        try:
            return self.user().name
        except AttributeError:
            return self.userid

However, (a) there are so many users that filling the cache takes tens of seconds which means that the first person to query the site after a reboot is going to suffer an unacceptable delay; and (b) the cache is going to get out of date and need refreshing. Solving both of these is a Simple Matter of Programming, but this is looking over-complicated and error prone. Plan B would surely be better than this!

3. The solution

You can have two fields in a Django model that refer to the same database column, but with different field types!

class Task(models.Model):
    id = models.IntegerField(primary_key=True, db_column='taskid')
    user = models.ForeignKey(User, null=True,² db_column='userid')
    userid = models.CharField()

    def user_name(self):
        try:
            return self.user.name
        except ValueError:
            return self.userid

Obvious when you know how…

^ Made up example.
^ One of the effects of setting null=True for a foreign key is that the select_related() method “by default, does not follow foreign keys that have null=True”. You need to request the field explicitly: in this case, by calling select_related('user').

CTC ride to Buntingford

Sun, 10 Mar 2013 00:00:00 GMT

I looked out of the window this morning to see snow flakes falling from a dark cloudy sky. It was chilly out: just barely above freezing, and there was a cold north-easterly wind. Just the kind of day on which it would have been nice to sit in a warm house and drink tea. If only I hadn’t promised to the lead day ride.

The first challenge was to get to Brookside: it was the day of the Cambridge half-marathon, and the city centre was wrapped in orange tape and surrounded with race marshals. I made a detour around the blockage via Grange Road and Barton Road, and at Brookside I found three riders: Tom, Neil, and Conrad. A pretty good turnout given the conditions.

My plan for the day was to ride directly to coffee, getting there a bit early, and then to have a longer detour between coffee and lunch. It worked out pretty well: the north-easterly wind blew us quickly down the B1368 and up the hill to Barkway, where we turned west towards Reed. The snow had been forecast to stop, but as we climbed up into the Hertfordshire hills, it got heavier, and the north-facing fields were covered with white. Never has arriving at the Silver Ball Café seemed so welcome!

Near Wood End, Hertfordshire.

Neil had to turn home, so there were only three of us as we set out into the narrow lanes west of the A10, with the snow still swirling around us. The roads were well above freezing, so the snow was melting rather settling, but many of the minor lanes were in quite shocking condition: mud and potholes and puddles everywhere. I had washed my bike on Saturday, but now it was utterly filthy again, as were my feet and legs. On the road from Ardeley to Great Munden, Conrad punctured: a sharp little flint had penetrated the tyre (he was sensibly riding his winter bike).

Fixing a puncture, near Wood End, Hertfordshire.

Due to this enforced stop, we were a bit late getting to lunch. Since we were passing through Westmill, We looked in at the Westmill Tea Room (in the old post office building) to see if they had a table for us, but they were booked solid. So we rode on a couple of miles to Buntingford and stopped at the excellent Buntingford Coffee Shop. The warmth of the café was as welcome as the toasted sandwich: I hadn’t been able to feel my toes for some time!

When we left Buntingford at about 14:15 the snow had stopped, but now we were riding directly into the headwind. On the tops of the hills it was pretty tough making progress. We took a direct route through Wyddial, Barkway, Great Chishill, Heydon, Elmdon and over the hill to Ickleton, where we met the afternoon riders. A good day’s ride despite the appalling weather: about 67 miles.

View CTC ride to Reed, Buntingford and Ickleton in a larger map.

Great North Road

Tue, 15 Jan 2013 00:00:00 GMT

The very first sentence of Peter F. Hamilton’s Great North Road (2012) reads:

As midnight approached, the wild neon colours of the borealis storm came shimmering through the soft snow falling gently across Newcastle upon Tyne.

How is that possible? Auroras take place high in the sky, in the thermosphere (above 80 km), but if it’s snowing then it must be cloudy in the lower troposphere (stratus clouds have a base around 2 km), so it would seem very unlikely that you could see an aurora “through snow”. Some more explanation is needed, and it’s not provided. So is this just an unlucky slip, or an example of systematic failure to think through the consequences of the settings and events in the novel? Let’s read on and find out.¹

★

The background to this novel, which includes interplanetary and interstellar travel, is rather derivative of media franchises like Stargate, Predator, and Aliens.² But the cyberpunkish police-procedural scenes in a mid-22nd century Newcastle are quite different in tone from the space operatic scenes, and this down-to-earth milieu reveals the presence of shackles on the imagination: some of the chapters portray limitless technological wish-fulfilment, but then other chapters portray minor aspects of early 21st century middle-class life in the UK as immune from change. This is a world in which there are commercial fusion plants everywhere, but people are still dependent on petrol-powered motor vehicles. A world in which “lightwave ships” cross the solar system at close to the speed of light, but the fastest way to cross Europe is by jet plane. A world in which there is interstellar teleportation, but no-one has managed to solve the problem of how children can get to school when both parents have jobs:

“You’ll have to take them to school for me,” Sid announced quickly, hoping it would get overlooked in the general morning chaos.

“No bloody way!” Jacinta exclaimed.³ “We agreed. I’ve got a full cardio replacement booked for this morning.”

A world in which “phase-change materials” act as near-perfect heat stores but the protagonist is unaware that houses can be retrofitted with insulation:

A pleasant enough home, but its age meant it was never designed for the cold of today’s winters, so it cost a fortune to heat.

A world in which it is established that detective Sid has a computer embedded in his skin, that can play images directly into his retina and sounds directly into his ears, and yet:

The alarm clock’s sharp buzz dragged Sid awake. He groaned and reached out for the snooze button. […] Sid picked up the clock and held it close to his face—the only way he could make out the glowing green figures.

The novel is set in 2143,⁴ in an alternate and more technologically advanced universe (in which human cloning was carried out in 2007), but in the Newcastle sections of the story, there is little evidence for this beyond references to colder winters and hotter summers. For example, a murder victim is found in the water by the Gateshead Millennium Bridge. Now, sea level rise could be as much as 2 m by the beginning of the 22nd century⁵, and that would surely put the embankments around the Tyne under water, at least at some states of the tide.⁶ So this scene would be an easy opportunity to put in a bit of futuristic backdrop, but the text gives absolutely no indication that anything has changed.

The novel presents little evidence of social change having occurred since 2012. Young people wear the same kinds of clothes to go out to the same kinds of nightclubs. Toyota still make reliable mid-price motor vehicles. Car parking remains a problem:

Sid left for the city morgue, which was housed in a modern annex next to the glass and steel towers of Arevalo Medical’s Royal Victoria Infirmary. As he drew into the car park next to the city morgue block Sid saw notices proclaiming that parking would be suspended for two months so footings could be sunk for the new oncology clinic. “So where do we park?” he muttered to himself.

In this scene Sid drove to the morgue from the police station at the junction of Pilgrim Street and Market Street, a distance of about a kilometre. He learns nothing there that couldn’t have been communicated electronically, but even if he had to go in person, why didn’t he walk? Is this passage supposed to establish his laziness and selfishness?

Selfishness is a bit of a theme of Great North Road. The majority of the characters in the novel work for government agencies such as the police and the military, but that doesn’t stop them engaging in boilerplate libertarian moans about the dead hand of government bureaucracy and the burden of the taxes that pay their salaries. At the same time, the corporate world is presented as a Hobbesian war of all against all, in which everyone is corrupt, and one of the rewards for the winners is the opportunity to sexually exploit the losers. In fact, the novel is so devoid of sympathetic characters that I felt a bit disappointed that the alien monster failed to kill everyone.

^ I was given this book for Christmas, and I have to say that having got used to e-books it felt really awkward to lug about a 1,100-page tome that’s nearly as big as my head. I felt a bit embarassed reading it on the Tube.
^ There’s a scene early on where ~~Ripley~~ Angela, the only survivor of an alien attack, is persuaded to return to the planet in search of the alien monster. Escorted by a gung-ho team of ~~Colonial Marines~~ HDA Legionnaires, she warns them that their confidence is unwarranted: “You’re on a trip to poke a stick into a monster’s nest. ~~It'll breed. You'll die. Everyone in the fucking company. Will die.~~ It will kill you. It will kill all of you. You won’t stand a chance.”
^ If you guessed from this passage that Hamilton’s writing suffers from a bit of a case of “said-bookism” then you’d be right.
^ But two references to the action taking place in the “twenty-third century” suggest that an earlier draft had placed the novel somewhat farther in the future.
^ See Global Sea Level Rise Scenarios for the United States National Climate Assessment, especially figure 10, reproduced below. The novel’s “cheap oil” scenario is surely going to push things towards the high end of the range of predictions.
^ Researching this, I was surprised to learn that the Tyne already floods its embankments under rare conditions: for example, this photo was taken on 2011-11-27 when a storm surge in the North Sea coincided with a spring tide. These events are only going to become more frequent.

The last lousy bug

Sun, 06 Jan 2013 00:00:00 GMT

Inspired by David Jones’ “My Favourite Bugs”, here’s one of my own war stories.

Console video games are demanding pieces of software to write: in addition to being entertaining, they must be playable by six-year-olds and self-explanatory to adults. They must be able to gracefully handle corrupted save files. They must never run out of memory, be unresponsive to user input, or take longer than 16 ms to draw the next frame. Oh, and you typically only get one chance to get it right: when the game has to go out to manufacture on cartridge or DVD there’s no chance to fix things with a version 1.1.

But there are compensations. The ratio of testers to programmers is high, and nothing gives you confidence in the quality of your work as much as knowing that thirty people have been trying to make it break for the last week and they have only found one way to make it crash.

In the early days of testing, the bugs pile up rapidly, and it’s a scramble to keep the number of open reports down to a manageable level, but enough of the bugs are easy to fix that you can do it. The most exhilarating experience is when you’re working with testers who are in a time zone halfway around the world. You arrive at work in the morning to find a pile of bugs has arrived overnight, you work hard all day, and ship out a new release in the early evening, just in time for the testers to start verifying your fixes and finding new bugs.

After this has been going on for a few weeks, the low-hanging fruit have been plucked, and a handful of serious bugs have bubbled to the surface. These are the bugs that you haven’t been able to reproduce: the intermittent errors and mysterious crashes. And then eventually you come down to the last lousy bug, the one that you’ve been putting off in the hope that it would go away by itself, and you have to roll up your sleeves and get your hands dirty.

Bug 142 was especially mysterious because it happened after the program quit running. On this particular console, returning from the game to main menu was implemented by rebooting: a small region of memory survives the reboot and contains instructions for the boot loader telling it which application to run next. (This might sound strange, but it makes a lot of sense from a robustness point of view: applications aren’t required to co-exist, can’t cause each other to crash, and always start from a known machine configuration.) So when the player requests a quit, the game saves its state, and calls the Reboot function, which writes the necessary instructions to the protected region of memory, and then reboots the console. Except that our testers were reporting that on rare occasions our game was quitting but then crashing in the boot loader (which reported that it had been passed a bogus argument).

I always try to bear in mind Richard Stallman’s advice from Debugging with GDB:

Often people who encounter a bug spend a lot of time investigating which changes to the input ﬁle will make the bug go away and which changes will not affect it. [That is, trying to find the “envelope” of the bug.—GDR]

This is often time consuming and not very useful, because the way we will ﬁnd the bug is by running a single example under the debugger with breakpoints, not by pure deduction from a series of examples. We recommend that you save your time for something else.

The trouble was that I couldn’t get the bug to manifest itself in the debugger. The testers were running the game on consumer hardware, so there was no way to attach a debugger when they provoked the bug. And whenever I stepped through the relevant sequence of events the correct arguments were always passed to the boot loader. So all I had to go on was the envelope of the bug, and the only salient fact about the envelope was that the bug only manifested itself in the RELEASE build of the game (with all debugging information and assertions stripped); never in the DEBUG build. (Which should have suggested that the problem might have something to do with memory layout, though I didn’t realize this until later.) I checked the documentation for the shutdown sequence and everything looked OK. We faded the screen to black, waited for the graphics pipeline to empty, faded out the music, waited for any pending save operation to finish, shut down any libraries that required it, and finally called Reboot.

So all I could do was to prepare a special RELEASE build with extra diagnostic support, and work on the other bugs, hoping that eventually something would turn up. But the pile of outstanding bugs grew smaller and smaller, and eventually it was clear that bug 142 would be the last lousy bug in the game.

The testers kept at it, using the special diagnostic build, and eventually one of them struck gold by getting our game to crash before it quit (rather than after sending bogus arguments to the boot loader), which caused our fatal error handler to run, which drew the stack trace you can see in the screenshot at the right. Knowing what I now know about the cause of the bug, the window of opportunity for this crash was very small, and I dread to think about how many times they had to provoke the bug before they got this result. Testers, I salute your dedication!

On the PowerPC, a DSI (Data Storage Interrupt) exception “occurs when a data memory access cannot be performed”—it’s the PowerPC equivalent of a segmentation fault or memory protection failure. Consulting the application map, it was possible to decode the stack trace as follows (everything from main up to menuTick is our code, and then Reboot and above are library code):

PC	function
`800E9D5C`	`strlen`
`800C8AF0`	`PrepareBootArgs`
`800C8DE8`	`Reboot`
`80097B54`	`menuTick`
`8009FD90`	`mainTick`
`800A0518`	`main`
`800041A4`	`_init`

We didn’t have the source code for the library functions, but we did have a debugger, so armed with the stack trace we spent a morning stepping through the sequence of events, reading the assembler, and reverse-engineering the library. It became clear that Reboot carried out the following steps:

Allocate a 4 KiB buffer starting at 0x81280000.
Use this buffer to assemble the command-line arguments for the boot loader.
Copy these command-line arguments to the protected area of memory that will survive the reboot. (This is where strlen gets called, and where the DSI exception sometimes occurred.)
Reboot.

The writers of Reboot didn’t worry about whether the memory at 0x81280000 would be in use by the game, because the function never returns. And if it never returns, it can’t possibly conflict with other uses of the memory. Or can it? We looked at our memory map, and in the RELEASE build 0x81280000 was right in the middle of one of our external frame buffers.

(For readers unfamiliar with video game programming: an external frame buffer is a region of memory which contains a single video frame in a format suitable for sending to the display. Typically you have two external frame buffers: one is being displayed and the other is receiving (or waiting to receive) a copy of the next frame from the GPU. The GPU has direct access to main memory, so the copy happens asynchronously with respect to the CPU. When the next frame has been copied, the game waits for a safe moment (known as a ‘retrace’ or ‘vsync’ after the vertical retrace or synchronization on analog televisions) and then swaps the roles of the two external frame buffers, so that the second is now displayed and the first is now waiting for a copy of the next frame.)

So even though we had programmed a wait for the graphics pipeline to empty before shutting down the game, there might still be a frame being worked on by the GPU, and if the timing were just right, then it would copy its last frame to the external frame buffer while Reboot was assembling its command line in that same region of memory, corrupting it. (This would also explain why the crash only happened in the RELEASE build: in the DEBUG build the memory layout was different and the external framebuffers did not overlap the critical region.)

It was easy to test this theory in the debugger: by putting a breakpoint in different places in Reboot we could arrange for the copied framebuffer to cause both of the observed effects (the crash in strlen and the corrupted arguments to the boot loader), and several other kinds of crash and corruption.

So after all that, it was completely trivial to fix the bug. All we had to do was wait for the GPU to finish copying out the last frame (not just for the pipeline to clear) before calling Reboot.

The Hydrogen Sonata

Wed, 02 Jan 2013 00:00:00 GMT

The Hydrogen Sonata is the ninth novel in the Culture series by Iain M. Banks. It is set in the civilization of the Gzilt, which is on the verge of ‘subliming’—that is, departing the physical universe to become immortal beings of pure energy. As the big day approaches, one question still niggles: what was the origin of the Gzilt’s suspiciously well-informed ‘Book of Truth’? Genuine religious inspiration or hoax by ancient meddling aliens? The only person who might know the answer is T. C. Vilabier,¹ composer of the famously unplayable sonata of the title,² and the only person who might be able to find him and persuade him to dish up the dirt is elevenstring player Vyr Cossont. And so a bunch of well-meaning observers from the Culture decide to set a quest in motion whose outcome may cause the Gzilt to think again about the whole sublimation project.

‘Subliming’ started as a throw-away idea serving a narrative function in Banks’ 1987 novel Consider Phlebas. The idea that powerful species eventually mostly ‘retired’ from involvement in the material universe provided a quick explanation as to how relatively young civilisations like the Culture and the Idirans could be significant players in a galaxy that’s billions of years old:

Now it was obvious why the Dra’Azon had made Schar’s World one of their Planets of the Dead. If you were a pure-energy superspecies long retired from the normal, matter-based life of the galaxy, and your conceit was to cordon off and preserve the odd planet or two you thought might serve as a fitting monument to death and futility, Schar’s World with its short and sordid history sounded like the sort of place you’d put pretty near the top of your list.

The witty name ‘subliming’ for this phenomenon first appeared in Excession (1996), which added some details about the results of the process:

Sublimed Elders, become as gods to all intents and purposes, seemed to be derelict in the duties which the more naive and less developed societies they left behind ascribed to such entities. With certain very limited exceptions, the Elder species subsequently took almost nothing to do with the rest of life in the galaxy whose physical trappings they invariably left behind; tyrants went unchecked, hegemonies went unchallenged, genocides went unstopped and whole nascent civilisations were snuffed out just because their planet suffered a comet-strike or happened to be too near a super-nova, even though these events occurred under the metaphorical noses of the sublimed ones.

The implication was that the very ideas, the actual concepts of good, of fairness and of justice just ceased to matter once one had gone for sublimation, no matter how creditable, progressive and unselfish one’s behaviour had been as a species pre-sublimation.

The trouble with taking this idea and running with it as the cornerstone of a whole novel is that “what we know” about subliming makes it look like a really unpromising theme: it’s a mysterious process (from Look To Windward: “the only way fully to understand it appeared to be to go ahead and do it”) typically preceded by “a degree of society-wide ennui”, and typically followed by quiescence.

Banks joins this unpromising theme to a plot that makes no sense. The conflict in the novel arises because there is a faction among the Gzilt that are keen to ensure that their sublimation runs to schedule, and don’t want the distraction of a debate about the origin of the Book of Truth: so keen, in fact, that they are willing to commit mass murder and provoke interstellar war in order to cover up a message from the Zihdren admitting that it was all a childish prank and they are very sorry. But this driver of the plot is witless on multiple levels:

First, in Banks’ Culture setting, it’s not clear what the difference is between a genuine religious inspiration and one that’s merely handed down by godlike beings, nor why anyone would care. I’m open to the possibility that there might be genuine religious fervour, but I need to be convinced, and Banks doesn’t even try: the only character in the novel who actually believes in the religion is Cossont’s mother, and she’s treated as a sad and deluded figure. (The villains who are out to suppress the relevation do so for reasons of realpolitik: they are only concerned that other people will care about it, and want to avoid a scandal at a politically inopportune moment.) Surely for the Gzilt, who are a populous advanced technological civilization in contact with many other civilizations perfectly capable of hoaxing up religious texts to order, the relevation cannot possibly come as a surprise? Every possible theory as to the origin of their religion will have been studied and debated by academics, and every crazy variation used as a plot device in popular media. The likelihood that it was a hoax is going to have been debated to death, and the society is going to have come to terms with it.³

Second, the revelation cannot possibly be compelling. The message that the Zihdren attempted to deliver, and the evidence uncovered by Cossont in the course of her quest, consist of digital records of eyewitness testimony from thousands of years ago. It’s well established in the Culture setting that this kind of evidence can easily be faked or mistaken or the result of brainwashing. So no-one who was really dogmatic about the Book of Truth would be put in a position where they were obliged to change their mind.

Third, the motivation of the villains is premised crucially on the idea that information can be suppressed by secretly killing the messenger. How is this supposed to work in a world where everyone has access to networked information? Have these people not heard of backups? And in any case, the cover-up is bound to be worse than the crime. As explained above, it’s hard to imagine the relevation about the Book of Truth being much of a scandal in the first place. But the murder of the Zihdren messenger and then the mass murder of a group of people who happened to find out about it is something else entirely.

Fourth, this kind of murderous cover-up seems very naïve: surely the modern public relations industry knows many better ways to suppress undesired revelations? Attack the credibility of the messenger; flood the network with distractions and misinformation; unleash an army of trolls. Or just leak the revelation early enough and let peoples’ short attention spans work their course.

★

Like several other of Banks’ Culture novels, The Hydrogen Sonata is a bitter exercise in cynicism and disappointment. All expectations are dashed. The villains’ motivations are extraordinarily petty. No-one is punished for any of their crimes. The Culture ships directing Cossont’s quest decide (after far too many interminable committee meetings) not to tell anyone about what they have discovered. Cossont’s comically oversize elevenstring gets comically ferried round with her in spite of her ambivalence towards the instrument and in spite of the desperate situations from which it must be recovered, but then plays no role in events. Cossont has a robotic ‘familiar’ called Pyan, whose only role is to make occasional cute comments on the action. Cossont’s relationship with her mother is not resolved. Loose ends flap around everywhere you look.

In fact, the novel is so permeated by a sense of ennui that you don’t have to read very deeply to see that it is a desperate plea from the author. Recast in metafictional terms it’s easy to decode the novel as follows: The Culture series is preparing to depart to the great backlist in the sky, but there is a small possibility that an interesting plot development will be found that will keep it going. The author sets out on a quest in search of it, but the interesting development cannot be found on the planet Zyse; it cannot be found in high-tech laser battles between robots and mechas; it cannot be found in the evil motivations of evil villains; and it cannot be found in fights inside dirigibles, not even if the dirigibles fly around in a giant network of tubes. Gradually it becomes clear that the plot development is not so interesting after all. There’s simply no joy left in over-the-top evil and Obligatory Deadly Vengeance any more. The author decides it would be better not to follow up the plot development: better that the Culture series be left to quietly slip away into the twilight.

And I can’t help but agree.⁴

^ This isn’t, strictly speaking, a correct summary of the book. In fact, Vilabier is dead, and it’s his old friend QiRia who’s the object of the quest. But the shorter plot summary is much clearer, and it’s not as if you know or care who any of these people are anyway.
^ This piece plays no other role in the events of the novel, and some reviewers apparently had difficulty with this (for example, Ian Sales).

The title has a thematic connection to the events in the novel rather than a literal connection. The sonata, we learn, was deliberately composed to be “near impossible [for humans] to play acceptably, let alone perfectly”, and even if it is played perfectly (for example, by a machine) then all that comes out is meaningless unpleasant noise: “As a challenge, without peer. As music, without merit.” So the question is, what value is there a human attempting to play it at all? But this obviously parallels the main question in the book, what value is there in living in the imperfect physical universe when you could sublime away into perfection? So the answer has to do with the joy of striving and the value of a goal that hasn’t yet been reached, as opposed to the sterility of perfection.

I don’t know if Banks is making a deliberate reference, but the pieces that come immediately to mind are John Cage’s Freeman Études, which were deliberately composed to be almost impossible to play by humans, and even if they are played very well (as in this performance by Irvine Arditti) all that comes out is meaningless noise. Cage describes the purpose of these études as a celebration of the ability to do hard work: “this music, which is almost impossible, gives an instance of the practicality of the impossible.” (See James Pritchett for more analysis of these works.)
^ John Clute makes the same point: “Though we are in fact shown almost nothing of it, we are meant to think of the Gzilt civilization as enormously complex and sophisticated, most unlikely not to have taken on board centuries earlier the possibility that its Bible is a confidence trick; and Banks’s dodging of any attempt to create a portrait of Gzilt life only intensifies a sense that the McGuffin in this novel is an even more blatant damp squib than usual.”
^ I wrote this review before reading any others, and now that I’ve had a look around the web, it seems that I’m a bit of an outlier in my reaction. A few reviewers commented on how they felt let down by the ending, but treated it as an aberration in an otherwise enjoyable space adventure rather than a thematic part of a cynical whole. Kristen at Fantasy Book Café: “Toward the end was very action-packed and exciting, but the conclusion seemed a bit hastily explained and was a bit of a letdown since not much was learned that hadn’t already been speculated about.” Donna at Radish Reviews: “It felt a little nihilistic, and that would be fine except it seemed to render the entire plot and galaxy tour Vyr goes on moot. Still, there are lots of clever, clever stops on that tour.” But the prize for best reading goes to Andrew J. Wilson at the Scotsman, who sees “a satire on the Scottish independence debate”.