Agile Planet

My book “Secrets of a Buccaneer-Scholar” will be published in September by Simon and Schuster. It’s a book about how I approach self-education, personal branding, and original thinking. It’s the story of how I got started and how I got along all these years without any formal education or certification. It’s not a book about testing, but it will be highly relevant to testers. It will be relevant to any knowledge worker who wants to compete better in a down economy.

The book is finished and delivered, it just takes a long time to get through production and sell into book stores. While I’m waiting and writing the next buccaneer book in the series, I’ve started a new site at buccaneerscholar.com and a new blog. There I will talk about the more general topics of thinking and learning that are not specific to testing.

I’m also creating a class to teach self-education (not as ironic as it sounds… but it does sound ironic, eh?).

This blog will continue to deal exclusively with testing and issues directly related to the software development industry.

In a previous life I took a music degree in Bass Trombone, which is a discipline that’s even more geeky and with a worse gender balance than Software (see this meeting if you don’t believe that’s possible). Over the course of the degree, I raised my bar from Enthusiastic to Not-too-embarrasing-to-appear-in-public. In the spirit of this famous article on Pair Programming, here are a few things I learned there that (if I squint) seem to apply to software.

Quality follows a power law Every time I practiced and got to play with better ensembles, the next level up was an order of magnitude improvement, not just linear. I could imagine being as good as the players one level above me, but two levels was a huge jump. The standard of serious professionals nowadays is just astonishing. As an example, one of our ensembles worked on an Edgar Varèse (obscure but influential modernist composer) piece. For the première in the 1920s, Varèse complained bitterly that he only had 100 rehearsals, we did it in about 10 rehearsals, our conductor performed it with a group of freelancers after working on it for an afternoon.

Quality is fractal To put it another way, good performances requires attention to detail at all levels: from the conductor’s management of the overall structure, to players’ phrasing of their individual lines. The better the ensemble, the more levels just work. It was quite a shift for me when I started joining groups that played in tune, a whole area of insecurity just disappeared and I could now use the effort I’d spent on compensating for inaccurate tuning for something more important.

Play for the audience They’ve paid to hear you. They don’t want to hear your technique, they want to hear music. One of my teachers liked to point out that anyone can make great music work, but good players can make bad music sound better. At the other end of the scale, I once heard a scaled-down version of The Rite of Spring where the Bass Trombonist dominated the orchestra; as a practitioner, I was impressed but it was ugly and self-indulgent.

Don’t take it up unless you mean it The perfoming arts are tremendously rewarding if that’s what you want do. If not, it’s a hard trade involving lots of stress and effort—and there’s a limit to how long you can stand discussing mouthpiece diameters. I met several older professionals who hated the business but had nowhere else to go, and the story goes that one of the trombonists in Toscanini’s NBC orchestra wrapped his instrument around his music stand at the end of his last day before retirement.

The section takes the blame (and credit) together Brass playing, particularly at the low frequency end, is a collaborative activity. You spend much of your time playing chords, so no-one can tell that you’re better than the rest of the section. Your best hope is to try to raise everyone’s standard. When it works, it’s just fantastic.

Some people’s abilities are just obvious I met a few players who were clearly here on Earth to play their instrument, that’s just who they were. These are the kind of people who get top-rank jobs before they’ve finished college. Our Head of Brass had been a child virtuoso, he didn’t understand why people played wrong notes. Why would they want to do that? At the other end, there were some who should have had their instruments confiscated.

Sometimes people take a while to shine Other players are not so obvious. I was lucky enough to meet Ed Anderson (Cleveland Orchestra) who was one of the best. He told me that at college, he’d played in the opera orchestra because he didn’t get enough sessions with the (higher prestige) concert orchestra.

Reality, deal with it In a performing discipline there is nowhere to hide, everyone knows how good you are all the time. I had a bit of a crisis in my second year when a new Tuba student took the time to point out my shortcomings; the message wasn’t pleasant to hear but it worked, I was much better by the end of the year. Later a one-off lesson with Arnold Jacobs, who spent a lifetime researching the mechanics of wind playing, changed my playing life because he could show me what was happening to my breathing and how it needed fixing.

How do you get to Carnegie Hall? Practice! It turns out that this topic has just come up on Tim O”Reilly’s blog.

What’s the dynamic range of a bass trombone? On or off

In my previous post, Working Hard, or Hardly Working?, I mentioned how it was possible to tell the wheat from the chaff by observing.  Alistair Cockburn rightly suggested that the manner of observing that I suggested was more appropriate for a team leader than for upper management.  Yes, he’s quite right about that.  He had in mind a Vice President who, when told that the development team had estimated something would take two weeks, asked him,

How can I tell if it really should take that long, or they’re just padding to make their life easier?

How would you answer that question?Alistair says he had no answer.  I have a question.  What does “should” mean?

At one level, it’s a tautology.  If they’ve already done the work and it took two weeks, then “bearing in mind that there are many factors of which I am unaware, since the event has already occurred, it is obvious that” this work should have taken two weeks.

Or it’s a self-fulfilling prophecy.  For these developers in this situation, it’s going to take at least two weeks if they’ve decided that.

Now it could be they’ve estimated poorly, and they’ll deliver it in one week to everyone’s surprise.  So it shouldn’t take that long and it didn’t.  If there’s padding, it’s probably to prevent disappointment rather than to avoid work, else they’d have stretched the work out for the full two weeks.  You’re likely to see this behavior where estimates are treated as commitments.  You’re also likely to see this when developers have become accustomed to such treatment elsewhere.

It could be that some other set of developers could easily do this work in one week, but the best that this set can do is two weeks.  Does that mean this set of developers should be able to do it in one week?  Should they have the experience with this sort of problem, with this language, or the skills to work faster?

It could be that these developers could do the work in one week if it weren’t for all the impediments in the environment.  Maybe the computers are old and slow and have too little memory.  Maybe there are delays waiting for information or confirmation or projects on which this one depends.  The developers should be able to do it in one week, but due to circumstances beyond their control they can’t.  Should the circumstances be different?

It could be that these developers could do the work in one week, but they think it implies other work that isn’t part of what’s being asked of them.  If it were discovered that they thought this extra work were part of the request, then they could be disabused of this notion and complete the work in one week.  But it’s not discovered, so the developers do some unwanted speculative work and it takes two weeks.  Should they have noticed and done it in one week by doing just what was wanted?

It could be the developers feel that they’re treated poorly.  They feel demoralized and it’s hard to get out of bed to come to work, much less to focus on the work they have to do.  If they weren’t so depressed and disorganized, perhaps they could get it done in a week or less.  But everything seems so hopeless to them that it actually takes two weeks.  Should they be able to shake off their depression and get it done more quickly?

It could be the developers are lazy SOBs who are out to do as little work as possible while drawing their salary.  They take frequent coffee breaks where they talk about sports.  When at their desks, they spend hours playing Quake or surfing random websites.  Should they work harder on the company’s business when on the clock?

These are all quite different scenarios.  The world “should” seems to mean very different things in each of them.

Should someone who has risen to the level of VP be about to tell the difference between these scenarios?  Well, of course.  But this use of “should” has as many meanings as the one concerning the developers.

Should she have some comparable experience to use as a sanity check?  This would also be nice.  But maybe it’s a new industry.  And given all the possible situations listed above, how can she now if it’s comparable?

Should she take the time to talk with each developer, one on one, to learn how they view their work and the company?  If the question is important to her, then I think this is the way to go.  Having these conversations can, if handled well, contribute to mutual trust.  And the very wording in the question indicates to me that in this situation, trust is not as strong as it might be.

But the direct benefit is insight.  Some of the questions in my head would be

• What is it about this job that makes it take two weeks?
• How sure are you that it will take two weeks?  What are the chances it might take one week?  Or three weeks?
• What are the unknowns about this work?
• What similar work has been done by this team?  How long did that take?  How does this work differ?

I might not ask these questions directly, but instead talk with each developer until the answers seemed clear to me (or that the answer would never be clear).  If Ms. VP hasn’t practiced this, she might benefit from something like the Unearthing the Data You Need session I attended at AYE 2008.

And, if it really is just a case of a goldbricking team, then it’s unlikely that all of them are equally skilled at pulling the wool over our VP’s eyes.

George Dinwiddie is considering a discussion about velocity as a performance measure and how to tell whether people are working hard that started on the scrumdevelopment yahoo list.

Here's the original question, posted by Graeme Matthew.

The unknown in all of this is that if a team have a velocity of 6 how do you tell if they should have a velocity of 8 i.e. they are underperforming. It gets complex. If they have a velocity of 16 are they doing well or have they estimated at the higher scale of story points.

0. Using velocity as a performance measure is wrong, wrong, wrong. Velocity just is, and you work to improve it over time by improving engineering practices, the work process, working relationships, and skills.

The assumption behind the questions is that people might be slacking off, not working as hard as they can.

I don’t want people to work as hard as they can. I want them to work hard, but at a pace where they aren’t making themselves more error prone, and aren’t exhausting themselves.

[aside]
If the team seems to lack a sense of urgency, it might be because they are missing information about why the project is urgent. The fact that a senior manager made a bet on the golf course doesn't make the project urgent. That some sales person promised the moon and stars without any knowledge of the development effort is not urgent--for the development team. Making a once-a-year sales window...that's urgent.)
[/aside]

Some managers seem to want to know who is working hard, and how isn't.

My experience is that some people have to work harder to accomplish the same results that another person can accomplish easily. Some people look like the are working mighty hard, but are not accomplishing much.

Thinking that we have to have everyone working equally hard doesn’t make sense to me…Why would you want to punish the person who gets his work done quickly by piling more on?

Further, there’s almost always someone how isn’t the best programmer, but things just go better when he/she is on the team. They may not look like they are working hard, but take them off the team, and everyone's performance goes down.

Perhaps some different questions:

Are the conditions present for every member of the team to do his or her best?

Have the managers put together people with the mix of skills and expertise needed to do the work the company needs done?

Are individuals performing to the expectations of their pay level?

If there’s a perception that people are not working hard, what might the reasons for that be? (Systems, policies, procedures that dis-incent performance; work they don’t care for; disenchantment with direct managers?)

As a manager, I'm more interested in who is struggling and/or who doesn't have the skills or interest in doing the job they've been hired to do. In a manager-led team, that pretty easy to notice if you meeting with people on a regular basis.

On a self-organizing team, it's harder, but not impossible. First, there has to be enough trust to support transparency. When there's visibility into the work, then a manager can see where things are binding up, and investigate the source of the problem. The source might be a person, but might also be the process or some other factor.

This post might be better titled, "How (and how not) to help yourself when Google doesn't have the answer: A whirlwind tour through Rails' source" if only I wasn't too lazy to change the max length of the database field for titles to my blog entries. Google sometimes seems as if it has the sum of all human knowledge within the confines of its search index. It might even be the case that it does. Even if you prefer to think that's true, there may come a time when humanity does not yet have the knowledge you are seeking. How ...

My old friend Simon Voice at Connections Recruitment now has a voice of his own.

I wrote a teaser post about iterative and incremental rebranding of eXperience Agile in September… It’s been four months and almost as many newsletters since then . So time to de-tease the blog…

Marc suggested to do part of the upcoming newsletter as a Temperature Reading, so besides getting some information, our readers also learn a technique, and it helps us structuring our own reflection over the past year. It’s been about a year since Marc, Rob and yours truly sat together after xp days london to discuss closer collaboration with courses and coaching (but still as loosely coupled as possible, because we like our independence).

A Temperature Reading has five questions, the order of which you can vary, depending on the context (see Temperature Reading for explanation and the default order, I’m going to do something different here).

1 New Information

We proudly presented QWAN = Quality Without A Name

We started handing out these rubber ducks at CITCON Amsterdam, for those cases when we are not around to pair (program, brainstorm, reflect, be innovative) with. Talking out loud helps, and ‘rubber ducking’ is a well known technique for this (e.g. Kathy Sierra writes that talking to pets works even better. We came across it the first time in the Pragmatic Programmer ). We chose QWAN as a name, because we are still inspired by Christopher Alexander’s early work on patterns (Timeless Way Of Building) most notably. QWAN is a label for stuff done by Rob Westgeest, Marc Evers and yours truly, often in co-operation with others.

QWAN co-created a bunch of new courses and workshops in 2008: eXperience Refactoring (also with Octos’ Emmanuel Gaillot), Unit Testing Masterclass, responsibility driven design with mock objects, dirty jobs (with Tjakko Kleinhuis), pimp my retrospective (with Topic’s Nicole Belilos), political games (with Emmanuel Gaillot), story testing with Rspec, and a revamped rightsizing your unit tests (and then some that I forgot. This collaboration seems productive, many more than last year it seems. We supported conferences in one form or another, e.g. by sponsoring, as co-organizers or ‘just’ running a session: CITCON Amsterdam, several Agile Opens, XP Days Benelux and London, ESSAP summerschool, Agile2009, AgileHolland.

2 Appreciations

We’d like to thank everyone who participated in our courses, conferences and workshops. Your enthusiasm and feedback helps us to improve and, not least, have fun in our jobs. Everybody who provided encouraging feedback when we published our brochure earlier this year. Especially Nynke Fokma, Becky Winant and other keys for giving detailed suggestions for improvement. We’ve had several great customers (you know who you are), who created amazing conditions for us to focus on our work. Thank you!

3 Complaints with recommendations

We’ve made some progress in supporting ourselves with a Customer Relationship Management system. There are still some improvements that would make our work easier (usability) and that of our customers (sign-up and notification of new courses). (Marc wishes for a more sustainable pace. I’m careful what I wish for - a more sustainable pace would be nice, but I’m also quite happy with our productivity, passion and income at the moment)

4 Puzzles

We made progress in selling open enrollment courses, and are still puzzling on better ways of selling those (as well as making it easier on our customers to plan and buy participation). The context for courses seems to be shifting at the moment - working more effectively has value, but many large corporations have the knee-jerk reaction of putting all training on hold when sales slow. At these corporations people have little time for training when things are good - too busy, and no budget for training when things are bad. Luckily we also have customers with a long term perspective, who invest in training, education and hiring continuously.

5 Hopes and wishes

We hope the ‘interesting’ economy will mark the breakthrough of lean and agile ways of working in more places. We hope to carry out more integrated on-site coaching … With integrated we mean doing whatever works best in a given context : a whole systems approach using combinations of lean, scrum, extreme programming with ’soft’ skills inspired by Satir and systems thinking.

We wish you a healthy, fun and prosperous 2009, and look forward to collaborating with many of you.

(meta-appreciation: thanks to Marc for coming up with the idea for a Temperature Reading, and giving feedback while listening to the items while writing them. Thanks to Marc & Maroesja’s kids Maris, Marden and Marijn for providing the necessary distractions )

A while ago I was chatting with a post-doc on his way to an academic career. He was asking me about research topics wanting my input as he felt I could inform him on what would be research of practical use. I wasn't very helpful, but I did mention that the best way to do this would be to spend some time in industry to get a feel of how software development works in the wild and what problems could do with some research effort.

His answer to this thought was very troubling. He said he'd be up to do that, but if he spent time in industry that would ruin his chances of getting a job in academia. Competition for academic jobs is high, and what they look it is your publication history. A year or two in industry would create a gap in your publication history that would be lethal to your job prospects.

The divide between academia and industry has always been an awkward one for software (as indeed for other professions). My contacts with academia have been stilted at best. The academics I respect are, I'm told, not highly regarded within academia because the things that I count as useful are usually dismissed by the academic community.

A good example of where this came to a head is the patterns community. Those involved in the patterns world were keen to look at practice to discover, package, and document techniques that had been proved through experience. But this is in direct opposition to academic standards which consider value to lie in novel things. My work, for example, is generally dismissed because all I do is write about stuff that is old hat (at least to some).

I think this is a terrible shame, not because I'm looking for an academic post, but because I think there is huge value in mining effective techniques from the experience of software development. To me it seems that trying to draw lessons from our experience is a very worthwhile academic activity. By devaluing it the academic world is ignoring a fruitful avenue to improve the capabilities of our profession.

If my opinion counted, I'd argue that any academic department worthy of note should include a group of faculty with a long experience of the day-to-day of industrial software development. They would be valued on how they had reflected on this experience and drew from the lessons to inform their teaching and research. I'd like to see a regular rotation of people from the academic to the industrial world, where it's common to see people spend several years in industry, then academia, then industry again, and so on.

This problem isn't only in software. A friend of mine had the chief engineer role in one of the most challenging engineering projects in the world. He fancied a stint in academia, but was only able to get a second-class position reserved for people who weren't considered to be real academics, certainly not something that was tenured or would lead to tenure. I find it hard to believe that students wouldn't gain an enormous amount from being taught by people with a long and thoughtful experience in the profession they are entering.

It's always frustrating to see communication gaps between different groups within the same profession. I've become a big fan using Rotation to help open up communication channels, as people are the key to good knowledge transfer. Being tolerant of academic rotation, indeed encouraging it, could do a great deal to make academia more aware of where industry needs help and industry more aware of where academics can improve practice.

Will DSLs allow business people to write software rules without involving programmers?

When people talk about DSLs it's common to raise the question of business people writing code for themselves. I like to apply the COBOL inference to this line of thought. That is that one of the original aims of COBOL was to allow people to write software without programmers, and we know how that worked out. So when any scheme is hatched to write code without programmers, I have to ask what's special this time that would make it succeed where COBOL (and so many other things) have failed.

I do think that programming involves a particular mind-set, an ability to both give precise instructions to a machine and the ability to structure a large amount of such instructions to make a comprehensible program. That talent, and the time involved to understand and build a program, is why programming has resisted being disintermediated for so long. It's also why many "non-programming" environments end up breeding their own class of programmers-in-fact.

That said, I do think that the greatest potential benefit of DSLs comes when business people participate directly in the writing of the DSL code. The sweet spot, however is in making DSLs business-readable rather than business-writeable. If business people are able to look at the DSL code and understand it, then we can build a deep and rich communication channel between software development and the underlying domain. Since this is the Yawning Crevasse of Doom in software, DSLs have great value if they can help address it.

With a business-readable DSL, programmers write the code but they show that code frequently to business people who can understand what it means. These customers can then make changes, maybe draft some code, but it's the programmers who make it solid and do the debugging and testing.

This isn't to say that there's no benefit in a business-writable DSL. Indeed a couple of years ago some colleagues of mine built a system that included just that, and it was much appreciated by the business. It's just that the effort in creating a decent editing environment, meaningful error messages, debugging and testing tools raises the cost significantly.

While I'm quick to use the COBOL inference to diss many tools that seek to avoid programmers, I also have to acknowledge the big exception: spreadsheets. All over the world suprisingly big business functions are run off the back of Excel. Serious programmers tend to look down their noses at these, but we need to take them more seriously and try to understand why they have been as successful as they are. It's certainly part of the reason that drives many LanguageWorkbench developers to provide a different vision of software development.

When the team self-organizes, the manager needs to step back, but not too far back; she needs to step in, but not to quickly.

My article on the manager's role in self-organizing teams is on the cover of Better Software Magazine and available on-line here.

Finally, we got along to XPDay today, albeit only for a few hours. We went specifically to see my brother, Marc Baker, and Dan Jones from the Lean Enterprise Academy do their keynote. It was great to meet Dan and talk with him over lunch.

Dan talked about the evolution of Lean. Marc talked about Lean thinking and how they're conducting 'experiments' in various industries, particularly healthcare, to compress value streams, eliminate unnecessary waste and increase throughput. It was a good talk. Refreshingly different, I thought, but still pertinent. The anecdotes were amusing and the metrics were both shocking and impressive, in terms of what was going on before the experiments and what the experiments achieved. And I'm actually glad they didn't talk about Lean in the context of software.

The slides are here.

When I saw the proposed stages for agile 2009 last week, they looked a bit bland to me. A lot of the fun and risky stuff seems to have been taken out (musical stage, breaking acts, questioning agile). Ok, open space (I love open space) is still in there.

The 2008 conference was quite good for such a large event. Things that contributed to it were, in my opinion, scrapping experience reports as a separate track and instead having lots of experience reports everywhere as well as the breaking acts and questioning agile stages. Yes, you can have those topics spread out, but having those topics front and center (what is the first thing you see when you look at the conference? its the stages. So the first thing you see is: ah, critical thinking, wild new cideas).

If you want to support new ideas, and at the very least keep up the outward appearance that the agile community invites and supports them, leave a comment on David Andersons’ blog post “The case for an agile fringe” .

I also left one at Johanna Rothmann’s (2009 conference chair) response “I’m Disappointing Already” outlining some of the forces I’m seeing. Johanna ems to take it personal, which I think is unfortunate. Sometimes folks in the kanban camp seem to do the same - getting their sessions rejected for earlier conferences must have been very painful. The reasons for rejection were probably systemic, and it seems some of the systemic reasons have been fixed in 2008, now 2009 seems to be sliding back (and yes, kanban is now establishment, but it will happen again to new new stuff).

So, let’s stop taking it personal, and improve the system; no regression, if we can easily prevent that please - carry out one refactoring “replace experience reports stage with fringe stage”, that’s it.

That’s it for today, maybe I’ll post more later. I’m going back to xp days london (which is one big fringe lots of new ideas and critical thinking going on, and the open space is an integral part of the program, which seems to work fairly well at this scale.).

This is the "z0mg it's Christmastime and have I really left so many of my goals for the year incomplete?!" edition of Programming Quotables . If you don't know - I don't like to have too many microposts on this blog ( me on twitter for that ), so save them up as I run across them, and every once in a while I'll post a few of them. The idea is to post quotes about programming that have one or more of the following attributes: I find funny I find asinine I find insightfully true ...