Keeping with my plan to do more shorter, local conferences and not keeping with my plan to avoid weekend conferences, I’ll be hosting Robert Chatley and Matt Wynne’s eXtreme Startup at the Devnology Community Day

It was great fun to run it at last years’ XP Days Benelux. It’s always amazing to see how focusing on incoming feature requests lets you easily forget the big picture.
Participants at Devnology should have at least as much frustration So bring your laptop or pair up with someone and join the fun. The only
thing you need is your favorite programming environment and a way to respond to HTTP requests.

FOSDEM – Sunday February 5, Brussels, Belgium

If you break a plan, do it in style. So I’ll also be doing something on the Sunday. Stephan Eggermont has arranged a Smalltalk Devroom at FOSDEM

We’ll probably be doing a longer version of Back to the Future, (Re)Learn Smalltalk like we did at the SPA Conference last year.

To keep my sustainable pace, I’ll just move my weekend to the Monday and Tuesday after these conferences. We’ll see how that works out.

Mini XP Days Benelux – Monday April 23, Heeze, Netherlands

Keeping up with the two main session themes of last year, smalltalk and configuration management, Stephan Eggermont and yours truly will rerun our session on getting started with Chef and Puppet at mini XP Days Benelux, the full program is yet to be announced.

Rob Westgeest kindly championed our session, as he assumed we learnt from the feedback from the previous session. The lego-themed slides were well received, but we could have focused the introductory stories more on one or two complete examples, as opposed to telling a bunch of benefits related to things we’ve done. The same went more or less for the code samples. It wasn’t clear to all participants how the ‘big picture’ fit together, so we’ll see if we can visualize that.

I look forward to seeing you at one of these conferences!

I was at the chamber of commerce yesterday to change my Inc (Besloten.Vennootschap.) to a personal company (eenmanszaak). Writing down the details of the takeover my personal company does from the Inc, I noticed it was exactly ten years ago, on 27 December 2001 my business started.

Yves Hanoulle suggested I write a short post about it. So here it goes.

The reason I’m de-incorporating my company is that incorporation was doing more harm than good. I originally incorporated to participate in the founding of Seek You Too, since this year better known as Seecr. They are ten years today, because Eric (my then business partner) and I had to incorporate our personal holdings first. Why did we do this? Our then accountant advised us to do so. When we wound our collaboration down in 2004 it did provide somewhat useful as we at least got the basic rules of engagement down in a contract.

Before that we’d already changed accountant. Our new accountant frowned somewhat at the complexity and cost of three Inc’s for two people. But it’s a lot harder to get out of incorporation than it is to get into, probably because it is less common. Apparently when a business grows, it incorporates, and when it fails it is liquidated.

The state where a business is profitable, but the overhead of an Inc is too much is rarer, although the person who handled the registration of my new company at the chamber of commerce said he saw quite a lot of it lately. I scoured the internet for hints on how to unwind a BV, but nothing much turned up. If you’re interested I’ll do a dutch write-up after it’s completed.

If you’re making less than 150.000 Euros in profit after costs and your salary it’s not worth the trouble, unless you need it for other reasons. Paying yourself a minimum salary of about 34000 euros is obligatory here, because the tax man is afraid you’re not going to pay enough taxes otherwise. Pro tip: wind the incorporated company down when you’re still profitable, being incorporated while not making more than your salary + costs sucks. there were many months I had to figure out how to pay income tax while waiting for payments to come in. Luckily I had two very good years, so I could compensate for the losses I made the years before, and I’ll start the next year as simple as necessary. Same name – Living Software – new company.

As I wrote in about the name  a couple of years ago, a company is still the best way for me to search for those moments and situations when I am most alive. I’m looking forward to the next ten years!

What do Extreme Startup, Smalltalk and Server Login Considered Harmful have in common?

I’ve made some slides to promote sessions about them at XP Days Benelux. Competition for the most hilarious Official Half Minute Pitch at each half day of the conference is fierce. This year sessionorganisers can send in slides. Luckily Stephan Eggermont reminded me that the deadline is right about now. With a few iterations, film poster-like slides are my themes for this year.

When Robert Chatley asked if I wanted to be stand-in for Matt Wynne, who has more sensible priorities than most of us I thought for half a minute and said yes. The eXtreme Startup is a hands-on session that lasts a morning, but allows for new participants halfway through. I attended it at Agile Cambridge, and found it loads of fun and educational. Can you and your partner code against the market and your competitors? Will you deliver the most business value against changing demand? Can you balance the flow of euros against technical debt? Join us this friday at 9:30 in room 16 and find out.

One of the ways to iterate quickly while finding out what is a good market fit, is to use a language and environment that was build to explore the unknown. Join Stephan Eggermont and yours truly this Friday at 16:30 in Room 16 and find out how web development can be like desktop GUI development, and how Debugger Driven Development optimizes your inner software development loop! No code was harmed in the making of this session.

So, you’ve found your strategy to find your market and keep up with it and you’ve found a development sweet spot. Now your deployments can’t keep up. Servers burn down, your mail is a spam magnet and as a systems administrator you are getting tired of late night phone calls. Devops configuration management to the rescue? Come to our session “Server Login Considered Harmful” and find out if Chef can save your Puppet’s bacon. Find out this Thursday at 11:00, again – surprise surprise? – in Room 16.

XP Days Benelux is already sold out, I just wanted to share the fun Ihad in preparing it. Still hoping to see you there .

Regardless of what I say next: creating a pan-european agile/lean event that is not tied to one of the schools of thought, nor any functional silo (looking at you, coachcamps and testing days!) is a major achievement. I tried to pull it off with about twenty other folks a couple of years ago, and failed (early, but nevertheless. It is not something I bake cake for ).

First highlight: I found a community that welcomes skepticism without suffocating it. Thanks, amongst others, to @OlafLewitz and @ojuncu for open discussion on twitter beforehand, and to those who offered help instead of rotten tomatoes after I threatened to put trainers and coaches out of a job in a lightning talk . And those who took my call for more coaches combining hands-on work with coaching (too?) seriously. More on that in follow up posts.

Although many of the scheduled talks rehashed things I hoped would have vanished from agile / lean conference agenda’s to make place for new things, I was pleasantly surprised by a number of talks. (See below in talks)

Building a car using a distributed team.

Wikispeed is a collaborative, distributed effort to build a fuel efficient car (one that is also efficient at high speed, unlike some other fuel efficient car). Thorsten Kalnin described how they use whatever works to get there. At this moment that includes for instance aspects of kanban, scrum, using object oriented principles for mechanical engineering and finding storage boxes that can be rented for cheap, while still being suitable to do engineering work.

I’m listening to vinylbaustein’s wikispeed dreams while writing this. Finding new music at a conference, what do you want more
Vinylbaustein – WIKISPEED dream – SGT01 mix by vinylbaustein

The inside wikispeed video is worth watching if you want to see more:

Your baby is ugly

Seeing Stephen Parry in action again, during his talk and afterwards during dinner. Olaf Lewitz has described other aspects of Stephens’ session. What I particularly liked was ‘your baby is ugly’ – get employees to analyze the companies performance as seen through their customers’ and describe that to their peers and managers (also mostly employees, lets’ not forget that). This is probably a lot more effective than outsiders saying that (e.g. Yours truly tweeting about how SAP is going lean but their products prevent their clients from evolving), and I guess it takes a lot of time and patience. Something I would like to try, nevertheless.

A3.

I stayed out of the Claudio Perrone’s A3 presentation after seeing the intro slides from outside which looked so heroic (Big Agile Transition), that I was afraid I would not be able to shut up during the presentation and heckle ehm excite. Turns out the rest of the slide deck is a lot more to my liking, including a pragmatic use of the satir change model somewhere in the middle. So I was wrong. Looking forward to see the video of it.

In the meantime I would recommend reading Toyota Kata by Mike Rother. I took it on the road this week and am pleased by how it does not try to make this kind of open ended coaching look easy, and uses storytelling plus questions to you, the reader, to show how this might work in your practice.

One more for the skeptic

It was only talks. Tip for next years’ organisers: if you want more hands-on sessions make some space for scheduled sessions that last longer than thirty minutes, some sessions that require preparation don’t magically happen in open space. Stephan Eggermont and I enjoy the challenge of seeing what of a three hour session we can meaningfully cram in thirty minutes, but we may be the exception. It seemed most people were watching as opposed to downloading the zip and following along.

I was looking forward to ‘exciters’, Walldorf and Statler from the muppets were given as example, and bar tables were placed in front of the presentations for people to heckle from. I haven’t seen that happen, and didn’t dare to either, preferring the safety of my twitter feed instead . I asked Stephan Eggermont to heckle during my lightning talk, which he did, but I was so focused I didn’t hear him do it…

It seems the open space explanation left out the butterflies. When I described my open space experience to another participant at the end, he stared blankly at me. It also seemed mostly open space veterans where butterflying and bumblebeeing around, as well as applying the law of two feet (sorry, the politically correct wording used in the opening was so complicated I failed to remember it . Shorter PC versions I’ve seen include ‘the law of motion’ or ‘when in doubt, move out’ ).

So I butterflied, mostly, except to host an open space session because someone wanted to pick my brain about session selection processes. The best suggestion in that session came from @LLillian, who stated the obvious thing I’d missed before: besides playing a perfection game, encourage session organizers to Ask for Help, to lower the barrier for new session organisers. Duh. Why didn’t I think of that before! Other than that, it struck me that few of the open space topics were original. It seems we have been struggling with the same problems for a couple of years. Maybe it is rotation of participants, maybe we are asking the wrong questions. Of course, I could have taken responsibility by posing a question as a session. It seems the stuff I’m struggling with right now is in a stage where I don’t know how to formulate the questions yet. It seems grumpy tweets, responses to that and hallway discussions help me move forward .

It will probably take a week or so to pinpoint my newfound frustrations and puzzles, at least I feel my time in Berlin was well spent. Especially now the QWAN Learning Community has some more first followers, I got over the scariest two minutes presenting I’ve done in quite a while, and I got useful feedback on the idea from various people. So thanks everybody for making it happen.

Ten years after the agile manifesto I believe we’re better of with it, than without it. To me, it served as a rallying cry for those who were developing software to improve themselves by looking at what works in practice.

Remember? “We are uncovering better ways of developing software by doing it and helping others do it.”.

The reason this appealed to me, was that I was working at the time, and had been working before, with good willing, hard working people who advised others on building and delivering software, but they didn’t get their hands dirty (and if they did, their practices looked nothing like what they advocated for clients). With Agile, I believed we had a fighting chance to do better than what came before, prevent methodology wars by finding what unites us and market ‘lightweight methods’ a lot more effectively.

These days, when I travel to an agile or lean related conference (and even to some craftsmanship related events) most participants are full time managers or coaches, occasionally coding at night if they are so inclined. In the early days, if I would run into a fellow coach, it would be someone like me, who worked at all levels, with all stakeholders while also working hands-on with and in teams. These days, if I get cynical, it is, more often than not, someone who helps a team to move post-its better, either through kanban or scrum.

Not that the intentions behind scrum or kanban are bad (‘improving the world of work’ and ‘Evolutionary change for your technology business), it’s just that obsessing over the mechanics, such as the right way to move post-its or the best practice for stand-ups costs less energy than thinking for ourselves about how we work, and what we can improve.

As I’m leaving for ALE2011, which on the surface looks like it’s packed with post-it moving folks, I hope to be proved wrong and be surrounded by complex systems thinking hackers, like  I was at XP2001, just after the agile manifesto had launched. That ALE opens with a massive coding dojo gives me some hope.

Otherwise, it’s just going to be 3M who benefited the most from lean and agile software development

I’m live blogging from the uk government IT meeting on agile at the SPA conference. Nothing beats a blogpost fired off in anger

Here are some tweets I would like to elaborate on:

Me: “Offshoring government it is stupid”

Eric Lefevre (@elefevre): “why? on paper, if there is any way to commoditize software for government, it would sound like a win, wouldn’t it?”

Me: “Software is executable knowledge. Offshoring software is giving a potential enemy exclusive posession of that knowledge.”

Imagine, if you will, the UK code breakers in the 1920’s outsourcing some of their work to Germany, because labour there in the crisis after the first world war was cheap. Manual computations, but nevertheless. What would have happened by 1938?

This is a bit of a straw man of course. However, the problem with outsourcing and/or offshoring is that you put executable knowledge, plus the expertise to grow that knowledge and build knew software from it in the hands of one party. That leads to a dependency on that party. As long as things go fine, that is swell. When things go bad, not so much.

If you think that documentation is going to help. It might, if you are lucky. Most of the knowledge that goes into software is tacit, and very difficult and costly to transfer.

So if you are a government agency (or a company for that matter), don’t be stupid. Keep some good technical people on permanent staff, who know how your crucial systems work, and can make modifications. If you add some external staff to that to be flexible in capacity or bring in specialist knowledge, great!

Just don’t forget to let them work side by side with your permanent staff on a daily basis, so your organisation does not leak knowledge that it can’t recover on its own.

Just remember, knowledge is power. If you’re not careful, you will learn that lesson the hard way.

The underlying assumption is that adding more features after the minimal set will make your product more valuable.

At some point adding features makes a product less valuable to users. Less features are easier to understand for users and for developers, which in turn can help improve the value of each feature.

So next time you build a product, think about it. When will more features diminish the value of your product to its users? Why not on your current one? Are you there yet? How far still to go? Past the point already? Could I make this post more valuable by asking less questions or using fewer words?

We constructed a web and mail server for an early stage startup. Stephan wrote the application for it together with Diego Lont, and they were looking for a smart way to deploy it. I had looked into automating systems administration tasks with scripting tools such as puppet and chef before, had tried puppet a year before and failed to earn back the time I put in.

This time our goal was to be able to build a new server quickly, because the startup did not have the funds for more expensive disaster recovery options. We also wanted to not think separately about writing documentation: if left as a a separate task, documentation is often just left.

Stephan was somewhat new to linux systems administration and wanted to learn, I have done linux administration on the side for almost ten years now, but many things I still don’t really understand (e-mail, firewalls), even though I administer a couple of servers that seem to be running ok.

We hoped that documenting ourselves in the form of scripts would help us understand, and that writing these scripts in a Domain Specific Language designed to describe servers (puppet in this case, but it could have easily been another tool) would force us to be precise in describing our understanding.

So why did we need to log in?

Our web server would not start, but puppet did not pass the output of the web server (lighttpd) on to us.

Why did our web server not start?

We made errors writing its configuration file.

Why did we make errors in writing the configuration file?

We used lighttpd, a web server I hadn’t used in a couple of years. Its configuration is not that hard, but you have to use exactly the right lines. Especially with secure sockets (a bit tricky on most servers) this turned out to be harder than we thought.

Why…

We don’t have automated tests yet for the configuration, Stephan and Diego’s software has automated tests on the inside, but the mail and web server stuff does not have automated tests. I have a virtual machine that is, with the puppet scripts, roughly a copy of the production machine, but learning puppet and figuring out how to automate tests for the environment was too big a step.

There are many more whys. Sometimes we were in a hurry, so we didn’t sit together to think through the next step, do it, and reflect on it. Sometimes we still did things directly on the server, because we could not figure out how to quickly do it in puppet. The first steps in automation required a lot more patience than I expected. The object oriented database used does not come with a ubuntu package, but needed to be installed with a couple of shell scripts that required manual intervention. We did not have the courage or understanding to rewrite the scripts as (automated) puppet recipes or an APT package.

It didn’t occur to us at first that logging in was a smell. We just started to notice after a while, that whenever we logged in to the production server, there was something we had overlooked. We broke something, or could not see whether something was running properly or not. Sometimes we took action to prevent having to log in the next time, sometimes we thought it was too much work for now, and accepted that our automation is not yet ‘perfect’.

If you made it this far, congratulations! You may want to hear more. If so, join us this Sunday at 15:00 In the configuration management devroom during FOSDEM 2011 in Brussels (no registration required). We’ll be talking about configuration management for developers. Learn from our mistakes, so you can make different ones

Thanks to Stephan for coming up with the first title. And Marc Evers for shaping the second one.

Why is running a session at SPA a good idea? Because you will lay awake at night after having a nightmare of rowdy SPA regulars tearing your session apart? Not only that Running a session here is an excellent way to learn, and since the conference is relatively small, there is ample opportunity to continue working on your ideas after hours. SPA attendees are critical, curious and have lots of experience, which can make your session interesting in various ways .

In case you have not organized a session at SPA before and are interested in running one, I’d like to highlight some things that are different compared to most other conferences:

The level of interactivity of most sessions – with interactivity we don’t just mean questions and answer blocks within sessions, but participants actively involved through hands-on exercises, dialogue (through e.g. fishbowl discussions), and simulations. We sometimes allow presentations, but this is the exception, not the rule. The SPA site has a list formats with explanation, and we are open to new formats, so be creative!
SPA participants are highly experienced and educated. This means they have seen fads come and go, and take nothing at face value. If you use this well, this can make your session an experience you won’t soon forget. If you use this badly, this can make your session an experience
you won’t soon forget . Be prepared for vocal, and sometimes rowdy (in a good natured sense) participants.

And some things to consider when writing a session proposal:

A session proposal has a couple of audiences:

• You, sitting down to write your idea for a session makes it brutally concrete
• Prospective attendees : why should they attend your session? What can they learn? How is it going to be fun? You will have the opportunity to edit the participant facing part of your description after it is accepted.
• Reviewers: How will this session work? Is it feasible in the time? Will it be fun? How can participants contribute?
• The program committee: does it fit in the program this year? Does it have enough SPA-ness?

You can ask participants to prepare something in advance (e.g.position papers, experience reports, code samples, even exercises). We
will keep the number of sessions that require this kind of preparation limited: SPA attendees are usually quite busy; your session proposal
should also include a backup plan in case there is not much prepared work sent in, and should usually accept attendees who did not prepare something in advance (we are of course willing to make exceptions – like participants, each session is unique).

SPA is a conference that has been long running, so it has some traditions. The session proposal form is pretty long and detailed. My take on it, is that it forces you to think through at least a couple of concerns related to your session. However, don’t let this stop you from doing something extracurricular , e.g. writing a blog post to describe your idea, or putting up a screencast of you coding (in case there is coding) on youtube or vimeo.

Finally, a disclaimer by the voice of reason: even when we encourage you to work your idea out into a full-blown session proposal, it still might not be accepted. Eventually a review committee consisting of SPA regulars will decide whether your session is a good fit for this years’ programme and has sufficient ‘SPA-ness’.

This is not to discourage you. Running a session at SPA is usually a lot of fun! I often find it very rewarding (especially running the session. preparation a bit less…). You will get fairly detailed feedback on your proposal once it gets formally reviewed. Before that, you can get feedback from your peers and give it to them, so all the proposals get better. Finally, if your session gets accepted, we will assign a ‘shepherd’ to you; someone who has run a session at SPA before to support you in the final preparation. We also ask the participants to give feedback on the session when you run it, which we will pass on to you as well.

If you’ve made it this far: congratulations! Now head over to the SPA conference website and put in a draft: you can iterate on it after your first post, so a title and abstract are enough to get started!

I look forward to seeing your sessions arrive! In case you have any questions, don’t hesitate to contact Rob and me.

I hadn’t expected the speedup would be this dramatic – my laptop is now serving between 1000 requests per second (most time spent in the first cache miss from drupal) and 9000 requests per second (no cache miss), where it otherwise would fail on just 1000 requests with 100 concurrent ‘users’ (also running on the same machine).

In sharing the gotchas, our configuration and some useful sources I want to give something back for the countless blog and forum posts that helped me set things up.

Gotchas

(pertaining to Ubuntu 10.10 and Debian Lenny, Varnish 2.1.3 and Drupal 6.x)
Out of the box varnish comes with an empty configuration file and will cache nothing. The idea seems to be that you can verify that your site works as it did before, and then add your own vcl (varrnish configuration language) in small steps, verifying at each step that it still works.
For Drupal 6 you need to install pressflow and the varnish module, and make sure they work. I haven’t tried this with Drupal 7 yet.
The drupal varnish module currently needs a patch to work properly: patch: http://drupal.org/files/issues/808314.patch issue: http://drupal.org/node/808314
Debian lenny’s version of varnish is outdated, as varnish is relatively young and seems to be actively developed. I got varnish from their own apt repository : http://www.varnish-cache.org/installation/debian was very useful.
Test! A couple of times I thought it was working, but nothing was actually cached. My weapons of choice where ab (Apache Bench), which can fire off a large number of requests, also in paralel, curl -I which will show the headers that come back from the server, and some custom VCL instructions to add an additional http headers to show cache HIT or MISS. This is all about attention to detail.

Process summary

Install on your own machine first, then on a server, doing it at least twice forces you to better understand how it works.

Benchmark with ab to get a baseline, see what difference it makes

Install varnish, not on port 80 at first (default is 6180), only switch with your regular webserver once you are sure enough that it works well.

Install pressflow

Configure everything, benchmark again

Test functionality of your site; We will do things such as removing cookies, in my case that broke logins for registered users when I first tried it – VCL is a (domain specific) programming language and its power deserves to be treated with the same respect as your PHP code.

Celebrate! For anonymous users your site will now be a couple of orders of magnitude faster. In our case it also means that some parts that expose very slow backend systems now actually work, as opposed to giving errors all the time and bringing our server to a crawl (work to make the backend systems perform is in the pipeline – this is a stopgap solution of course).

Detailed steps

Benchmark to establish a baseline

I use

ab -c 100 -n 1000 <frontwebpage>

which gives you 100 parallel requests for a total of 1000. It might be that your server can not handle 100 parallel (-c) requests at first, so if your computer seems to hang (like mine did…), lower that at first.

Install varnish – the version in Ubuntu 10.10 is recent enough. On debian Lenny I used http://www.varnish-cache.org/installation/debian

Extract pressflow over drupal installation (your DocumentRoot), then go to admin/settings/performance and set cache to: external

Install Varnish module (I use drush for that), go to admin/settings/varnish. Settings:

flush page cache on cron -> off

check that port / hostname of varnish control terminal matches (is set to varnish default port 6082 on localhost)

Varnish cache clearing -> drupal default

add the authentication token as documented here http://www.varnish-cache.org/trac/wiki/CLI The token can be found in /etc/varnish/secret check the status test at the bottom, varnish control terminal should be accessible. If not: see what ports varnish has open with sudo netstat -ap | grep varnish. The port where varnish runs at is determined in /etc/default/varnish . This is what I needed the patch for. Other things you may want to verify: http://drupal.org/node/941788

Develop your own VCL. I took the one from four kitchens as a starting point: https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+pressflow and added a http header to indicate cache HIT or MISS I found elsewhere. I put my default.vcl on github https://gist.github.com/744643 for your inspiration.

Eventually, also change: /etc/apache2/ports.conf to: NameVirtualHost *:8000 and Listen 127.0.0.1:8000. Then edit all your virtual hosts configuration files, if they contain a port number.

Enjoy!

Further reading:

http://blog.justinlintz.com/2010/08/configuring-varnish/ (also has comparison with varnish)

http://www.howtoforge.com/putting-varnish-in-front-of-apache-on-ubuntu-debian :

apache or another webserver will see 127.0.0.1 as ip adress of your visitor, add apt-get install libapache2-mod-rpaf so that logging in apache will show the correct ip .

http://www.varnish-cache.org/trac/wiki/

http://developmentseed.org/blog/2010/oct/12/leveraging-aegir-deploying-drupal-7-ssl-and-varnish

I’m working on a puppet recipe to automate most of this stuff, maybe I’ll publish it when I’ve done some more installations like this, so far it’s been running well on my laptop and on a test server, but it still needs to see some heavy use. The fact that both drupal.org and varnish-cache.org (both heavy traffic sites) now use a similar setup and work smoothly does give me confidence though…
Next up is improving performance for logged-in users, there the most valuable options are a lot less clear to me.
I hope this helps you in getting a better performing drupal site!