Something that set today apart from yesterday is that I had opportunities to talk with several of the speakers. When I arrived, Chet Hendrickson and Ron Jeffries were sitting by themselves at a breakfast table, so I joined them. They were having a conversation about human nature, and whether it doomed us all to failure (heavy stuff for 8AM). By the end of the conversation we were discussing doing Agile in University settings… and unusual experiences with plumbing.
I had lunch with Eduardo Jezierski, who just before lunch gave a riveting presentation: Architecture and Agility with Lives at Stake. He had many stories to tell. One was about developing, in real-time, a system to save lives after the earthquake in Haiti. He had a product owner in a tent at the airport in Haiti, with a low-bandwidth connection, tweeting user stories as he received calls and people came to him with information. He was right near the runaway for days, with C-130s constantly landing with supplies and taking off again. They were able to broadcast SMS messages to people with cell phones, telling them to text a specific number if they were trapped or injured and needed help, or knew someone who did. His team was developing a system to get those incoming messages into an RSS feed that could be accessed by local responders. It was also fed back to the Haitian diaspora in the US, where volunteers who knew the neighborhoods used Google Maps to help pinpoint locations where people were trapped, based on descriptive information. He said in situations like this, they’ve had to get help for their programmers who develop post traumatic stress disorder. When you receive a message like “I’m pregnant, I’m trapped in a collapsed building, and I’m hurt” and you’re trying to develop code in real time to get that message where it needs to go… well, you can get just a bit stressed. I was fascinated by the work his organization did in Haiti and elsewhere. They have often turned into the facilitators between governments, NGOs, and local communities, as they try to leverage IT to solve local health problems and address emergencies.
The other talks I attended were also good. But it’s past my bedtime, so I’ll skip summarizing them for now, and close by mentioning I attended my first Philly WordPress Meetup after the conference. It was a great group. I met the organizer, Brad Williams, and we talked about the next Wordcamp Philly this Fall. It’s still a ways off, but I may give a talk about my current experiences with applying Agile coding practices to WordPress plugin development.
The first day of the Philadelphia Emerging Technologies for the Enterprise conference was great. Molly Holzschlag‘s Keynote gave us some peace, love, and understanding, at least as far as web standards are concerned (she even made references to the Age of Aquarius…). After that, it was a challenge to choose from among the simultaneous sessions throughout the day. I’m there not just for me, but also for my team at SOMIS. They have a variety of interests, so I made sure to attend at least one session on each major theme of the conference (Agile, mobile design, management, languages, and infrastructure).
I’ll focus on my favorite talk, by Chet Hendrickson and Ron Jeffries, who years ago helped define many Agile practices. They presented A Retrospective Rant: 15 Years in the “Agile” Business. They’re both pleased that Agile has become a success and has gone mainstream, but also unnerved that it’s so mainstream that the stodgy Project Management Institute is going to offer scrum certifications (imagine Emperor Palpatine realizing the rebels actually aren’t so bad, and joining forces). Their presentation was very lively, and at times reminded me of Abbot and Costello, as they constantly played off each other. They focused on the most common failings they’ve seen in teams that are trying, but not fully succeeding, with Agile:
- No shippable product increment at the end of an iteration. Chet’s advise: think of every sprint as if it were the last one (not in the sense of stress, but in the sense of focusing on developing complete, functioning, tested features). The idea is that, if you had to go live, you could do so at the end of any sprint (you wouldn’t have the complete feature set, but every feature that you’ve worked on is complete).
- Too many defects: these are “negative features” that handicap the product owner’s ability to plan. They argued for a defect count of no greater than 1. They gave a couple real-life examples of (non-Agile) companies with backlogs of as many as 1,000 defects. Even after stopping all other work for months to fix the defects, after they fixed the 1,000, they had discovered almost 1,000 more.
- Failure to integrate: individual features are not integrated into a functioning set of features before the sprint ends
- System designs that degrade over time: architecture requires constant adjustment as the system evolves
- Partitioned teams: when team members handoff to each other instead of working together, misunderstandings and mistakes proliferate
- Failure to prioritize: saying yes to every project leads to multi-tasking, which leads to frequent context switching for the team, which results in everything taking longer and more mistakes. Work on as few projects as possible at the same time – ideally one.
The other talks I attended were all very good:
- Say Yes or Say No? What to Do When Faced with the Impossible. This took Jeffries and Hendrickson’s point about failing to prioritize projects, and went in-depth on its terrible consequences, and what you can do to prevent it.
- Are You Satisfied With Your Tests? This was a great talk on what I’d call “code smells” for tests – that is, bad habits in testing that you should kick: slow tests, blase attitude towards failing tests, over-mocking, unnecessary complex object builds, gratuitous use of the database, overly “meta” tests, and testing private methods. Here are another attendee’s complete notes.
- Polyglot persistence for Java developers – moving out of the relational comfort zone: the first half of the talk was a comparison of 3 major noSQL databases: Redis, Cassandra, and MongoDB. The last half focused on using them with Java Spring, and assumed a knowledge of Spring that I didn’t have, so I cut out early, but the first part was very informative.
I’m looking forward to Day 2!
It’s been over a month since I blogged or tweeted. Aside from this post, it’ll be probably another month before I do so again. I’d especially like to apologize to the people looking for help with Shashin and my other plugins, as I have not been responding to support requests (for my plugin users, please see this post).
As I mentioned back in the Spring, I’ve been leading our web team’s transition to scrum. Since then we’ve been working with Agile/scrum training coaches Bob Hartman and Darian Rashid, and they’ve done an amazing job helping us make the transition a successful one.
Before starting with scrum we had poor visibility into our future work – planning was extremely difficult. Now we’re getting better visibility, and it’s something of a “be careful what you wish for” situation. I’ve been working nights and weekends for the past month, getting a handle on all our projects and our schedule, so I can manage expectations for both my team and for our stakeholders. Work is the first thing I think about when I wake up, and the last think I think about before I go to sleep at night. It’s going to stay that way for at least a few more weeks (possibly months), as we get through this transition.
We have several goals: improving quality, teamwork, etc. But our first is to improve our planning: to align our workload with our actual capacity, establish a sustainable pace, and create reliable expectations for our stakeholders. With scrum’s velocity measures and other metrics, my ultimate goal is to clearly demonstrate to our stakeholders what our team already knows: that we do an incredible amount of quality work with a very small staff, and that if we’re expected to do even more, we need more people.
UPDATE: A disclaimer to calm some readers who were worried: this post is by no means an endorsement of Rumsfeld. I’m just taking a peculiar statement of his and using it as a way to think about scrum (and I just thought it was funny).
“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don’t know. But there are also unknown unknowns. These are things we do not know we don’t know.”
- United States Secretary of Defense Donald Rumsfeld, February 12, 2002
It’s not an accident the name of my department at U Penn’s School of Medicine is Information Services, not Information Technology. We develop software, and for that we are adopting scrum, as I mentioned in an earlier post. But we also provide services, such as meeting various ad hoc reporting needs for our clients. Until now we have tended to be more reactive than proactive, in that we typically wait for service requests to come to us (and they often come on short notice) and then we scramble to fulfill them. Our current workflow is all about multi-tasking and being interrupt-driven, both of which are productivity killers. In a certain sense we are extremely far out on the agile spectrum, in that the idea of locking down requirements and schedules for even as short a duration as a 2 week sprint will be introducing a lot more workflow structure than we’re accustomed to.
To make the change, we need to become more proactive than reactive. Do all those ad hoc requests really need to be quite so ad hoc? Can we do more to anticipate service needs and plan better? This is where Rumsfeld’s seemingly mystifying statement has relevance:
- A known known is the work we are planning for a sprint. We prioritize user stories, estimate them, etc. and plan our sprint. This is the work we know we will be doing.
- A known unknown is work we know is out there, but we lack good information on when and/or what exactly it entails. An example is a complex, crucial report we know needs to be run on a certain date, but we don’t know what kind of data issues might come up. The report might turn out just fine and only require a few minutes of effort to run, or the data may contain very subtle errors that take hours to investigate.
- An unknown unknown is work that cannot be anticipated: a newly exposed bug on an essential system that urgently needs to be corrected; the Dean asking for some new report that no one knew he would ask for (and asking the Dean to get in line is generally not an option…), etc.
A notion we’ll be testing as we move to scrum is whether we can move a lot of our current known unknowns into the known knowns column. To continue with the report example, if we know when it will be run, we could put some slack around it in the schedule, since it’s an important report. We also could do a test run of the report ahead of schedule and review it with the client to see if the numbers add up as expected, to minimize the risk of a last minute surprise. This will require us to be more proactive with our clients, in that we’ll need to seek out and identify upcoming service needs and incorporate them into our sprint planning. We also need to involve our clients more in the overall quality equation, such as asking (and expecting) them to test and review important reports ahead of their due dates. This calls for a culture change for some of our clients, which I expect will be the most significant challenge for us in adopting scrum.
With unknown unknowns such as the Dean asking for unexpected work, we may never be able to control them entirely, but with the tools scrum gives us (such as burndown charts) we can at least more clearly illustrate for our clients how ad hoc requests are impacting their project schedules. It’s also my belief that work requests that are truly impossible to anticipate don’t actually happen very often. The harder cases are bugs that take hours to unwind, or code changes in one place that have unexpected consequences somewhere else. We can reduce these over time by introducing technical practices such as test driven development and regression testing. Both of these will require training and the development of a testing infrastructure, so it will take a while to get there, but we can get there. In the meantime, I’m planning to not have our scrum teams completely fill their sprint time in their sprint planning. We’ll leave some time in the schedule for handling unexpected work.
You might be wondering about unknown knowns – things we don’t know we know. That gets us into the realm of repressed childhood memories and past lives, and I’m hoping that such things won’t impact our sprint planning…
I’ve barely blogged at all for the past several months, and the main reason is that my new job has been keeping me busy, and using up my creative energy. But it’s all good – I’m enjoying the challenges. In case you missed my previous tweets about it, I’m now Director of the 14 person web team in U Penn’s School of Medicine Information Services. I’ve been a member of the team for 5 years, so my role is new but much of the territory is familiar.
Our team has created over 80 web-based applications (including our own development framework), and well over 150 static web sites, and we’re adding new applications and sites all the time. It’s amazing work for such a small group. Check out the portfolio of some of our static sites, and here’s an example of one of our applications: the PennMed Clinical Trials search site (most of our application are for internal PennMed use only, so this is one of just a few that are public).
We’ve started two pilot projects with scrum, which is an “agile” methodology (click the link for a basic definition). Currently, we have a workflow that’s evolved over the years, and is scrum-like in some respects: we expect and allow change, we communicate frequently with our clients. But we’ve held on to “waterfall”-like habits, mainly because of how we were educated as programmers: we try to model as much as possible up front, then we build all the tables, then code all the business logic, etc. Because we are caught between these two approaches, we often run into frustrations: we’ve tended to see refactoring as an indication that we did something wrong up front, we’ve never been sure about the right ways to document our work, and we’ve struggled with scheduling and allocating resources.
I gave a presentation to our team last week on why we’re evaluating scrum, and why we’re doing it now. There’s nothing top secret about it, so I’ve uploaded it to SlideShare, and it’s embedded below.
UPDATE 12/3/2010: I’ve revised and replaced the original slideshow: