Month: December 2009

Planning – and Executing the Plan – are Part of the Job

Along with writing good code, planning and meeting the plan are part of an engineer’s responsibilities, in order for the product to be successful and the business to thrive

Being an engineer entails more than writing good code. It also requires being a good corporate citizen. We write and test software so that it can be used by our customers. The Engineering team is one of the teams that constitute the business. As such, we need to coordinate our activities with those of the other teams in the business: Marketing, Sales, Operations, and Support. We are dependent on these other teams for our software to find its way into the hands of our customers. We also depend on them for the business to survive. Let’s not forget that Engineering is an expense center, and that without the Sales team, there would not be any paycheck.

Our obligation to the other teams in the company can be summarized fairly simply: we need to deliver what we promised, on time. We thus need to be able to forecast within a reasonable horizon what we will be able to create, and then deliver against our forecast.

Planning is Difficult but Necessary

Some argue that writing software is a creative and innovative endeavor, which, as such cannot be predicted. The comparison is made with Civil Engineering where designing a new building is akin to applying well documented formulas and following well defined processes lending themselves to formulaic forecasting. While there is truth to the argument, it cannot be taken to the limit. It does not means that forecasting a software project is impossible, but rather that it is hard.
This being said, we don’t have a choice. As I often point out to my colleagues, sales people have to forecast every quarter, and one can argue that forecasting sales is eminently more challenging, since it relies on the behavior of people over whom we have very little control: our customers. Yet, no company can operate without a sales forecast, and forecasting is one of the skills that salespeople need to develop, along with their sales acumen. Engineers are in the same situation.

More specifically, the reasons we make plans are:

  • To forecast when a given release will be complete. This in turn will drive forecasts for sales projections, staffing assignment in services – which in turn drive financial projections, and how the company manages its expenditures – such as our salaries
  • To make strategic decisions: for example, if certain set of features take too long, or too many resources, we may decide to postpone their implementation, and allocate resources to another product or set of features.
  • To make our own decisions: by knowing how much work each task will take allows us to staff projects appropriately, and thus be as efficient as possible.  Over-staffing and under-staffing both have negative consequences that are easy to understand
  • To align internal resources: the most obvious example is that the QA team needs to know when a certain feature will be ready to be tested.
    The above illustrates how important it is to meet our commitments, once we have announced our plans. If we don’t meet our plans, we let other people down, and force them to scramble to make alternate plans. Yet, meeting one’s commitments is not only about working hard. It starts with making good plans.

Making Good Plans

How does one make good plans?

  • First and foremost: include everything (easier said than done but none the less critical)
    • Think through ALL the tasks that are required to complete the job: create a new Maven project, become familiar with the idiosyncrasies of a new software package, upgrade libraries to a new version, organize design reviews, code, unit tests, integration tests, performance tests, error recovery tests, security intrusion tests, documentation, training, etc.
    • Account for everything that happens in a typical day/week: e.g. Meetings, interrupts from Ops, support, or other
  • Be realistic: Engineers tend to be optimistic – make sure that you take into account that something at some point is going to go wrong
    • The best technique that I know is to use history as a reference. Have you typically been late/early on your past projects. Are there activities that you typically fail to account for?
  • Build some buffer – because it is important to meet the commitment (and if you don’t need the buffer, you’ll use the time to implement an extra feature, or start the next release early)

Tracking Progress

A tool like Atlassian’s Jira allows each developer to enter their tasks and the time for each task. It is critical that each developer enter their own time estimate. No task should be longer than 2-3 days. If it is, it is best to break it up. I have found it to be the right balance between having enough detail in the task to grasp its whole scope, while keeping the total number of tasks manageable.

It is important to think of a task as a complete project: including reviewing requirements, design, code, integration, testing, documentation, hand-off to QA. Of course, each of these tasks can be spelled out when their scope warrants it. Again, include the typical daily overhead in the estimates .
Once we have entered the tasks in Jira, it is critical to track them accurately. Don’t be shy about entering time beyond your original estimates if you are running late: your teammates, and your team lead, need to know — so that they can make alternate plans if necessary. Progress tracking tools are not meant to find faults, but for project management and communication: it is a much worse offense to your team to keep quiet about your being late, or struggling, on a task, than the fact of being late.  Being late is a problem that can be dealt with – keeping quiet is a professional fault that hurts the project even more than bad code.

One important note: a task is DONE when you won’t need to put any more work into it. In particular, this means a piece of code is not done until it has been fully tested and validated.

Agile Processes for Formal Releases

2-4 week milestones culminating in a show-and-tell where Engineering and Product Owner(s) engage in a discussion about priorities deliver a lot of the advantages of Agile methodologies, even without official buy-in from the management team

Engineering can  follow a mostly Agile methodology, even if the rest of the company does not. For example, you can still break up the development effort into 2-4 week sprints/milestones, even if the Product Owner does not indulge in reviewing priorities for each milestone. In fact, I contend that by having frequent end-of-milestone review, you will in effect elicit prioritization from the Product Management team.

2-4 Week Sprints/Milestones

Regular milestones (every 2 to 4 weeks) are essential for a several reasons, each sufficient in its own right

  • 2-4 weeks is the proper horizon for planning. While it is not impossible to make plans over longer horizons, the accuracy of these plans drops significantly when they extend beyond 4 weeks. Per Agile, the plan for each Sprint needs to be made “bottoms-up” by the developers who are working on the project
  • Commitment to the plan – Since the developers created the plan themselves, we can ask them to commit to its timely execution. Accuracy in estimating one’s work is a skill that each developer must fine-tune
  • Visibility of progress. By having an “almost shippable” release tested at the end of each sprint, we can all assess progress realistically. As the Manifesto for Agile Software Development states “Working software is the primary measure of progress.”. My measure of progress is binary – if a feature passes all the tests then it is 100% done, if not it is not done (0% complete).

With the rhythm of 2-4 week milestones, every one on the team can see the product being built, with the confidence that true progress is being made and the expectation that no nasty surprises are lurking at the tail of the project.
Show-and-Tell

Show-and-tell is the culmination of the milestone, where the project team demonstrates the new features to their colleagues inside and outside of Engineering. It is critical to advertise the Show-and-Tell outside of the Engineering team, including to the CEO, VP Sales, VP Marketing, etc. The benefits of Show-and-Tell sessions are multiple:

  • Rewards for the engineers: The show-and-tell is a perfect opportunity to acknowledge the contribution of each engineer on the project and offer them public recognitions
  • Avoid surprises at the end: the last thing you want when you have toiled away for 3-6 months on a project is to hear something like “Nice work guys … but this is not what I expected!”, whether it is from your own team, or from customers. Thanks to regular show-and-tell, there are no surprises. We also give the tools to the Product Management team to share these early releases with customers, as appropriate.
  • Reassure Management: By demonstrating regular forward progress to the management team, we can relieve some of their anxiety as to whether we will be able to meet our deliverables. Equally important, when the project was too ambitious to start with, Engineering can give early warning to the management team that alternative plans need to be made, whether it is to reinforce the Engineering team, or to manage customer expectations.

Just like the milestones ensure that the Engineering team can manage its progress without surprises., the Show-and-Tell perform the same function for the business team.
Furthermore, the “Show-and-Tell” give “fair warning” to the rest of the company that the release is on its way, so that the marketing and sales machines can rev up in anticipation of the completion of the release (rather than “wait-and-see” until it’s officially released).
Engage into Discussions about Prioritization

While the business team may not embrace the Agile methodology, by holding the Show-and-Tell events, we effectively engage the Product Management team in a discussion about prioritization. Even when their perspective is “Everything is important, everything must be delivered”, by witnessing the progress, a discussion naturally ensues about whether we need to enhance, or rework, what has already been built, what features should be developed next, and the reaction of customers to the early releases.
There is no magic formula to make these discussions happen, but, in my experience, it simply happens naturally.

More than ever, in today’s fast-changing environment, Engineering must be both predictable and adaptable. Predictable so that the rest of the company can operate efficiently (e.g. by starting to market a release before it is actually complete) and adaptable in order to respond quickly to changes in the market and/or the competition. Having frequent milestones, and show-and-tell, give visibility to progress and set the stage to review priorities , and adjust them if necessary, with minimal impact on the efficiency of the Engineering team.