Rapid Backlog Prioritization

It's been a while since I posted. I think this one will be short.

Recently I realized that in my roles as a product manager and development executive I've devised a simple way to rapidly evaluate backlog item priority. While it is critical to prioritize a backlog in a way that aligns with corporate priorities and give that best bang for the buck it is also critical to be efficient in the prioritization process. Most backlogs quickly get ridiculously long. In one brainstorming session you could easily add 10 or 15 new items to the backlog. Each of the 10 or 15 need to be prioritized quickly and moved in to the backlog.

Each new item on the list can be evaluated by assigning a value to two dimensions: effort and impact. Effort is an estimated amount of work to complete the feature. Impact is the amount of impact the feature will have the organization.

A little more on impact: Impact could represent anything of value to the organization. Normally it would take the form of increased organizational efficiency, sales opportunity or other revenue opportunity.

Of course, each of these items could be split into an almost infinite level of nuance, but fight the urge to do it. Keep the two dimensions raw. The reason you want to keep these dimensions raw is because you will be assigning arbitrary values to them anyways. Any more granularity is just false precision.

When assigning an effort and impact use an integer scale that is the same for both dimensions. For instance, you might use a simple 1 to 5 scale. However, I think there is an advantage to using a more asymmetrical scale. You might want to establish a scale more like 1, 3, 9, 20. For instance if you assign something an effort of 20 you clearly think it is massive or if you assign something a impact of 1 you clearly think it has minimal impact on the organization.

So, let's end with an example. I have a coffee maker that makes drip coffee very well and has a low price point in the market place. We have a session with executives and some key customers to talk about the product. Out of that comes a few suggested feature additions.

• Allow the coffee maker to make lattes
• Add a clock and timer to set a time to automatically start the coffee maker
• Add an auto shutoff
• Allow the coffee maker to make toast
• Output hot water for tea, soup, oatmeal, etc.

Feature	Impact	Effort
Allow the coffee maker to make lattes	3	9
Add a clock and timer to set a time to automatically start the coffee maker	30	3
Add an auto shutoff	9	1
Allow the coffee maker to make toast	30	1
Output hot water for tea, soup, oatmeal, etc.	9	3

Don't underestimate the importance of dev velocity in Scrum

I just had a discussion with a director of development at a major on-demand business applications provider. I asked him how his organization measure development velocity, the rate of work completed per development cycle, for each of the products they are building. He mentioned that it is up to the development manager how they measure velocity (usually story points) but, that, organizationally, they do not do a great job of tracking velocity.

This is common when larger organizations adopt the Scrum methodologies. It makes me think of the book Money Ball by Michael Lewis. Money Ball covers how the Oakland A's have consistently been a competitive Major League Baseball team even though they have significantly less money than big-market teams. The A's have stayed on top by exploiting player attributes that are under-valued by the large-market teams. The focus of the A's in 2002 was On Base Percentage (OBP), a measure of how often a player gets on-base regardless of how they get on base (fielding errors excepted). OBP was often under-valued by other teams more interested in batting average and slugging percentage.

The point of the book is to show how a small player can exploit inefficiencies caused by under-valuing key performance metrics. I see large organizations often under-valuing software development velocity when managing software projects. Development velocity is the only measure that can give you an objective view into how long it will take to get a project done. And it only gets more accurate over time. But, a release date can be dialed in very accurately with a fairly small sample size (3 or 4 iterations of 1 to 4 weeks).

I think there is an opportunity for large companies to put key features into the hands of key clients quicker and with more delivery-estimate accuracy if they would focus on development velocity as the key project management metric.

Measuring Requirements and Design Effectiveness in Agile Software Projects

Good user experience is critical to a good software product, especially a software product delivered within the confines and quirks of modern web browsers. For this reason all design and requirements gathering activities at Accolo are performed by full time employees, while off-shoring almost all implementation through contractors and vendors.

We know how to measure implementation effectiveness and productivity using a development backlog and concepts like Running Tested Features (RTF).

So, how do we measure design and requirements effectiveness? Implementation measures do not do a good job of measuring design effectiveness, so we devised a measurement that we lovingly call the Feature Punt Ratio (FPR).

For you football fans (“American Football” for our friends outside the U.S.) you are painfully aware of what a punt means. You didn’t reach your goal and you need to relinquish control of the rock to the other team. It’s not much different in software development, you didn’t reach your goal on a particular feature and you have to punt it away into another development iteration.

The Feature Punt is the number of features that were planned for implementation in an iteration that failed to get implemented because of requirements and design inconsistencies, changes or flaws. Compare the number of features punted due to design and requirements problems against the number of features implemented in the iteration and you have the Feature Punt Ratio.

Features may be punted for various reasons. Like most applications, we have actions in our application that are available to some users and not others. We had this user permissions information sprinkled throughout each action’s requirements document. During the iteration it was clear that the engineers did not understand how the action permissions related to each other. The best way to communicate the permission interactivity was to put together a permissions matrix. When the engineers got the permissions matrix, mid-iteration, they determined they would not be able to complete the feature.

More generically, we’ve seen features punted because we failed to consider a particular use-case. We changed the requirements to account for the missed use-case in the middle of the iteration. The changes were significant enough that the engineer was no longer confident that the feature could be implemented in the original iteration, so it was moved to the next iteration.

It is not uncommon to realize some missed use-cases when implementation begins, so this is our most common reason for feature punt.

After an iteration is complete an inventory of features implemented is taken. Hopefully all features that were committed to were completed in the iteration and maybe some more. If some features were not implemented you evaluate the features to determine if they were not implemented because of design and/or requirements issues or because of implementation problems.

You then apply this basic ratio:

Features Punted/Total Features implemented * 100

You get a percentage of features punted versus the total features implemented for the iteration.

We strive for a feature punt ratio of <= 5%.

Agile Development Off-shore and Problems with the Backlog

A couple of weeks ago I wrote about Agile software development off-shore. In that post I mentioned that the single most important item when planning and managing an Agile project off-shore was a well developed, force-ranked backlog of development priorities. While this is absolutely true, there is one key challenge to managing an Agile project off-shore through a development backlog.

The problem? It is difficult to accurately estimating delivery dates. For engineers the idea of committing to a delivery date is like asking Bob Dylan to let us know when All Along the Watchtower will be finished and ready to record. You don't rush genius ;). However, we all know that the stakeholders and business folks cannot run a business on "it'll be done when it's done." We all have resource constraints, so in all but the rarest cases a delivery estimate is necessary.

The problem is that Agile process requires that designs are only completed right before implementation begins. That means that most feature development time estimates are done without any designs to reference. That is a problem when your technical architect and lead designer are 10 hours apart and don't speak the same native language. In traditional development teams the designer and the architect would sit down together to estimate time to implement each feature based on the technical architects's understanding of implementation and the designers understanding of future feature design.

Without the ability to quickly and easily discuss implementation time and feature designs it is necessary to sit down and go over each feature estimate, talk about the design and determine if the estimate is accurate. I'm not yet sure what the optimal frequency of these meetings is, but I think it should be done each time there is a major update to the backlog.

Agile Development Off-shore

At Accolo, we identified a specific product development opportunity and went out and secured a small round of funding to support it. After we shored up the round of funding we took a look at what we were trying to accomplish and our staffing options to accomplish our goals. There were a few obvious approaches:

1. Hire a small team of full-time employees as developers, qa personnel and design folk and purchase all the tools to support the development effort (project management and QA tools)
2. Hire a mix of full-time and contract employees for development, qa and design and purchase all the tools to support the development effort (project management and QA tools)
3. Hire full-time design personnel and outsource the development and qa effort to an on-shore vendor
4. Hire full-time design personnel and outsource the development and qa effort to an off-shore vendor

Option 1 and 2 were both out immediately. We didn't have the funds to pay the high prices for people in the bay area nor the time to recruit and manage those folks. Option 3 seemed like a good idea, but my experience with vendors on-shore in our price range has been bad (one of these experiences may be a topic for a future blog entry). So, option 4 seemed like the best bet.

So, the next step was to look around for potential off-shore vendors. We picked a vendor in the Ukraine that has both a good track record in actual product development, not just IT tasks, and who also happens to be under the same VC umbrella as we are. We figured that if they are under the same VC umbrella then there would be extra incentive to do well for us. So, since August Accolo has been working with a 10 person team in the Ukraine to build our primary product development initiative. We have maintained design and functional requirements writing on-shore, most recently adding a person filling a business analyst role (we actually stole her from another unit in our company).

We chose an agile approach to the project with the Ukrainian vendor. So, what are the key lessons I've learn in making agile development techniques successful with an off-shore development team? The main success factors to off-shoring an Agile development project are (1) collaboration tools, (2) development backlog, (3) Well defined iteration management process and (4) mockups.

Collaboration Tools

Collaboration tools become extremely important in off-shoring agile projects. Because of the time-zone and language differences written communication is vital.

Collaboration tools must facilitate three things: (1) Rapid person-to-person communication during overlapping hours (2) communication of requirements and mockups (3) iteration management

The good new is that it is not necessary to go out and buy expensive collaboration tools to make this possible. Most off-shore vendors made the investment in project and iteration management tools, now all you need is a way to communicate with the team members. At Accolo we've been able to accomplish our collaboration goals using all free tools, Skype, Trac and Google Docs. Skype provides not only free voice communication, but also a nice IM tool. We use Trac's Wiki features to embed mockups and write requirements around those mockups because it allows for easy linking between different design documents. You could really use any Wiki for this. We use Google Docs for managing our feature backlog (more on this shortly), permissions model and complex requirements that we want to communicate using spreadsheets.

Development Backlog

Usually the first engagement with the off-shore project team will be with a project manager or technical lead that will work to understand your business, requirements and direction. We spent hours with our newly assigned project manager from Kyiv in conference rooms looking at demos, drawing on white boards and discussing our short, mid and long term goals. This was all great stuff and the PM left the US feeling like he understood our business and goals. However, we left out one key thing that caused a lot of confusion and wasted time, we did not produce a development backlog. We went for a few months of shifting priorities and unreasonable expectations before we realized we needed a backlog to manage our priorities. This was an especially big problem when trying to set stakeholder expectations. Stakeholders typically don't care about development challenges, they just want to know what is going to be built and when. Without a backlog of sufficient detail it is very difficult to communicate priority and timing with stakeholders.

There are different approaches to development backlogs, our definition at Accolo is a list of features, each of which can be completed within a development iteration, all force ranked relative to every other feature.

My advice to anyone starting an agile project with an off-shore team is to create a development backlog first. At Accolo we are fortunate to have an existing application. So, for our first pass at the development backlog we listed all the key pages in the application in a Google Spreadsheet; one application web page per spreadsheet row. Then we added row after row of subfunctionality per page until will had a 250+ row spreadsheet. After that we went back to our feature wish-list (all the things that have come up in brainstorming sessions) and added a row to the backlog spreadsheet for each feature. So, what Accolo ended up with was a list of 400+ features all force ranked in a list.

The next, and most important step, is to force rank every entry in the backlog relative to the other features in the backlog. This is tedious work, but it will go faster then expected. At Accolo we started from the top with 1000, with the feature with a priority of 1000 being the most important, and then working down from there evaluating every feature's priority against the last ranked feature. We use a "Priority" column in our spreadsheet that can be resorted as priorities change.

TIP: Try to keep as much of the related sub-functionality prioritized together. For instance, if you are creating a search feature with two subfeatures, autocomplete and reset, try to prioritize autocomplete together in the backlog. As developers are working on functionality they tend to "get in the zone," meaning the developer is very focused on the feature and he has a keen understanding of how the feature works. It is often most efficient to have the developer complete as much of the subfunctionality as possible while the developer is "in the zone."

The next step is to identify the minimum feature set for releases. For instance, at Accolo we identified the minimum functionality for an internal only, non-production alpha release, an open beta release and a full-featured production release. Because our backlog is in a Google Spreadsheet we put a line in the spreadsheet deliniating the minimum requirements for each release (we make these lines orange so they stand out). Defining minimum feature set for releases is a good way set expectations. You can tell stakeholders that you have completed x number of features while we need to complete y features for a release. You can even estimate delivery by looking at statistics about implementation time for the completed features and estimate the gap between completed and not completed features.

The importance of a backlog cannot be overstated. The lack of a backlog at the beginning of our project is the single biggest mistake we have made when off-shoring development using Agile development techniques.

Well defined iteration management process

Early in any Agile project an iteration duration is chosen. I've often seen iterations of 2 or 3 weeks, but never less than 1 week. This, of course, depends on your project, but most web application projects do well with a 2 or 3 week iteration length. Once the duration of the project has been agreed upon by all development personnel the iteration must be broken down into deliverables that will ensure iteration quality and provide requirements for future iterations in a timely manner. Here are the basic requirements for iteration planning:

• Features are broken down into small enough chunks so that each chunk is completable in an iteration
• Feature is complete ONLY if QA and stakeholders have accepted the feature as fully functional - we use the concept of Running Tested Features (RTF)

• Engineers commit to the number of features that can be completed in the iteration based on designs and test-cases

In order to support the basic requirements for iteration planning we have developed a schedule of deliverables based on our 3 week iterations. The deliverables are assigned to designers, engineers or project management and are expected at the same point during every iteration. Below is a diagram of a typical iteration plan and deliverables. The example below goes from the end of iteration 12, through iteration 13, to the beginning of iteration 14. The engineer commitment for iteration 13 happens on the last day of iteration 12, Friday. On the first day of iteration 13, Monday, a discussion happens to determine priorities for Iteration 14. By setting the priorities for iteration 14 the designers and requirements folks now know what to design. The designs for Iteration 14 will be due at the end of the second week of iteration 13, Friday. Once the designs for iteration 14 are complete at the end of week 2 of iteration 14 the QA folks start writing test cases for the new requirements. The test cases for iteration 14 are written during the last week of iteration 13. The engineers then use the designs and test-cases to estimate development effort and make a commitment for iteration 14. Of course, during this entire time the engineers are developing the features for iteration 13 and the QA personnel is testing iteration 13.

Well defined roles and team structure

Accolo is a web application, so we have typical web application project team roles. We have management, design, implementation/engineering and QA. Of course the roles depend on the project. We are heavy on the web design as opposed to, for instance, middleware vendors who will be heavier on the engineering design. Otherwise, this split is pretty common for any software project.

One key thing you must understand when managing off-shore vendors: Your off-shore vendor doesn't know what type of personnel you need better than you do. Even if you consider yourself non-technical you have a better understanding of how your team should be staffed than the off-shore vendor. Don't let the vendor dictate staffing decisions that don't make sense. Don't be afraid to tell the vendor the types of people you need and make changes as quickly as possible if you see delivery problems. Off-shore development engagements are not successful if you hope to just "throw requirements over the fence." You must actively manage the vendor and the personnel choices that vendor makes.

The Accolo application is very user-centric, making heavy use of complex user interfaces. The most important aspects of our application is a simple and highly-usable user interface with heavy automation done on the back-end. So, when our vendor began staffing our project with exclusively back-end developers it didn't seem right. However, we were brand-new to off-shoring development with an established vendor and we didn't realize the importance of actively managing the vendor. We ended up with back-end developers trying to do front-end work. Back-end developers are typically un-able, un-willing and un-interested in doing front-end work. So, we were getting poor front-end work and angering our back-end developers. It took us a few months to realize we needed to actively manage the vendor. We made it clear to the vendor that we though a 1:1 front-end to back-end developer ration was more appropriate. It took a project manager change, but we finally got what we wanted. Today we have 3 pure front-end developers and 4 back-end developers (with some front-end capabilities).

However, while development/engineering staff and structure is important QA is critical. QA polices the requirements, making sure that what is implemented meets the requirements. We were lucky to get a very good QA person. If you choose to off-shore the QA function of an Agile project make sure you get a QA person that understands your business, takes ownership of the requirements you produce and is not afraid to mix it up with engineers. I think a 2 developers to every QA person is a good ratio even when off-shoring an Agile project.

We are starting to see problems with only 2 QA people. We don't have anyone on our team who can write front-end automated tests (we have standard automated unit-tests for the back-end). If you have an application using complex front end technology for a rich user-experience make sure you have a person available, if not dedicated to your team, that can architect and write automated front end testing.

Design is key, mockups are critical

If you have an application that requirements a rich user experience you cannot off-shore the design function. The front-end design must be heavily controlled by the people closest to your customers, you. Trying to communicate the nuance of required functionality to off-shore designers will not be successful.

There is a saying that floats around our team, "pictures save a thousand words." I think anyone involved in software design can relate to this adage. There is no better way to clarify requirements than to show what the user-interface will look like. You can then add detail around each element of the page and speak to the back-end processing related to the feature group. In off-shored Agile projects this is even more true for two obvious reasons:

1. Time Zones: A lot of time can be wasted clarifying requirements when the implementation team is on the other side of the world. As stated previously, there is no better way to clarify requirements that a picture of what the page should look like
2. Language Barriers: There are varying degrees of English proficiency between team members on any off-shored development team. I would extend the adage written earlier to say "pictures save a thousand words and doesn't depend on a particular language." Obviously this has limits because pages have words and labels, but the point is valid. A person can look at the layout regardless of language proficiency and get the gist of the requirements.

The production of the mockups and supporting requirements must be done on-shore with direct access to the business stakeholders. So much nuance is derived from the short meetings and informal encounters between business folks and design folks. Hire FTEs and/or contractors to do the design and requirements work.

Conclusion

Don't make the mistakes we made early in our project. Have at least the beginnings of a development backlog that has been force ranked before engaging an off-shore team. Make sure that the off-shore vendor comes with a project management software suite for development and issue tracking. Establish the way that mockups will be produced and requirements detail built around the mockups. Develop an iteration deliverables plan so that every team member knows what deliverables they are responsible for and when those deliverables are due. A consistent iteration deliverable pattern will allow the development team to get into a rhythm. Hire great design personnel on-shore and focus on mockups with written requirements augmenting the mockups, not the other way around. And definitely don't try provide only written requirements, you will find yourself going around and around in circles clarifying requirements.

My off-the-top-of-my-head-because-I-haven't-had-time-to-research-it estimate of productivity is that off-shoring an Agile project gets you about 80% productivity compared to a full on-shore team. This is raw productivity not taking into account innovation benefits you get from an on-shore, full-time team. But, if you can get an off-shore dev team at 80% or less of what it would cost you for an on-shore team it is justifiable.

What Wikis mean to small, Agile development groups

I've been looking at Wikis now for a while, but mostly in terms of large, public implementations like Wikipedia. I, like most people on the web, use Wikipedia all the time and truely believe in the peer-pressure model for editing quality. Now that's not to say that I will base a recommendation I am writing on information from Wikipedia and I certainly would never reference it as a source. There have been a lot of articles, like this one from CIO magazine, describing the vandalism, fraud and pranks that Wikipedia invites. However I believe these cases to be the exception, not the rule.

So, the larger public implementations of wikis makes sense. However, I work in a small, distrubuted software development group and I am in charge of make sure the team communicates effectively. Where do wikis fit, or, more importantly, not fit, into my small, fast-paced, distributed software development team? Also, what are the security concerns considering the open-editing nature of a wiki?

Where wikis don't fit

After the claim above that I would never do so, I will reference Wikipedia as a source regularly in this blog. After all, this is just the blog of a half-wit, not a recommendation of a half-wit. So, for starters, this article on small group communications from Wikipedia does a good job of outlining some of the more important research done in the last century on small group communication.

This article covers the Input-Process-Output (IPO) nature of group communications in meetings. Now, I reacted to the idea of evaluating my team's communications based around the meeting structure in a way that is familiar to a lot of you in the software industry: "Meetings? Our product and team is way to dynamic to be relevant in long drawn out corporate, talking-head meetings." While this is probably true, by considering a meeting as any communication between team members, no matter how brief, this concept is very relevant.

An input (the I in IPO) for a meeting is usually the task that the group has to accomplish (an input is also any leadership structures and dynamics developed over the life of a team). Process (the P in IPO) is any communication between the team members during a meeting. And outputs (the O in IPO) can be divided into two main categories: 1) Who is to accomplish what and when 2) The cohesiveness of the group. Do members identify with the group?

How is this relevant to a Wiki? Glad you asked. We all know that communication has been greatly sped up, simplified and truncated by technology. The idea of siting down, outlining and composing a snail-mail letter just seems ridiculous. However, going back as little as 20 years ago, this was the standard form of long distance communication and could take weeks to reach the recipient.

None of us would stand for that lag time today. We now send roughly the same information we sent in a well-thought-out letter 20 years ago instantly in hundreds (maybe thousands) of pieces in the form of emails, IMs and...wait for it...collaboration tools. Hey, a wiki is a collaboration tool. Right! I'll get back to that.

Ok, so each email or IM thread can be considered a meeting. A task is brought up by one team member to one or more of the other team members. The team members process the task by discussing strategies for accomplishing it. Finally, the team decides, based on team leadership and dynamics, who is responsible for each piece of the task (note: we are not evaluating the effectiveness of the team members in accomplishing the task).

Wikis are designed to be collaborative authoring tools. Authoring, and editing, collaboration tools are great for long-running dialogs between author and editors. So, of course, a wiki is great for generating very dynamic documents with large amounts of information that ebbs and flows with changes from multiple editors. However, this collaboration is very slow relative to face-to-face, IM and even email communications and completely inefficient for task definition and assignment on a small, fast-moving software development team.

So, wikis cannot provide a tool for communication and task definition within a small, fast-moving software development team. I believe IM to be the best communication tool for all aspects of the development process if used responsibly. Phone is a close second with email taking up the rear.

So, how can a wiki contribute to a small, fast-paced software development team?

Where wikis do fit

I see exactly ONE use for wikis in a small, Agile development group. But, it happens to be an important use: development process and standards documentation.

As of yet, I have not talked about team communication in terms of Agile development, but I think it is relevant when discussing project documentation. Agile development methods have been widely criticized for the lack of documentation generated compared to other software development methods. But the lack of formal product documentation allows a development team to spend their time actually developing a better product, not documenting the problems with and workarounds for a bad product.

Our team is no different. We spend little time documenting and a lot of time improving our product's usability. However, we do need to be able to produce and edit development documentation like coding standards, general architecture, version control usage standards, etc. so new developers can come up to speed quickly. Various members of the team regularly tweak and add to our development process and standards, so the accompanying documentation needs to be equally fluid. This is where the Wiki is perfect.

For instance, Kimberly, the head of our User Interface (UI) design and developer tools, regularly produces great new tools to make UI implementations more efficient and uniform. She needs to be able to easily add the new additions to a central document describing the UI tools. A wiki implementation of the UI development standards documents makes this a cinch.

So, my point here is that there seems to be some misunderstanding about what a wiki can contribute to a small, Agile development team. A wiki cannot contribute to improved communication in the regular development dialog between team members necessary to accomplish day-to-day tasks. It can, however, greatly improved the ability to create and maintain general process documentation.

Wiki security

A quick note on wiki security (wikurity?). The original intent behind a wiki was unfettered access to anyone who wants to add or edit a topic. However, in most commercial organizations your process information is not something you want out in the world. The organization typically gains much competitive advantage from its proprietary business processes. So, we seem to have a dilemma. How do we match the unfettered access to your wiki with information security?

With no password type security how to you limit access to your wiki to only people you want to have access to the wiki? Maybe this kills the whole idea of a wiki.

I don't have the answer.

First

Alright, this is my first blog on my new Transmogrified blog. The point of Transmogrified is to share my experience with management and technical issues around web applications.

You may want to know, 'Who the hell are you?'.

I'm glad you asked. I've been in the software industry for over 10 years. I've served as a web designer writing simple web pages, a development intern writing version control tools based on RCS, an engineer building a mobile application development platform for a major ERP vendor, and the head of development and information technology for a small startup (my current post).

In my current position I have been tasked with building the development group from nothing. We are now 6 people strong with very good processes based around some of the tenets of Agile (I will blog on my feelings about Agile later).

So, I guess this blog has turned into an introduction. What I have on the agenda for blog topics are 'Wikis in the development process' and 'What Agile tenets work in a small development team'. So check back.