Metrics of success in development – Part 1

How do you find out if your development organization is functioning well? Naturally, if you are getting products out on time, consistently, and the world around you is happy with the results, you have nothing to worry about.

But what if there ARE complaints? Can you determine whether you’re hearing gripes that have little to do with you? Or whether there really is room for improvement?

I’m working on a self-assessment tool, probably containing about 10 questions, that will help you evaluate whether you are on track. The way to use the tool is to answer the questions and then count the points at the end.

Here are the first three questions in my current working draft:

1. Counting only the past 3 projects and products under your management, how many have been completed on time? For each project/product not completed on time, how much later were they completed?

[4 points] for each project completed on or before the originally scheduled completion date
[3 points] for each project completed within 120% of the originally scheduled duration
[2 points] for each project completed within 150% of the originally scheduled duration
[1 point ] for each project completed after more than 150% of the originally scheduled duration
[0 points] for each project which is never completed

2. During the time that those 3 projects or products were being developed, how much turnover did you experience among your technical staff, including first-level supervisors and managers? Turnover means departed or transferred out from project teams or management without being invited to do so by you or your managers.

[3 points] less than 5%
[2 points] between 5% and 15%
[1 points] between 15% and 30%
[0 points] over 30%

3. In those 3 projects or products, how much of the planned functionality was delivered (at the completion date you used for question 1)?

[add 3 points] for each project in which you delivered all of the planned functionality, plus additional functionality defined after the start of the project.
[add 2 points] for each project in which you delivered all of the planned functionality
[add 1 point] for each project in which you delivered at least half of the planned functionality
[add 0 points] for each project in which you delivered less than half of the planned functionality

If your total points add up to 24, you have a perfectly-performing development organization and have no need for improvement. For the rest of us, the points are probably in the following ranges:
Excellent: 18 to 24 points
Good: 12 to 17 points
Fair: 6 to 11 points
Poor: 5 points or fewer

Shipping products on time is the key result that most of your stakeholders want. Shipping the product with the features and functions that they asked for — or expect — is next in line as a measure of your success. But if you are burning out your development crew — and causing turnover as a result — you won’t be able to sustain the results. Therefore, these are the first few measures that tell you whether the organization is successful and sustainable.

Next time we’ll begin looking at other measures that can tell you whether your management practices are helping you build a development organization that is consistent and successful for the long run.

Constant Reinvention = Survival

Nothing lasts forever. Even the best-conceived business strategies eventually become constraints on growth.

Consider Dell. “Dell succumbed to complacency in the belief that its business model would always keep it far ahead of the pack.” But the competitors got better while Dell failed “to invest in new business lines, talent, or innovation that could provide another competitive edge.” * [see Business Week citation below]

As a leader in technology or product development, you may think that your entire job is to execute well on the development plans laid out by Marketing or a strategic planning group. But you can do more. You can help the executive staff recognize that the business has other opportunities.

Consider what the Business Week authors went on to say: “Long-term success demands constant reinvention.” This means that while you’re turning out products that meet the current set of goals for functions, price and quality, the viability of the company may depend on your pointing out where innovative products or services could come from, using the brains you already have on your staff. Reinvention means re-thinking the orthodoxies everyone has accepted as the characteristics of the company. Do you have some independent thinkers on your staff who keep coming up with off-the-wall ideas? Maybe some of those ideas are actually your ticket to survival.

“Nurture the next growth platform long before it’s needed.” This means you have to carve out the budget to support the radical ideas from the operating budget you’re supposed to use for mainstream development. If you can’t convince your executive staff to fund a skunkworks operation, then you should look at having some of your key contributors doing some off-the-record investigations. Is this risky in your company? Then maybe the company needs some shaking up.

“Most [companies] don’t [nurture the next ideas]. Distracted by the demands of their current success, they re lulled into a false sense of security.” It’s easy to focus only on the tasks that will satisfy the demands of current customers and current ways of doing business. And while it’s not easy to perform on those tasks at extraordinary levels, you can get lost in gunning for the immediate satisfactions of meeting this quarter’s goals. Can you be a VP or Director of Engineering and still make time for thinking about next year’s products and the businesses that you haven’t entered yet? Consider this: who else is better positioned to view what’s possible, who is out there needing better functions or services, and what can be combined to make a new business?

I suggest taking an advocate’s role as part of your commitment to the long term success of your company. You don’t have to be a marketer or business analyst to know what’s exciting and feasible in the next generation of products and services. Carve out a niche as a visionary and a keeper of the wild ideas that can open up new busineses for your company. Do it regularly, and you’ll be twice as valuable as the person who only meets the usual goals of Development. Besides, it may help your company survive.

———-
* “Where Dell Went Wrong” by Nanette Byrnes and Peter Burrows in Business Week, February 19, 2007, pages 62-63.

http://www.businessweek.com/magazine/content/07_08/b4022074.htm?chan=search

Is your software on fire?

The spectacle of Dell laptops on fire in the summer of 2006 due to Sony battery problems has prodded me to think about product failures. There is nothing so attention-getting as a fire in a conference room. Few people who see this sort of failure will forget what they have seen.

Software failures may not be so spectacular, but they can be just as memorable to the people who witness them.

Examples from large-scale software systems: if you were waiting for your baggage in the new Denver airport a few years ago, you may have waited until human intervention delivered your bags, because the bag sorting system failed. And what if you dialed 911 and the call did not go through?

Examples from embedded systems: your cell phone drops a call due to a software glitch in the phone; your hard disk loses track of its position and takes an extra several seconds to recover.

Examples from everyday use of an operating system: Windows gets confused while processing interrupts from the web browser, and the browser hangs until you reboot; Outlook misses a beat and an email doesn’t appear on the screen when you expect it to.

If the computer or phone were to catch fire when any one of these failures occurs, you can bet that the manufacturer would do a massive recall the way Dell has done. But they didn’t. Instead, they let the users keep on running with a piece of software that “catches fire” regularly. If you’re like most users, you have become accustomed to seeing these fires and dealing with them. But do you like them? Of course not.

What are the consequences? Word of mouth travels quickly, and these failures have created a large population of users who resent having to use devices and software that fail. Resentment leads users to search for a better alternative. This is good for competitors who offer a better, unfailing solution.

But the whole world of software (and digital devices that depend on software) suffers from a bad image because of these failures. From consumer devices to mission-critical industrial control systems, everyone who has to deal with modern digital devices is gun-shy about failures. And rightfully so.

Is your software on fire?

There are known methods to assure that the software-dependent devices you make will not catch fire. If you’re not certain what these methods are, or who can implement them for you, you need to find someone who can help. But before you call in a consultant, be sure that you’re willing to pay the price: It takes time and money to make software reliable, just as it does in batteries and laptops. Are you ready to buy in?

Development Process Stability

After shipping a first product, successful companies face a number of challenging problems in product development, including lack of development process stability as development work scales up. Here are some responses that have worked well in the computer, software, storage and consumer electronics industries.

Why process selection matters now

As Product Development scales up to involve multiple concurrent projects, three things happen to stress the development environment and to threaten its results:
1. The informal processes used at startup no longer work reliably to get quality products produced on time.
2. More managers are needed as the development department becomes too large for one person to manage directly.
3. Supporting customers and manufacturing takes time away from development but offers opportunities for feedback that must not be ignored.

This is an opportunity to choose good processes for the next 5 years. There will never be time to reconsider process selections. The cost of change only goes up.

Managing multiple projects is more complex. Engineering and Product Marketing must cooperate in new ways to assure that the next generation of products is successful.

Founders and early employees who are technologists have been crucial to success, but they may not be willing or able to make the transition into a development environment that is sustainable for the long run.

Development processes

Development must move out of crisis mode, so that projects can be completed on a predictable schedule.
There may have to be changes clarifying who is responsible for setting project goals.
Project management tools need to be used to manage schedules and feature lists without overloading the development team with overhead tasks.
When a milestone is missed, rapid analysis and decision-making is critical to staying on track for product introductions. Certain metrics are useful here, and project teams need feedback about how they’re doing.

Development tools

Who is responsible for Quality? The Development department must get serious about product quality, even if QA is managed from Operations or elsewhere.
Bring in hardware and software tools for testing, establish disciplined procedures for release, and stay in the issue/correction feedback loop.
In addition, Development can provide useful input to product direction in the next cycle through interaction with customers and field staff.

—–
If you would like more information about how we assist growing companies with managing product development for the long term, please visit http://johnlevyconsulting.com, call 415 663-1818 or email info@johnlevyconsulting.com

Why is Engineering the last to call for help?

Engineering and the product development organization are critical to a company’s survival. In successful companies, they deal daily with a vast array of problems, from technology shifts to people loss. One of the key talents of successful technical managers is to deal with changing priorities and resource availability. They manage these dynamically whether by PERT charts or just seat-of-the-pants intuition.

So why is it that they don’t often ask for help?

I believe it has to do with two aspects of the occupation itself.

(1) When your daily life is filled with adaptation and improvisation, you have trouble imagining that there is anything anyone can do to help. Your talent as a technologist managing others is to be able to evaluate technical directions in an instant, moving people around to cover the top priorities of the day, and communicating to your bosses what is going on. How could a consultant or an internal mentor help this kind of activity?

(2) You are already in the midst of trying to improve the engineering process and the way your people accomplish projects. You have the credibility with them, so you can influence their work to improve a little at a time. It is inconceivable that an outsider, or a non-specialist insider, could have more influence on your staff.

The Marketing Department and even the Finance people know that Engineering is in trouble when products don’t get completed on schedule, turnover is high, or products need extensive tweaking to meet customer needs. But inside Product Development, life is normal: dynamically adapting to the shifting priorities, making quick decisions about fixes, and just getting the next product out the door.

The only way to get the processes to improve significantly is to get perspective. And perspective is the one thing that most Engineering departments don’t have. They’s too busy meeting their commitments. Perspective is what consultants and internal mentors have.

Managing and listening

What makes management difficult for people who are technical experts? In a way, it’s like the reputation that medical doctors have when they are managing their investments — they are so accustomed to being the ones who “know” they have trouble taking advice from financial advisors. As a result, docs are reputed to be have the worst record as self-managed investors.

I can sympathize. As a technologist, I tried managing my own investments over a long period of time. Eventually, I realized that I “knew too much” about the technology and the companies as technology sources. So I would invest in companies that had great technology, but they would turn out to have poor businesses or inadequate marketing — things I didn’t recognize.

Moving from technical contributor (engineer, programmer, analyst, etc.) to manager is another difficult transition in which the contributor is accustomed to “knowing.” As a resource for others on technology, we’re used to being the authority. So the first thing we have to learn as managers is humility.

Actually, the first thing we need to learn is that management is an honorable profession with its own set of objectives, methods and styles. Our training is in “hard” sciences and technology, so we’re rarely prepared to deal with the “soft” stuff of people interactions, influencing, leading, and communications. So let’s be clear: there are a lot of new things to learn about.

Since we tend to manage our interactions by intuition and by reference to our upbringing, most of what we do as managers is not conscious: we don’t see it as skills, but temperament. Believe me, however, you CAN change your interactions. The keys? Being interested in becoming effective as a manager. Becoming aware of the effect we are having on people. Being willing to listen to feedback. Being willing to listen.

Being a manager is all about dealing with non-quantitative stuff. Let the MBAs bring out their spreadsheets. When we need to do quantitative measurements, we’ll have plenty of expertise with the methods and the tools. What we need is a willingness to listen, learn and improve.

Improve what? How do we measure ourselves as managers?  We’ll address this question in a future blog post.