Dynamics of Software Development

The recent “failed IPO” of BATS has to be a cautionary tale. This wasn’t just a deal that didn’t price or trade according to plan. A software bug caused it to be withdrawn. You don’t hear that one too often.

A bug killed an IPO?

There is no argument that we live in a world of staggering speed, where competitors race to meet customer needs and time to market matters. Innovation is always factored by the ticking click, who gets the jump and the competitive advantage, when a cost center becomes a profit center. Information compounds on our desktops, the team with analysis paralysis most often loses to the nimble risk takers—but all this means is that in product development, the role of Quality Assurance (QA) has never been more critical.

I have often heard the mantra from development teams: “Better, Faster, Cheaper—we can give you any two and a half.” Believe me, I understand trade-offs. All product development is tempered by tough decisions that incorporate a series of smart and well-balanced quid pro quos. You want to cut the budget, give us more time or expect fewer features. You want to tighten the schedule, give us more capital or reduce the scope of benefits. You want an industry defining product, show us the money or don’t ask for a date.

Surely these threads have become clichés, and as such, they are not without some underlying truth. There are even schools of thought that proclaim speed over accuracy is the game-winning formula, entire companies built on this premise, hugely successful in their own right. Yet when the Decision Maker, whoever that is, makes the call to greenlight a software or product release, another question comes to mind: Is the call transparent or opaque? Said another way, are the risks inherent in the release from staging to live known to the Decision Maker, or is that person flying blind?

If the release is going live with known issues, that becomes a business decision with acknowledged acceptable risk. If a showstopper issue exists but is unknown to anyone, well, I don’t think I have ever seen that case in a going concern. If the release is going live with issues known to others but not the Decision Maker, that is a dysfunctional process, possibly the beginning of the end.

Here is the way I like to think about quality in product development: Quality Assurance is a Process, not a Department.

Like so many of the great lessons I have co-opted in this blog, this first became clear to me in hard-won experience with the magnificent QA Directors with whom I have worked over the years (several of whom reviewed this post in draft prior to publication), and second in Jim McCarthy’s brilliant book Dynamics of Software Development first published in 1995 and still a must read for any of my teams—non-tech even more than tech staff. The most critical constant of which I am aware in delivering great products to market consistently is for Quality to be owned by everyone involved in innovation—from designers to developers to marketers to feedback from end-users.

Of course every great development company will have a final step in the process called Quality Control or Quality Assurance, but it is my sense that the QA formal group is there to be the standard-bearer for Quality and rally the company around it, putting a final go or no-go procedure in place before the world gets its hands on a product, but not accepting proxy status for an otherwise poor process. A QA department is not a dumping ground, not a remote server where code is parked as a step function or convenient checkpoint in a perfunctory release approval, not a cynical target of blame. QA is the proxy for the customer, not management, and as such must have a voice that is shared throughout a company. If a Decision Maker chooses not to listen to either the process or a warning from fully objective and independent QA stewards, you get what you get.

I have always been enamored with QA teams, for their passion, for what they teach me, for how much they care about excellence. When QA is wound into the culture of a company, it is often because of the mutual and shared respect an organization has for the value of Quality as an intrinsic good that will most likely yield extrinsic rewards, but carries reward for itself in the form of realized creativity and pride. It is very hard to fake a love of Quality, and this applies to much more than software. Quality is a path to premium brands cautionary tale and premium prices in a landscape where speed and disseminated knowledge can commoditize just about anything if you let it. Quality is won when it is broadly embraced as a shared value, and then championed by a high energy team that inspires its adoption at the highest levels of management and all through the ranks. If top management does not buy this, Quality is doomed.

If top management at BATS did not know about the bug in their system—a software platform for trading equities like their own on IPO day and beyond —they did not do the hard work that is expected of them and now accept the business consequences. The downside illustrated in this real world example of an incomplete process is about as clear as it can be. To ignore or be ignorant of a showstopper in one’s own product is a reflection of a process that needs to be re-engineered. When you’re working with world-class engineers, it is much easier and far more fruitful to make sure the process is engineered correctly before the products go through it. The material cost of discovering a bug early in development is a tiny fraction of what it can cost you in the hands of the public. Give your engineers a voice and they will save you every time.

Listen to your QA stewards. If you built the right team, they are your first line of offense and your last line of defense. They know of what you speak.

Following up my last post on the scarcity of successful internet brand turnarounds, I had a number of interesting discussions with colleagues in search of the answer why. While it would be impossible for me to boil that down to a single concept in a single blog post, the common theme seemed to be the extraordinary difficulty in reinventing the key products or services under a once a powerful but now fading brand, such that the new offerings were up to the standards of the original breakthrough experiences.

Digging deeper into this theme of why a brand that was created of innovation could have such a hard time finding reinvention in subsequent cycles of innovation, the culprit fueling failure often found the compass needle pointing at process. Where some form of good process once allowed staggering creativity to flourish, failure to reinvigorate good process most often led to uninspired product development, lackluster offerings to customers, and ultimately a continuation of the downward spiral where a turnaround was the hope. Called out for especially pernicious result:

1) Tepid Innovation—believing that a little idea was a big idea almost in desperation for lack of identifying a big idea. Copycat products that responded to market leaders to segment small bits of their commanding market share consumed energy where market leading ideas remained in small supply.

2) Lack of shared vision—weak leadership failed to articulate a big idea around which teams could rally, so buy-in became compelled rather than organic. Empowering strong leaders to lead is not often enough a company’s core competency, because truly creative rebels don’t want to be managed, they want to be sponsored, so that they can make change happen in cultures that prefer conformity, and conformity is not how you win market share. When the right leaders are chosen and empowered, they can build a shared vision by leading, not managing.

3) Corporate intervention—a young company acquired for its Think Different mentality and bold new ideas becomes indoctrinated in the prevailing corporate culture of the parent, and begins to see the world through the lens of the acquirer rather than for the reasons it was acquired.

4) Lack of listening—there is no longer a measurable correlation between time on the job and quality of concept. The youngest arrival in a company may just have the best ideas, but if there is no forum for real listening and discussion, the most creative voice can be too easily silenced.

5) Marketing/Engineering Wars—rather than partner, the two necessary sides of the winning formula argue for the sake of winning the argument. They forget what it means to win—that their competition works at another company.

Many of these themes are explored in the brilliant and surprisingly nontechnical book Dynamics of Software Development by Jim McCarthy, which I have encouraged every senior executive I have met to read as well as every staff leader I have mentored. In my experience the core issue comes down to learning how to build a consensus, understanding that consensus building is not polling or voting or majority rules. Product development is an expertise, like any other profession. Many individuals can learn to do it adequately, but few can do it extraordinarily.

Look at some of the new wave of great companies and you see the top-tier of product development pros at the helms: Mark Zuckerberg, Reed Hastings, Elon Musk, Reid Hoffman. Of course those are all CEOs of substantial, important, disruptive companies and they are not available for brand turnarounds at internet companies on the rebound, but the question remains, are the people being put in charge of turnarounds somehow similar in nature and characteristics to the great leaders of product development? Are they able to articulate a vision around which people can rally willingly and with trust? Do they listen to a multitude of opinions and then make decisions that incorporate feedback because it is critical, not because it represents a political agenda? Do they have good taste, and can they see beyond the state of today to leapfrog a competitor with perhaps something that didn’t do well in a focus test? Have they built a process in their environment where good work can shine and fast iteration can overcome mediocrity in rapid succession?

Consensus is an astonishingly complex concept; it is not at all compromise. It begins with vision, It is evangelized through leadership, It becomes stronger through group participation and feedback, but it is guided to completion by the same vision and leadership from which it emanated. Consensus goes off track when a leader feels for whatever reason that bits of all contributions most be included to create the consensus, the proverbial camel with two humps. That is wrong, because all comments and critiques will not be right if breakthrough products are your goal. Group participation is a must to achieve organic buy-in, listening is a must for a leader to bypass being feared as a despot, but for great product development to triumph, a vision must remain strong, pure, unique, original. That can never be taken for granted, and everyone on a team will not always be happy at every twist and turn. Yet if you have ever had the privilege of working on a world-class, game changing product, you know that most sins are forgiven when it all comes together through good process, and it does not look anything like the system of checks and balances we see in representative government.

Product development is driven by a different motive set, of creative destruction and inspired disruption, a high stakes arena where often winner takes all, second prize means you go home. Democracy does not work there, and democratic compromise is not workplace consensus. Show me a great product, and I know you will find iteration, but I bet you won’t find haggling.

Vision is what launches a brand. Vision will always be key to reinventing a brand. Process is the key to translating ideal vision into working reality. Consensus is the element of process that gets everyone on a team to remain part of the team, because success is owned by the team, not by its individual components. Get most of that right and product development has a fighting chance, and so might reinvention.