This article extends Software Engineering Institute’s on Agile software development methodology.

SEI attempts to rigorously evaluate software development practices. Their work shows that incremental gains are possible, but, nothing close to claimed 10x productivity improvements. And, the two greatest risks are failure to communicate effectively and failure to customize methods to business context.

1,300 words, estimated reading time 10 minutes.

Rigorous Evaluation

The Software Engineering Institute (SEI) at Carnegie Mellon University has been studying and measuring software development practices, especially agile methodology, for decades. Unlike most commercial firms, SEI attempts to rigorously evaluate software practices using evidence-based methods.

SEI has quantified the value of various software development practices including: Waterfall, Spiral, CMMI, RUP, Extreme, Agile Methodology & Scrum, Lean, Kanban, TDD, SAFE, Nexus and maybe Agility. As software eats the world, no doubt new practices will emerge as well.

Here are some observations from an SEI leader from a recent IEEE publication on the state of agile methodology and DevOps art.


Nothing close to 10x productivity gains

The idea of a “silver bullet” software development methodology existed since the dawn of Agile software development. I think the “10x” idea came from the difference in productivity of software “stars” compared to more normal humans. Unfortunately, while superstars are always very nice to have, effective teamwork is more important today.

Nobody is reporting or realizing anything close to an order of magnitude productivity gain via any form of Agile. Incremental improvements via continuous learning are simply the norm. Key learnings over the decades at SEI include:

  • Processes with less ceremony have higher productivity;
  • A disciplined, committed and adequately budgeted process team is essential;
  • Agile practices must be customized to a company’s context (more below).

But the most significant category of productivity gain has always been removing “blockers”:

Blockers delay (“accidentally”, in Brook’s usage) a software process’s “essential” activities, such as design, coding, testing, etc.

There’s a good argument that Agile software productivity gains have simply evolved from Moore’s law:

  • Low cost cloud computing means no more constraints on machine time or availability.
    Internet has enabled large-scale, global, open-source projects creating reusable software components at low cost.
  • Cheap computing makes Containerization cost effective, so individual developers can instantiate more realistic environments and better test.
  • Programming languages continue to evolve in ways that improve productivity and portability while eliminating common bugs.
  • Cloud computing allows infrastructure to be configured and operated as software (DevOps).

As system operators realized that cloud infrastructure can be manipulated as software and data, they learned to gain productivity via DevOps automation: “Never do the same manual operation more than twice without automating.”

More broadly, DevOps can be defined as practices intended to reduce time from committing a system change to realizing it in production. By improving cycle and delivery times, DevOps certainly complements Agile to improve productivity.

DevOps can also reduce errors by improving the communications flow from development to operations via defined artifacts (errors always result in costly rework).

But an examination of documented software process failures singles out two broad and enduring challenges:


  • A Failure to Communicate
  • A Failure to Customize to Context

Failure to Communicate

Software processes orchestrate the essential activities of software development performed by people with different roles and responsibilities and inter-dependencies.

Especially pernicious are the essential problems of language-based requirements. Software is now a team sport, making the issues of language — verbal, written and formal — most important to productivity gains.

Frequent verbal on- and in-task communications helps productivity, hence agile methodologies emphasize co-location while new digital communications reduce the effects of time and distance.

But software development repeatedly demonstrates, to paraphrase the Talking Heads, that words alone, like facts, won’t do what we want them to. Communications between tasks or software roles, especially over time and size of team, requires written communications and transfer of data.

Ambiguous requirements may be the most essential problem of software without a good solution. Requirements Engineering is still a nascent discipline after decades because most firms cannot afford the time and training required.

After informal or ambiguous requirements, poorly written or missing handoffs between task performers in different roles are the most common source of error, rework and productivity loss within an Agile Methdology process.

Agile is now a “team sport” performed by humans. And that means the fundamental issues of human relationships will dominate.

Temperament and character will have a considerable impact. Managers can most effectively improve productivity not by automation and tooling, which have their place, but by building effective organizational cultures optimizing engagement and teamwork.

There’s no substitute for good people management. Multiple studies of sprint planning show that the manager in charge is the best source of estimation, not story point breakdowns, etc.

Failure to Customize to Context

Another SEI learning is that no Agile or DevOps software process should be applied without customization to a firm’s industry, business model, location, culture, workforce and history.  

One size process does not fit all.  Our articles about “Agile Methods — Keep This, Spike That” were written to make this essential point.  Reducing ceremony and unnecessary writing is worthwhile. But an “ad-hoc cherry-picking of activities”, hoping for design to emerge, is a recipe for failure.  And no Agile method can be productive if owners will not make binding decisions and  insist on changing their requirements frequently.

Much of failure literature comes from industries that must make public their failures, such as avionics, finance, command and control, healthcare and information security.   Not surprisingly, these are the industries with the most rigorous non-functional requirements, either for purposes of safety or government regulation. Neglecting architecture practices and testing are the most common root-cause.

Another root-cause is treating a software process independent of component life cycle.  E.g., the processes and methods for creating a new software system may be quite different from those for customizing an off-the-shelf component, or maintaining the system as it reaches end of life and decommissioning. 

Over time, most IT shops fight to spend more than a small fraction of their budget on innovation versus “keep the lights on” activities.  Underestimating the lifetime cost of even a relatively minor piece of software once it goes into production can be a huge source of productivity loss. 

The customize-to-context challenge is perhaps best viewed through the point of view of non-functional requirements (NFR).  NFRs are about system qualities, not functions it performs. The NFR categories we use attempt to capture the whole context of a software project, which helps our clients customize their Agile and DevOps processes while minimizing ceremony:

NFR Categories and Domain Experts

Category Description Primary NFR Content Provider
BUS NFRs determined by Business Business Product Manager
ARCH General Architecture Solution Architect
OPS Operations and Support Business Product Manager
SDLC System Development Life Cycle Development or Project Manager
ENG Platform Engineering Infrastructure Architect
DM Data Management Data Architect
CAP Capacity/Performance Business Product Manager
BC Business Continuity Business Product Manager
MW Middleware Architecture Solution Architect
NET Network Engineering Infrastructure Architect
STO Storage Engineering Infrastructure Architect
SEC Security Security Architect
ENV Environment Management Infrastructure Architect
SCRM Source Configuration and Release Management Development or Project Manager
QA Quality Assurance Development or Project Manager
ERM Enterprise Release Management Business Product Manager
DOC User Documentation Business Product Manager
UX User Experience Business Product Manager

A Business Process Like Any Other?

One of the Agile Manifesto’s authors has recently written that perhaps Agile is dead, and that we should instead emphasize ‘Agility’. The Manifesto was never intended to be prescriptive, but a collection of best practices to choose from, marketed well by a few thought leaders.

Perhaps the greatest indication that Agile has eaten the world is that its most enduring problems are now general management issues, not narrowly software development issues.

Communications, culture, teamwork, engagement, motivation, business context and industry knowledge — these are issues all managers deal with to raise productivity, not just Agile.

Only when the human element is removed might things change — deep learning / AI anyone? Just likely not anytime soon.

About the Author

David Ward

David Ward


David Brian Ward’s career spans 35 years of experience as an entrepreneur and executive leader of software innovation at firms large and small. Mr. Ward founded Telegraph Hill in 2010 to provide leading edge technical management consulting and onsite software development teams to fast-growing firms in the San Francisco Bay Area.

“The more virtual our social relationships, the more overwhelmingly important onsite physical presence is. That’s why face to face teams are a core principle of an agile methodology.”