In 2001, a group of technologists and software developers with unconventional theories about how software development ought to be managed met at Snowbird ski resort to try to commit some of those concepts to paper. The result was The Agile Manifesto—a deceptively simple document that has come to redefine the orthodoxy of software development. Agile software development has become the new global standard for software teams within organizations—Facebook, Amazon, Apple, Google and Netflix have all created internal development processes that conform to the basic concepts of the manifesto. Considering the sheer scale and social impact of its corporate adherents, Agile is easily the single most influential perspective on software development ever formalized. However, Agile is an ideology—a normative system of values and beliefs that has been almost completely assimilated into the software industry. Accordingly, the software industry presents an intriguing case study on the nature of ideological monism and the challenge of reconciling the nominal objectives of an ideology with their realization in practice.
In essence, Agile was a rebellion against the corporatization of the software development process, the first true recognition that software development was a complex and often mysterious process that should necessarily be exempt from the corporate tendency towards bureaucratization. Change, reinvention, flexibility, dynamism—these are the basic themes of the Agile manifesto. They have proven immensely compelling: one global survey indicates that around 97% of organizations are employing agile development principles in some form. This ubiquity has resulted in the near total nominalization of software development management theory—the term agile has come to describe the ideology, work methods and even the systems used to develop software in the modern organization. Agile is even disseminating beyond the software team, and is increasingly utilized by support functions such as finance and human resources. Evaluated as a generalized management theory, Agile has proven extraordinarily accessible and popular—despite a dearth of empirical support for its effectiveness and utility.
Interestingly, the Agile Manifesto did not attempt to articulate any specific work methods, rules, processes, systems or structures that should be utilized to develop software in an agile way. This is not surprising, as the Agile Manifesto was never intended to dictate a detailed means of achieving the aims of the manifesto. This apparent lack of direction did not limit the popularity of Agile: in fact, the rapid growth in demand for specific Agile tools and methods has resulted in the creation of a meta-industry for Agile resources, feeding the adoption and industry penetration of the Agile ideology and its derivatives. The most salient resources are rigorously defined Agile methodologies (such as Scrum and Kanban—i.e. detailed processes one must follow in order to implement the principles of the Agile Manifesto) and the specialized software platforms designed specifically to support Agile software development. The Australian technology company Atlassian sells a variety of such products, intended to support the Agile software development process, most notably Confluence™ and Jira™—products which have become the de facto software industry standard. To those outside the technology community, such products come across as rather arcane—a number of explanatory articles appeared before and after Atlassian listed on the NASDAQ, attempting to explain what exactly Atlassian was selling and why the company itself had such a large market capitalization.
As with the Atlassian software, the jargon that surrounds the Agile process and day-to-day Agile work methods has also become increasingly esoteric. Agile practitioners talk of sprints, Kanban boards, burndown charts, velocities, user stories, epics and retros—words that often change meaning depending on context and that may be affiliated with one or more explicitly defined Agile methodologies. Predictably, as the complexity of the Agile ecosystem increases, so too does the number of specialized consultants required to help make sense of it all. Bain & Company has over 1000 Agile practitioners available for clients—perhaps the most reliable measure of how lucrative the Agile consulting industry has become. Yet, if the Agile Manifesto is as simple as it seems, how is it that so many consultants are required? Has any of this resulted in measurable improvements to the quality and efficiency of the average technology company?
Despite the jargon, the specialized tools and the enormous body of resources available to anyone wishing to practice Agile software development in their own organization, it often remains unclear as to how accurately Agile has been implemented in practice—that is, implemented according to the will and intent of those who authored the Agile Manifesto. The Agile Manifesto was deliberately and necessarily abstract—and this seems to have resulted in the gradual perversion of the Agile methodology, and, by extension, of the management culture of the software industry itself. An enormous edifice has been constructed on what appeared to be simple foundations—a mechanism that has proved incredibly disappointing to those who laid the foundation in the first instance. Furthermore, the enduring popularity of Agile has rendered many of those who lack formal Agile qualification or experience less employable than their ostensibly Agile-proficient peers—there are numerous professional advantages available to those who claim to understand Agile work methods, a reality that encourages conformity over any attempts to challenge the dominance of Agile or seriously question its efficacy.
One of the original authors of the Agile Manifesto, Andy Hunt, laments the fact that the abstract nature of the original Manifesto has resulted in the creation and proliferation of endless context-free rules, which purport to form the basis of Agile software development. Over time, such rules become codified into specific methodologies and are followed without consideration of the original guiding principles of the Manifesto. Put another way, Agile is an incredibly difficult ideology to learn, internalize and practice—so some individuals rely on rigidly defined rules or heuristics that represent themselves as Agile, and then proceed to conflate following these rules (often outside their intended contexts) with practicing Agile according to the aims of the Manifesto. Instead of working to gradually refine a software development process, most organizations fall into the trap of assuming that the process should remain constant, foregoing incremental improvement in favor of squeezing the most possible work out of developers operating within largely arbitrary and rigidly defined frameworks. Organizations that fail to obtain any real benefit from Agile (of which there are many) consistently gravitate towards monitoring the implementation of some Agile process instead of the more nebulous and important process outcomes—i.e. the delivery of working software.
The rise of Scrum and Kanban represents at best an attempt to formalize and disseminate the Agile ideology—but at worst these methodologies become nothing more than additional bureaucracy, imposing further arbitrary rules and measures upon developers, for reasons that are often entirely empirically unfounded. Mediocre managers, consultants, developers and even organizations thrive under such conditions—a relentless focus on the nominal rules of the ideology becomes easier, and therefore preferable, to pursuing its actual aims. The software industry, in general, has become obsessed with measuring the inputs and outputs of Agile at the employee level—an obsession that has resulted in the neglect of the original Agile ethos, and a misguided focus on employee-level work statistics instead of process improvement on the organizational level.
The great irony of this perversion is that Agile was intended as a means to free the average software employee from the tyranny of micromanagement and unnecessary bureaucratic oversight—instead, the very essence of the ideology itself has been co-opted and become something unrecognizable to those who conceived it. More generally, the fate of Agile in the software industry serves as a conspicuous example of ideological perversion—whereby some small, abstract ideology is gradually subverted and distorted over time as its influence grows and attempts to realize it in practice are made.
How can you have an article about Agile which doesn’t once mention the word ‘test’?
SCRUM is NOT Agile!!!! It is anti-Agile. It is a RIGID methodology that imposes process over and above people and interactions. The only “Agile” aspect of SCRUM is that it facilitates responding to changing requirements.
My experience with Agile over several companies has left me with the impression that “Agile” is very like “Communism”: the theory becomes very endemic, and sometimes pathological, in practice. I did not know it started with a “Manifesto”… but there it is.
The reason, in my view, for the pathology, also echoes those of the famous communist attempts. It is a revolutionary ideal, and as such is guaranteed to disrupt management hierarchies. This is not the preference of the existing higher levels of management, therefore they will roll out the implementation in such a way as to preserve the incumbent power and authority. I’ve worked for Stalinist Agile teams, and I’ve worked for Maoist Agile teams, but never Agile teams.
This article’s description of the “Agile Manifesto” reminds me of my early days of software development back in the late 70s and early 80s, and the impact that Stallman’s GNU Manifesto had on the communities at that time – and the eventual slow creep of commercialism back into the licensing of “open source” in the past 20 years.
One last piece of advice. For any manager out there who struggling with change in scope and planning for it you can use scrum because that’s what it’s designed for or another flavor of agile. But that does not necessarily mean that you cannot use waterfall just establish a clear an easy change order process.
Hmm, so you said Scrum but called it a methodology but it’s not that. It is a framework that I’ve used many times to do the opposite of what you say. It weeds out bad developers… There is constant inspection of progress towards a Sprint goal (many people forget to establish one) if developers aren’t doing his or her part the team knows because they aren’t moving the needle. The team is taught to say hey he’s not pulling his weight can we fix this ? Should we remove him? (Or her) that is taught in the scrum framework. The issue with scrum is that people try to only use it partially which doesn’t work. (We made it our own) it’s like using a software framework for one little feature or using a framework incorrectly most of the time the outcome is bad. “I installed redux but I’m not going… Read more »
This is not surprising, as the Agile Manifesto was never intended to dictate a detailed means of achieving the aims of the manifesto …. The most salient resources are rigorously defined Agile methodologies (such as Scrum and Kanban—i.e. detailed processes one must follow in order to implement the principles of the Agile Manifesto)
So we have ‘never intended to dictate’ then ‘rigorously defined’. I’m a bit confused, but then again it’s been decades since I did much coding.
Cameron, this is a brilliant article! I am glad someone out there is sharing my frustrations, which is good because then only we can bring the focus back to the right behaviours expected out of organisations, leaders, and all individuals involved. Great work.
Agile is a theory. Like every theory that is put into practice, it will have a manifested thesis – which it already has – that people will develop several antitheses either by theory or practice, which will eventually morph into a new synthesis. Get used to it.
This is disinformation
Software development is difficult and remains so. The reason is that software allows the creation of complex systems relatively easily. Competition means that systems are always being developed which ar emore complex, with the minimum resource sin the least time possible. Complexitiy is almost always the limiting factor. As better tools, components and processes become available the systems we develop become more complex or we develop them quicker in less time. As a result many software development projects are late and are disappointing in the quality or reliability of the result. This has been true for at least my 40 years of experience. In this environment there is a great desire for a solution, a magic bullet, that makes software development easy and reliable. Magic bullets are proclaimed as arriving regularily, have fanatical cult like supporters and fade away when the limitations become apparent and the next one comes along.… Read more »