In 2001, a group of technologists and software developers with unconventional theories about how software development ought to be managed met at Snowbird ski resort to try to commit some of those concepts to paper. The result was The Agile Manifesto—a deceptively simple document that has come to redefine the orthodoxy of software development. Agile software development has become the new global standard for software teams within organizations—Facebook, Amazon, Apple, Google and Netflix have all created internal development processes that conform to the basic concepts of the manifesto. Considering the sheer scale and social impact of its corporate adherents, Agile is easily the single most influential perspective on software development ever formalized. However, Agile is an ideology—a normative system of values and beliefs that has been almost completely assimilated into the software industry. Accordingly, the software industry presents an intriguing case study on the nature of ideological monism and the challenge of reconciling the nominal objectives of an ideology with their realization in practice.
In essence, Agile was a rebellion against the corporatization of the software development process, the first true recognition that software development was a complex and often mysterious process that should necessarily be exempt from the corporate tendency towards bureaucratization. Change, reinvention, flexibility, dynamism—these are the basic themes of the Agile manifesto. They have proven immensely compelling: one global survey indicates that around 97% of organizations are employing agile development principles in some form. This ubiquity has resulted in the near total nominalization of software development management theory—the term agile has come to describe the ideology, work methods and even the systems used to develop software in the modern organization. Agile is even disseminating beyond the software team, and is increasingly utilized by support functions such as finance and human resources. Evaluated as a generalized management theory, Agile has proven extraordinarily accessible and popular—despite a dearth of empirical support for its effectiveness and utility.
Interestingly, the Agile Manifesto did not attempt to articulate any specific work methods, rules, processes, systems or structures that should be utilized to develop software in an agile way. This is not surprising, as the Agile Manifesto was never intended to dictate a detailed means of achieving the aims of the manifesto. This apparent lack of direction did not limit the popularity of Agile: in fact, the rapid growth in demand for specific Agile tools and methods has resulted in the creation of a meta-industry for Agile resources, feeding the adoption and industry penetration of the Agile ideology and its derivatives. The most salient resources are rigorously defined Agile methodologies (such as Scrum and Kanban—i.e. detailed processes one must follow in order to implement the principles of the Agile Manifesto) and the specialized software platforms designed specifically to support Agile software development. The Australian technology company Atlassian sells a variety of such products, intended to support the Agile software development process, most notably Confluence™ and Jira™—products which have become the de facto software industry standard. To those outside the technology community, such products come across as rather arcane—a number of explanatory articles appeared before and after Atlassian listed on the NASDAQ, attempting to explain what exactly Atlassian was selling and why the company itself had such a large market capitalization.
As with the Atlassian software, the jargon that surrounds the Agile process and day-to-day Agile work methods has also become increasingly esoteric. Agile practitioners talk of sprints, Kanban boards, burndown charts, velocities, user stories, epics and retros—words that often change meaning depending on context and that may be affiliated with one or more explicitly defined Agile methodologies. Predictably, as the complexity of the Agile ecosystem increases, so too does the number of specialized consultants required to help make sense of it all. Bain & Company has over 1000 Agile practitioners available for clients—perhaps the most reliable measure of how lucrative the Agile consulting industry has become. Yet, if the Agile Manifesto is as simple as it seems, how is it that so many consultants are required? Has any of this resulted in measurable improvements to the quality and efficiency of the average technology company?
Despite the jargon, the specialized tools and the enormous body of resources available to anyone wishing to practice Agile software development in their own organization, it often remains unclear as to how accurately Agile has been implemented in practice—that is, implemented according to the will and intent of those who authored the Agile Manifesto. The Agile Manifesto was deliberately and necessarily abstract—and this seems to have resulted in the gradual perversion of the Agile methodology, and, by extension, of the management culture of the software industry itself. An enormous edifice has been constructed on what appeared to be simple foundations—a mechanism that has proved incredibly disappointing to those who laid the foundation in the first instance. Furthermore, the enduring popularity of Agile has rendered many of those who lack formal Agile qualification or experience less employable than their ostensibly Agile-proficient peers—there are numerous professional advantages available to those who claim to understand Agile work methods, a reality that encourages conformity over any attempts to challenge the dominance of Agile or seriously question its efficacy.
One of the original authors of the Agile Manifesto, Andy Hunt, laments the fact that the abstract nature of the original Manifesto has resulted in the creation and proliferation of endless context-free rules, which purport to form the basis of Agile software development. Over time, such rules become codified into specific methodologies and are followed without consideration of the original guiding principles of the Manifesto. Put another way, Agile is an incredibly difficult ideology to learn, internalize and practice—so some individuals rely on rigidly defined rules or heuristics that represent themselves as Agile, and then proceed to conflate following these rules (often outside their intended contexts) with practicing Agile according to the aims of the Manifesto. Instead of working to gradually refine a software development process, most organizations fall into the trap of assuming that the process should remain constant, foregoing incremental improvement in favor of squeezing the most possible work out of developers operating within largely arbitrary and rigidly defined frameworks. Organizations that fail to obtain any real benefit from Agile (of which there are many) consistently gravitate towards monitoring the implementation of some Agile process instead of the more nebulous and important process outcomes—i.e. the delivery of working software.
The rise of Scrum and Kanban represents at best an attempt to formalize and disseminate the Agile ideology—but at worst these methodologies become nothing more than additional bureaucracy, imposing further arbitrary rules and measures upon developers, for reasons that are often entirely empirically unfounded. Mediocre managers, consultants, developers and even organizations thrive under such conditions—a relentless focus on the nominal rules of the ideology becomes easier, and therefore preferable, to pursuing its actual aims. The software industry, in general, has become obsessed with measuring the inputs and outputs of Agile at the employee level—an obsession that has resulted in the neglect of the original Agile ethos, and a misguided focus on employee-level work statistics instead of process improvement on the organizational level.
The great irony of this perversion is that Agile was intended as a means to free the average software employee from the tyranny of micromanagement and unnecessary bureaucratic oversight—instead, the very essence of the ideology itself has been co-opted and become something unrecognizable to those who conceived it. More generally, the fate of Agile in the software industry serves as a conspicuous example of ideological perversion—whereby some small, abstract ideology is gradually subverted and distorted over time as its influence grows and attempts to realize it in practice are made.