Agile and the Software Industry’s Ideology Problem

In 2001, a group of technologists and software developers with unconventional theories about how software development ought to be managed met at Snowbird ski resort to try to commit some of those concepts to paper. The result was The Agile Manifesto—a deceptively simple document that has come to redefine the orthodoxy of software development. Agile software development has become the new global standard for software teams within organizations—Facebook, Amazon, Apple, Google and Netflix have all created internal development processes that conform to the basic concepts of the manifesto. Considering the sheer scale and social impact of its corporate adherents, Agile is easily the single most influential perspective on software development ever formalized. However, Agile is an ideology—a normative system of values and beliefs that has been almost completely assimilated into the software industry. Accordingly, the software industry presents an intriguing case study on the nature of ideological monism and the challenge of reconciling the nominal objectives of an ideology with their realization in practice.

In essence, Agile was a rebellion against the corporatization of the software development process, the first true recognition that software development was a complex and often mysterious process that should necessarily be exempt from the corporate tendency towards bureaucratization. Change, reinvention, flexibility, dynamism—these are the basic themes of the Agile manifesto. They have proven immensely compelling: one global survey indicates that around 97% of organizations are employing agile development principles in some form. This ubiquity has resulted in the near total nominalization of software development management theory—the term agile has come to describe the ideology, work methods and even the systems used to develop software in the modern organization. Agile is even disseminating beyond the software team, and is increasingly utilized by support functions such as finance and human resources. Evaluated as a generalized management theory, Agile has proven extraordinarily accessible and popular—despite a dearth of empirical support for its effectiveness and utility.

Interestingly, the Agile Manifesto did not attempt to articulate any specific work methods, rules, processes, systems or structures that should be utilized to develop software in an agile way. This is not surprising, as the Agile Manifesto was never intended to dictate a detailed means of achieving the aims of the manifesto. This apparent lack of direction did not limit the popularity of Agile: in fact, the rapid growth in demand for specific Agile tools and methods has resulted in the creation of a meta-industry for Agile resources, feeding the adoption and industry penetration of the Agile ideology and its derivatives. The most salient resources are rigorously defined Agile methodologies (such as Scrum and Kanban—i.e. detailed processes one must follow in order to implement the principles of the Agile Manifesto) and the specialized software platforms designed specifically to support Agile software development. The Australian technology company Atlassian sells a variety of such products, intended to support the Agile software development process, most notably Confluence™ and Jira™—products which have become the de facto software industry standard. To those outside the technology community, such products come across as rather arcane—a number of explanatory articles appeared before and after Atlassian listed on the NASDAQ, attempting to explain what exactly Atlassian was selling and why the company itself had such a large market capitalization.

As with the Atlassian software, the jargon that surrounds the Agile process and day-to-day Agile work methods has also become increasingly esoteric. Agile practitioners talk of sprints, Kanban boards, burndown charts, velocities, user stories, epics and retros—words that often change meaning depending on context and that may be affiliated with one or more explicitly defined Agile methodologies. Predictably, as the complexity of the Agile ecosystem increases, so too does the number of specialized consultants required to help make sense of it all. Bain & Company has over 1000 Agile practitioners available for clients—perhaps the most reliable measure of how lucrative the Agile consulting industry has become. Yet, if the Agile Manifesto is as simple as it seems, how is it that so many consultants are required? Has any of this resulted in measurable improvements to the quality and efficiency of the average technology company?

Despite the jargon, the specialized tools and the enormous body of resources available to anyone wishing to practice Agile software development in their own organization, it often remains unclear as to how accurately Agile has been implemented in practice—that is, implemented according to the will and intent of those who authored the Agile Manifesto. The Agile Manifesto was deliberately and necessarily abstract—and this seems to have resulted in the gradual perversion of the Agile methodology, and, by extension, of the management culture of the software industry itself. An enormous edifice has been constructed on what appeared to be simple foundations—a mechanism that has proved incredibly disappointing to those who laid the foundation in the first instance. Furthermore, the enduring popularity of Agile has rendered many of those who lack formal Agile qualification or experience less employable than their ostensibly Agile-proficient peers—there are numerous professional advantages available to those who claim to understand Agile work methods, a reality that encourages conformity over any attempts to challenge the dominance of Agile or seriously question its efficacy.

One of the original authors of the Agile Manifesto, Andy Hunt, laments the fact that the abstract nature of the original Manifesto has resulted in the creation and proliferation of endless context-free rules, which purport to form the basis of Agile software development. Over time, such rules become codified into specific methodologies and are followed without consideration of the original guiding principles of the Manifesto. Put another way, Agile is an incredibly difficult ideology to learn, internalize and practice—so some individuals rely on rigidly defined rules or heuristics that represent themselves as Agile, and then proceed to conflate following these rules (often outside their intended contexts) with practicing Agile according to the aims of the Manifesto. Instead of working to gradually refine a software development process, most organizations fall into the trap of assuming that the process should remain constant, foregoing incremental improvement in favor of squeezing the most possible work out of developers operating within largely arbitrary and rigidly defined frameworks. Organizations that fail to obtain any real benefit from Agile (of which there are many) consistently gravitate towards monitoring the implementation of some Agile process instead of the more nebulous and important process outcomes—i.e. the delivery of working software.

The rise of Scrum and Kanban represents at best an attempt to formalize and disseminate the Agile ideology—but at worst these methodologies become nothing more than additional bureaucracy, imposing further arbitrary rules and measures upon developers, for reasons that are often entirely empirically unfounded. Mediocre managers, consultants, developers and even organizations thrive under such conditions—a relentless focus on the nominal rules of the ideology becomes easier, and therefore preferable, to pursuing its actual aims. The software industry, in general, has become obsessed with measuring the inputs and outputs of Agile at the employee level—an obsession that has resulted in the neglect of the original Agile ethos, and a misguided focus on employee-level work statistics instead of process improvement on the organizational level.

The great irony of this perversion is that Agile was intended as a means to free the average software employee from the tyranny of micromanagement and unnecessary bureaucratic oversight—instead, the very essence of the ideology itself has been co-opted and become something unrecognizable to those who conceived it. More generally, the fate of Agile in the software industry serves as a conspicuous example of ideological perversion—whereby some small, abstract ideology is gradually subverted and distorted over time as its influence grows and attempts to realize it in practice are made.

If you enjoy our articles, be a part of our growth and help us produce more writing for you:
Total
166
Shares

12 comments

  1. My experience with Agile over several companies has left me with the impression that “Agile” is very like “Communism”: the theory becomes very endemic, and sometimes pathological, in practice. I did not know it started with a “Manifesto”… but there it is.

    The reason, in my view, for the pathology, also echoes those of the famous communist attempts. It is a revolutionary ideal, and as such is guaranteed to disrupt management hierarchies. This is not the preference of the existing higher levels of management, therefore they will roll out the implementation in such a way as to preserve the incumbent power and authority. I’ve worked for Stalinist Agile teams, and I’ve worked for Maoist Agile teams, but never Agile teams.

  2. This article’s description of the “Agile Manifesto” reminds me of my early days of software development back in the late 70s and early 80s, and the impact that Stallman’s GNU Manifesto had on the communities at that time – and the eventual slow creep of commercialism back into the licensing of “open source” in the past 20 years.

  3. One last piece of advice. For any manager out there who struggling with change in scope and planning for it you can use scrum because that’s what it’s designed for or another flavor of agile. But that does not necessarily mean that you cannot use waterfall just establish a clear an easy change order process.

  4. Hmm, so you said Scrum but called it a methodology but it’s not that. It is a framework that I’ve used many times to do the opposite of what you say. It weeds out bad developers… There is constant inspection of progress towards a Sprint goal (many people forget to establish one) if developers aren’t doing his or her part the team knows because they aren’t moving the needle. The team is taught to say hey he’s not pulling his weight can we fix this ? Should we remove him? (Or her) that is taught in the scrum framework. The issue with scrum is that people try to only use it partially which doesn’t work. (We made it our own) it’s like using a software framework for one little feature or using a framework incorrectly most of the time the outcome is bad. “I installed redux but I’m not going to u

    Ive had waterfall projects and scrum projects go successful, usually because the process and framework is spelled out and I ensure constant communication and collaboration on the big picture. On my scrum teams they take the scrum open developer quiz and are coached into being collaborative (helps with transparency) and to shift the focus on the overall Sprint goal and avoiding things that could jeopardize it.

    Another thing that PMs do not do well in the beginning in their careers is impediment detection and removal.

    For large multi team projects feature dependency management is important. So the Product owner or Manager needs to organize the backlog and delegate in a way that keeps other teams from stepping on each other’s toes.

    I would say that we shouldn’t blame the processes, or frameworks but rather execution, motivation, transparency and straight up competency. If people want a project to succeed they will do everything in their power to learn the process correctly, communicate and engage their developers or managers to fully understand and create requirements

    Nothing is worse than lack of communication, it leads to silos and very often unvalidated assumptions. “Everything should be in this user story or requirement, nothing could possibly be missing. I’m just gonna work work work and not ask any questions” Fast forward to the end of the Sprint “oh I thought you meant this not that” it’s often times unavoidable no matter how much detail you put in a requirement. Often times it’s information overload. My best workers ask the right questions my worst ones don’t ask any questions.

    With that said I agree with a lot of the identified issues in this article but I can’t blame the processes (scrum or waterfall). Especially if I’ve seen good teams succeed at both ends of the spectrum. People need to learn the processes or frameworks properly and educate their workers on whatever process or framework they choose.

    3
    3
  5. This is not surprising, as the Agile Manifesto was never intended to dictate a detailed means of achieving the aims of the manifesto …. The most salient resources are rigorously defined Agile methodologies (such as Scrum and Kanban—i.e. detailed processes one must follow in order to implement the principles of the Agile Manifesto)

    So we have ‘never intended to dictate’ then ‘rigorously defined’. I’m a bit confused, but then again it’s been decades since I did much coding.

  6. Cameron, this is a brilliant article! I am glad someone out there is sharing my frustrations, which is good because then only we can bring the focus back to the right behaviours expected out of organisations, leaders, and all individuals involved. Great work.

    7
    1
  7. Agile is a theory. Like every theory that is put into practice, it will have a manifested thesis – which it already has – that people will develop several antitheses either by theory or practice, which will eventually morph into a new synthesis. Get used to it.

    4
    5
    1. Agile is a theory? You just show you have no clue about how it emerged. It all comes from people who were managing projects and programs in traditional way, and who got frustrated by related problems. They imagined improvements and tried them, PDCA, and kept repeating. This is empirical continuous improvement based on experimentation. This is how good practices emerge and get known (user stories by Kent Beck, story mapping, Scrum events or Kanban cadences…).

      3
      3
    1. The author: rigorously defined Agile methodologies (such as Scrum and Kanban—i.e. detailed processes one must follow

      The Scrum guide: Scrum is not a process, technique, or definitive method. Rather, it is a framework within which you can employ various processes and techniques.

      Another person who has failed to understand what Agile is and means. Reminds me of the talks about “hybrid agile methods” on PMI website…

      You want to eventually understand it? Maybe leave a bit software and check eduscrum, or watch the videos where Riccardo describes how he introduced Agile and Scrum to manage his restaurant staff, maybe you ll understand better then, I hope.

      2
      5
  8. Software development is difficult and remains so. The reason is that software allows the creation of complex systems relatively easily. Competition means that systems are always being developed which ar emore complex, with the minimum resource sin the least time possible. Complexitiy is almost always the limiting factor. As better tools, components and processes become available the systems we develop become more complex or we develop them quicker in less time. As a result many software development projects are late and are disappointing in the quality or reliability of the result. This has been true for at least my 40 years of experience. In this environment there is a great desire for a solution, a magic bullet, that makes software development easy and reliable. Magic bullets are proclaimed as arriving regularily, have fanatical cult like supporters and fade away when the limitations become apparent and the next one comes along. At their heart each ‘magic bullet’ always has some sensible and useful processes or heuristics which make sense in a paticular context and environment, which are massively oversold and applied as universal panaceas. Massively oversold and applied with poor understanding outside of the context they make sense within they achieve predictably poor results. This leads to the cycle continuing with the next magic bullet.

    Agile is just the same,perhaps more resistent to being discreditted due it’s amorphous indeterminate nature but there are plenty of agile disaster stories and in time it will be replaced by another overhyped development methodology. In the meantime away from the hype and religuous fervor, tools have improved, there is a better baseline, consensus understanding of what does and doesn’t work in software development but the number and complexity of systems continues to grow so the challenge remains.

    14
    1
    1. I too object to the broad claims of Agile universality and significance, in the primary article above. Yes, there has been a large and noisy advocation industry around Agile for a decade or two. But that is far from unique: these “methodologies” with their zealous self-serving advocates, appear, surface, suffer abuse, suffer backlash, and fade away regularly. There are usually several in the process at any moment. I am pretty sure that Agile, to the extend that it has any meaning at all, is well into the downslope of the cycle.

      At any rate, attempts to generalize, from observations of these “movements”, to anything more borad than the actual software industry, are highly dubious, largely irrelevant.

      2
      2

Leave a Reply

Inline
Inline