Agile as a corporate strategy to breed mediocrity and commoditise software developers
Agile is a methodology of flexible incremental change with rapid feedback and a good way to tackle wicked problems, those problems that resists solution by revealing new facets every time an interim solution is formulated. It does this by breaking a task into smaller tasks, eating the elephant one bite at a time and responding to change and new information in a flexible manner.
Agile has been criticised on a number of grounds [1,2] but most of these criticisms are related not to the problem solving methodology above but to the management process associated with it, for example that involving Atlassian's JIRA, and the way company cultures or even individual managers implement the process.
Here “Agile” will refer to the way it is implemented rather than the sensible problem solving process above.
Agile can be implemented to foster creativity, and develop developers or to destroy creativity, stifle innovation, breed conformity, foster mediocrity and prevent developers from developing anything more than their knowledge of the technologies used by their organisation.
In most companies Agile is applied only to Software development teams: not Architecture, not Operations, not Sales and Marketing, not business analysts, not individuals and definitely not Management, just Development and is used by managers to speed up delivery and track performance.
When implemented in ( perhaps subtly) dysfunctional organisations, then by an obvious extension of Conway's law it reflects and amplifies both the dysfunctionality and the company culture. One such dysfunction is the worship of “The Team” and the resultant commoditisation of workers especially developers.
Agile can trap capable and senior engineers in an endless treadmill of user stories, digital paperwork, email tsunamis, scrum meetings and a position of endless juniority, even if they are tasked with planning work for the rest of the team. And the better the seniors plan the longer other team members remain junior. Architecture, R&D, perhaps the most frustrating exciting and fulfilling aspect of working in IT (or any other field), Sales, Marketing, product development and Management in General do not lend themselves to the standard Jira focused atomisation of work and baby talk description of tasks that developers experience.
Poor Agile can result from poor management, poorer company culture, negative factors in the software industry and a militaristic top down control culture of business in general.
Finally Agile can give developers extra reasons to consider themselves an elite. Just as 80% of drivers consider themselves better than average, though statistically truly elite developers will be rare and many teams will be composed entirely of mediocre developers. When mediocre people believe themselves to be experts they become dangerously dogmatic. When others believe them experts risky and dangerous ideas can take root and damage a company badly. When true experts consider themselves mediocre ( and the more you know the more you realise how little you know) ideas are considered in depth before implementation and the major risk is overthinking a problem.
The historical trend towards teams
Throughout history there has been a trend away from individual expertise to a team based mode of operation from the Iliad with its hero warriors to the Greek phalanx, from the individual craftsman to the factory and from the recognition that workers are people to their current characterisation as Human Resources. This emphasis on the Team tends to commoditise developers who are ideally, from the management view point, interchangeable with none having any specific expertise and all being able to handle any task, Jacks of All Trades and masters of none. Teams can do a lot more than individuals but have dangers for society, enterprise and the team itself: The herd instinct, groupthink and dominant individuals (alpha males and females) being the most obvious.
Homer writes of heroes: Achilles and Hector, who fought like demons in the scrum of battle mowing down enemies like grass. Later the Greeks invented the phalanx, a shield wall with spears sticking out of it that depended on teamwork and the age of heroes was over. The Romans later converted the phalanx into the Testudo or Tortoise and teams of soldiers led to the gradual obsolescence of the medieval knight once firearms came to dominate the battle field.
Later in the 18th and 19th centuries when Mercantile Capitalism gave way to Industrialism corporations were formed with hierarchies that mirrored those of the military. Factories where each worker, like a well honed Java class, had a single well defined responsibility with a few coordinators to ensure the work was done properly replaced guild workshops where multi skilled craftsmen assembled products on their own. Similar specialisation overtook the white collar areas of industry as separate departments and typing pools arose.
Eventually the Wild West frontiers of the Computer Industry succumbed to the same forces as tasks were simplified and automated with complexity swept under the hood. The Lone Programmer, like the Lone Ranger, became a figure of historical fantasy, at least in the corporate world though they continue to hang on outside, masterless Ronins eking out a living and perhaps envied by those inside the gilded creativity destroying fur lined mousetrap [6] that the modern corporation offers to Information Technologists.
Nowadays the corporate coder must demonstrate they are a team player and demonstrate, or at least fake, the passion of the Lone Programmer, manifesting as long hours, sacrificing personal time, relationships and even health “for the team”, “for the company” and the latest buzzword “To add customer value”.
Problems with teams
Development teams are, at least under Agile, supposed to be composed of individuals who can handle any task. This fulfils management's decades long wet dream of turning workers into commodities, something that started when “Personnel” became “Human Resources”. The idea that teams could comprise specialists who might flounder outside their comfort zone, or heaven forbid, might have expertise not relevant to the team but to the wider enterprise, is often taken as a dysfunction within the team and the team member.
Perhaps more importantly a team, like the Greek Phalanx, has no room for heroes. If anyone can tackle any task all are equal and statistics tells us they will all be close to average while the Dunning-Kroeger effect tells us they will all think they are geniuses. Even if they are a carefully hand picked elite (and the jury is still out on what defines an elite developer) they will all have to submerge their ego: and developers, having been treated as superior for decades, have a lot of ego. The solution to this is for each developer to have one or two areas of expertise, thus preserving their ego which goes against the idea that anyone in the team can pick up any task.
In practice some team members become local experts in a particular subset of the team's activities while tackling less demanding tasks outside their comfort zone. The best then leave in order to concentrate on their speciality leaving a team with somewhat less collective competence.
Agile specific problems
Agile as implemented tends to reinforce hierarchical structures in an organisation, with the developers at the bottom, outranked by an entire management structure and that amorphous if not protean entity known as “The Business” a mixture that includes Business analysts, project managers and customer relations managers. Agile increases the feedback frequency between developers and their customers and prevents them going astray – assuming the customers know what they want, which is never a safe assumption. In terms of Conway's laws expanded to take the communication structure of an organisation as including its management structure Agile mirrors an organisation that is dysfunctional, perhaps subtly so with the workers on whom it all depends at the bottom. They have to work faster but often have no formal power other than perhaps vetoing a suggestion by pointing out it is unworkable.
Agile often leaves developers capable of more trapped in a junior role with no easy way to move up either on a technical path, as architects, or on a managerial path where their technical skills might allow The Business to avoid asking for provably impossible features each faster than the other ot into architecture or (my dream job) research.
Church [1] claims there is no place for a senior engineer in a Scrum team. Some organisations use senior engineers as technical leads planning and coordinating work while remaining in a Scrum team. It is usually better for the senior people to be separate from the team, though free to contribute to development if they desire. This lets the senior people look a few months ahead, something generally left to management.
Church also claims Scrum and Agile should be short term tactics for emergency. In support of this I have worked in a number of companies where productivity was high and targets met but neither Agile nor Scrum were in place. Against this is the view that software development is a wicked problem and Agile is a good way to tackle such problems. If that is the case however an Agile like methodology should be employed in scientific research and in political decision making.
The Wrap
Agile is regarded as a good way to tackle the wicked problem of software development where planning an entire project in advance is impossible. Development has to switch between exploration and evaluation to straightforward development [4] in a process reminiscent of real scientific research (which however manages to avoid tickets and Scrum).
The Agile Manifesto is framed in terms of Software development and Business. It is thus embedded in corporate culture and the problems many attribute to Agile are actually problems of Company culture, the culture of the software industry and the dysfunctions in Business culture. When Agile meets Business red in tooth and claw it tends to be implemented in a way that destroys creativity, forces mediocrity on developers and prevents the more experienced moving up into areas such as architecture and research.
When the Agile manifesto is examined without the distorting lens of software development and business it shows a strong resemblance to academic research, though the need for funding has made riskier research less easy to pursue, just as Agile has made it harder to pursue risky development.
References
-
The Fur lined mousetrap : C. Northcote Parkinson, Law Book Co of Australasia; New edition edition (20 Sept. 1983), ISBN-10: 0863600077, ISBN-13: 978-0863600074
