Is Scrum the ‘popping candy’ of Agile software development?
For the longest time I have been a skeptic of Agile methodologies, despite having actually seen one of them work first hand. Finally, after almost two decades, the almost religious defense of these methodologies has begun to fade and the internet appears willing to talk frankly about the realities of Agile and of Scrum. This has caught my eye, and so I feel it’s time to put my dichotomy to rest.
What is Agile?
In February 2001, the unthinkable happened. A group of seventeen software developers gathered together in a room in a Utah ski resort to discuss software development practices, and agreed with each other! What they came up with was a set of twelve principals of software development, which they felt should guide the practice, and they named these principals “The Agile Manifesto.”
If you ask me, naming these twelve principals a “Manifesto” was a mistake. It’s not that the word is inaccurate, it is not, but it’s a word that adds gravitas to the document. Surely this one word is not entirely responsible for what followed, but I like to imagine it contributed. So what followed the publishing of this manifesto? “The Buzz”
Agile as a Buzz word.
I don’t believe there is anyone in the software industry today that has not come across the word “Agile.” Unfortunately, the word is often heavily miss-used.
As an example, ask a developer about Agile and they may tell you “Unit testing is an essential part of Agile.” – Really? The manifesto does not mention the words “Unit testing” at all. The nearest principal of the manifesto, so far as I can tell, is “Working software is the primary measure of progress.” but this does not imply unit testing. In fact, one of the men that met in that room in Utah, Dave Thomas, has spoken out in public against unit testing as a practice.
For another example of the miss-use of the word Agile, you may be told that “Agile means not documenting anything”, again the manifesto says no such thing! The closest principal in the manifesto reads “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.” This does NOT mean that you should not create design documents, nor does it mean that you shouldn’t write anything down. It’s simply expressing that, as members of a development team work together they will convey information among each other most effectively, by talking to each other. Picture the typical development office with it’s high dividing walls and developers working quietly with their heads down, and you can’t miss the need for them getting together to talk once in a while.
The problem with Agile then, it seems, is that the original intentions of the manifesto have been lost to the marketing buzz, but why? Well, because people love to sell things. The Agile manifesto has spawned a large industry of organizations that want to sell Agile training, Agile certifications, Agile methodologies, books on Agile and software to help you to comply with this wonderful thing called Agile. I’ve heard, as I’m sure you have too, managers proudly proclaim that they conform to Agile practices, that their business is Agile, and that they have a shiny certificate or two to prove it. What transpires in practice under those managers however, is often the same tired old mess of policy that frustrates employees and delivers very little of it’s promise.
Rigidity of Agile.
As I see it, the problem here is actually nothing at all to do with the Agile manifesto, but far more to do with the way people like to think of themselves as professionals. Everyone likes to think that they know what they are doing with regards to their profession, of course they do, it makes them feel secure in their career and proud of what they do. Sadly, I think that this is something of an illusion, and we all know it. Come on, be honest, have you never had a moment in which you questioned your level of proficiency at work? If you answer “no”, this may be a time to reflect.
Project managers, like all of us, like to have a nice set of rigid but simple rules to follow. If the rules are simple, and following them is supposed to lead to success, well then it becomes quite easy to be a confident professional doesn’t it? In practice however, such a set of rules doesn’t exist outside the world of management training programs, and the mindset of believing such a thing does exist leads to some very rigid thinking.
As just one example, I once had a difference of opinion with a project manager over the S.W.O.T analysis system. S.W.O.T stands for Strengths, Weaknesses, Opportunities and Threats. It’s a system for the analysis of business decisions to help guide the decision making process. Essentially, you’re encouraged to write down the strengths, weaknesses, opportunities and threats that affect a course of action, then assign weights to them in order to decide how to navigate the decision. What may not be obvious is that S.W.O.T is actually an elegant way to make any decision. It’s intention is quite clearly to formalize the decision making process, and it’s flexible enough to cover any scenario. Unfortunately, my project manager had been trained on S.W.O.T as part of one of his certifications, and so when I suggested using it to guide a decision which was not related to business strategy, he insisted that this is not what S.W.O.T is for and that it couldn’t be used that way.
I suspect that he did not believe his own argument to be true, but was perhaps being contrary at my request for a more formal decision making process in that instance. None the less, he had been given a rule for conducting business in a rigid context and was not prepared to budge on his position.
Agile training has this same effect on project management. Fundamentally, Agile it’s-self, is a set of guiding principals, not rigid rules, and not a methodology. When someone attempts to give you ‘Agile Training’, well, how do you provide training for a set of guidelines? The answer is, you can’t. The only way to provide training for these guidelines is to interpret them as rules – rigid rules, and to train on those rules. Agile training courses then, are an interpretation of the Agile manifesto as a set of firm rules which, unfortunately, rarely have a direct relationship with the manifesto principals.
Scrum IS a methodology.
Going back to that room in Utah, with the twelve attendees constructing this Agile manifesto, two of the participants, Jeff Sutherland, and Ken Schwaber went on to develop an Agile methodology named Scrum. I’m not certain how much of the methodology each contributed, but I know that Jeff is often attributed as having devised Scrum with Kens assistance. I’ve read books by Ken, but not by Jeff on the subject.
If training is useful at all with regards to Agile, then what better methodology is there to train than one devised by people that were in the room as the Agile Manifesto was being written?! I’ve actually seen Scrum work successfully, and I’ve seen it fail. To be fair to Scum, when I’ve seen it fail it was not the system that failed but it’s implementation.
The primary focus of Scrum as I see it, is on the communications and iterative development principals of Agile. Scrum breaks down software development into sprints in which the software is iterated through feature development. Scrum also focuses on communications by way of daily face-to-face meetings called “stand-ups.” I’m not going to go into all of the details of Scrum here, there is no shortage of information on the methodology for the curious, however there is one part of Scrum which I think is miss-understood and is, in my opinion, the work of genius…
Velocity and Points
I am by no means an authority on Scrum, but speaking from my experiences with it, and my own understanding, the true genius of Scrum is the way in which it manages the interface between the actual software developers or engineers, and their management team. You see, there is an incompatibility between the way that a software developer thinks about the software they write, and the way management thinks about it.
Software developers care about the technical details of software. What features can be written? How will those features work? How should they be written, and in particular how to write them well from an engineering stand-point. Managers on the other hand, really only care about how much and how soon. There’s a well known idiom in software development that if you want software written, you must choose only two of three metrics; Time, Features and Quality. If you want higher quality, it’ll cost more time or have fewer features. If you want more features, it’ll cost you time or quality.
Well, Scrum attempts to address this incompatibility through the use of Points for time tracking and estimation. What is a point? I’m sure that only a developers mind could come up with such a thing as the points used in Scrum. A point is a measure of the amount of time it will take to develop a feature, however, it is NOT and MUST NOT be tied to an actual measure of time!
Confused? Okay, well it goes something like this:
Ask your developer / engineer to estimate how many points it will take to develop feature X, but instead of telling you it’ll take a number of hours, or days, or weeks, ask for a number of points. You tell that developer that they can decide for themselves what the duration of a point is. For instance, they can decide for themselves that a point is an hour, or thirty minutes, it does not matter. What matters is that the size of a point to that developer is consistent, if they first estimate feature X in points of 1-hour, from that time on, all features they estimate should use points of 1 hour. They should not share with you what they consider the size of a point to be.
Still Confused? Well, if you are, you’re not alone. This is, I think the most miss-understood component of the scrum system and in particular it’s miss-understood by project managers. You see, as I’ve said above, project managers care about how much they can get and in how short a time. They like to see how many hours something will take, so that they can schedule developer hours, as though a developers performance is consistent every hour, and as though the development of a feature can be accurately estimated in hours. The reality is, developers are human, they are not consistent every hour of every day, and software development is a sophisticated process, not easily estimated.
– Aside: If you want to know just how often and poorly understood this part of scrum is, I offer you this challenge. Find a piece of project management tracking software, among the hundreds of options available, that supports “Agile/Scrum” and which does not offer estimates in time, but exclusively in points!
What a project manager needs to understand is that their list of tasks to be completed, be-they features or bug fixes, will be estimated in points. They’ll be told how many points the development team believes they can complete in the next sprint, where a sprint is typically two or three weeks, and they then select tasks to be assigned to the development team, not exceeding that points budget. At this time, they may wish to convert points to some measure of time – if I have two developers at 40 hours a week, for two weeks, I have 160 hours and a points budget of 300 points, so a point must be a shade over half an hour…. Don’t do this! Performing this calculation brings you back to thinking in terms of time, and from one sprint to the next, the size of a point is going to change, so there’s little value in doing the math.
Wait? Did I say the size of a point will change?
Yes. This is the genius of estimating in terms of points. So long as nobody equates points to time, the system of scrum (usually managed by some software), is able to adjust the size of the points as required for the performance of the team. In doing so, it is able to smooth out failings in estimation. As each new development cycle begins, historical data regarding the actual performance of the team, vs their estimates in points, is used to calculate the teams ‘velocity.’ That-is, a calculation of how efficiently the team is working, which may be used to inform on how much work they can achieve in the next sprint.
Effectively, points manage the ‘Time’ component of the Time, Features, Quality triangle as part of the methodology, offering an interface between the goals of the developers and the goals of the management. This does however require “management buy-in.” A manger / project manager must accept that they won’t be able to track the performance of the team in terms of hours, but instead, will be given a stipend of this currency called ‘points’ with which they can buy the features they want. As a consolation, they’ll at least know what features they can expect to be completed within a fixed amount of time called a sprint, but as a sprint is usually two or three weeks, this pushes the micro-managers back from estimations in hours. Now, this will give the development team a little breathing room, but as I’ll discuss in a moment, it is by no means an escape from the actual work as managers may fear. It all comes down to estimation.
Most of us software developers are professionals. We want to do a good job, delivering high quality code for as many features as possible, in as short a time as possible. Believe it or not, most of us get a real kick out of it – and I don’t think I’ve ever met a developer that didn’t like to think they are good at their job. Software developers however are human, and so of course there are those that don’t like to pull their weight, there are periods of sickness, even the odd hang over or just a burned-out morning. Essentially, we don’t all work at the same pace all the time.
Estimating how long a software feature may take to write is an art, not a science. Software features are sophisticated and interact with the existing software in a wide variety of ways, far too many for even the most experienced developer to keep in mind at any time. It’s been a point of contention between software development teams and their management since the industry began, and it’s not going away any time soon. However,
The flexible points system of Scrum which I described above, goes a long way to resolving many of the problems of estimation, with relatively light time wasted on the estimation it’s self. There are various mechanisms available for estimation in points, such as simply asking for an estimate, to playing estimation games such as ‘planning poker’; and many more, the goal however remains to come up with some number of ‘points’ for a feature. So, lets put on our management glasses for a moment and look at some of the pitfalls, from staff attrition to a developer trying to ‘game the system’ for an easy time, how does this points system resolve these problems?
Concern #1: What if a developer is assigned some work but finishes early, either by rushing or simply having been assigned a task that was over estimated? Will they sit idle for the rest of a two/three week sprint?
Answer: No. First of all, Scrum schedules work from something called a backlog. This is a pre-estimated list of tasks which may be scheduled into a sprint. There is always more work on the backlog, that’s simply the nature of software development, the software develops and evolves over time with new features or bugs to be fixed. What is important here is that Scrum uses something called a burn-down chart to track the progress of work, both for a team and for individual developers, and so, if a developer gets done early, more work can be assigned to them from the backlog. We can see how much more work they have bandwidth for from the burn-down.
Concern #2: What if a developer simply works very slowly, taking their time to complete only the amount of work that has been scheduled to them, even if that workload is low.
Answer: There are many mitigations for this. First of all, the developers are expected to talk to each other each day to explain what they have done, what they plan to do today, and if there is anything blocking them from progressing. A dedicated ‘scrum-master’ is tasked with removing any blocking issues, or even rescheduling if necessary. On top of this, each developers progress is tracked, and so reporting tools can demonstrate if one of the developers is significantly slower performing than the others, and should this happen consistently, then management will have their usual discussions with that particular developer about why this is the case. As an additional aside, many developers would love to ‘cherry pick’ what they get to work on, allowing a developer that finishes early to cherry-pick the backlog is an added incentive for them to work as efficiently as possible.
Concern #3: Vacations, Sickness, and Staff attrition can impact the amount of ‘work hours’ available in a sprint.
Answer: Yes it does, but with relatively fast development cycles of just two or three weeks, staff sickness and attrition can be managed, either by moving scheduled work from this sprint back to the backlog for the next sprint, or in the case of vacation, by scheduling fewer points of work at the start of the sprint. If you expected scum to somehow continue to deliver the same amount of work each sprint regardless of such lost time, well I wonder what kind of miracle you were looking for, but understanding that development cycles should be short, and may be altered as required to accommodate such issues should be reassuring.
Concern #4: What if ‘Acme LLC’, our most important customer, needs a feature tomorrow?!
Answer: Well, then the management must decide which items on the current sprint should be pulled back out of the sprint to make room for the urgent work. Again, you can’t expect using Scrum to somehow magically give you more bandwidth, that’s asking for time travel which is currently only possible for people that own blue police call boxes, but Scrum provides you with a scheduling mechanism which is not set fast. Simply change the schedule. Of course I understand this is a difficult problem, you don’t always want to remove features from the current schedule, and it’s difficult to decide what is more important, but that’s not a problem that Scrum is designed to solve for you. Those are decisions that you continue to make the way you always did, but under Scrum you have a more informed understanding of how much you can hope to achieve.
There are of course many more concerns in a real world business scenario, but I hope that what I’ve explained here is that Scrum can help to manage the expectations of the management team and the development team, providing an interface between them which before Scrum did not exist (at least not in such an elegant and functional form). This enables teams to estimate more accurately, and to meet their estimates more consistently.
But you said you’ve seen Scrum fail?!
Yes. I’m a big believer in Scrum when implemented correctly – but I am not a man that takes anything on faith. I’ve seen it work, in person, and so I know it can. I’ve also seen it fail, and generally the reason for it’s failure comes back to the problem of “buy-in.” Which I think can be explained best by talking about a scenario that I’ve experienced for myself…
Who do you want to have estimate the ‘points’ cost of work?
If I want to have some plumbing done at my home, I call a plumber to give me the estimate. I don’t call and ask the plumbers bank manager or accountant, they would not be qualified to estimate the job, and so I want the plumber. The same is true in software development. Though it can be a complex problem to do estimation, you want the work to be estimated by the developers that will be responsible for meeting the estimate. The problem is that business don’t always work this way.
I’ve worked for companies that send out their sales people to sell software to new customers for example, and having demonstrated the product, their customer agrees to buy it… IF they add in some particular feature. Not wanting to miss out on a sale and knowing that the iron is hot, the sales person says “Yes, we’ll have that feature in for you by our next release cycle in three weeks.” – Ut oh. Does the sales person have any idea how much effort is involved in adding that feature?! Management of course wants the sale too, so do they push back against the sale? no. Well at this point, the management must remove scheduled work from the sprint to make room for the new feature, and if you have the aforementioned “buy-in,” that’s what they do… but in my example scenario they don’t, and simply expect the new work to be completed as well as the existing scheduled work load. – Time, Quality or Features…. When the development team either fails to deliver, or delivers poor quality code, does management accept that they were responsible because they over-loaded the team?
This is just one scenario, and if you have any experience in software development you’ll know, there are many such scenarios. A critical bug is found, a customer threatens to walk, the development team must stop what they are doing to fix the issue – but – we still want to meet our goals. There’s a top-down flow when things go wrong.
Proponents of Agile and Scrum will tell you that as the professional, you should stand up to the management and refuse to do a shoddy job, persistently explain to them the work load and how much they can expect of you. The problem is that these proponents tend to have well established careers and some amount of seniority. How many juniors are going to make this stand? I mean, software developers are becoming cheaper and easier to replace, is that a risk they, or you can take? Scrum can’t fix poor management, and the software industry is a very competitive one. If the organization does not fully embrace Scrum, then it’s simply not going to function.
I’ve had my fair share of poor managers, thankfully my current manager is not one of them, but I’ve worked in situations in which there were no less than seven people given some level of authority over my work, each with different goals. I’ve had the same project described as a success and a failure by different members of the management team, based on their own desired outcomes. It’s a difficult world to work in, and when things go wrong for a software developer, team lead, or middle manager, it could be significantly damaging to their career. It’s very uncomfortable to think that your superior might be able to make your employment prospects in the future more difficult by associating you with a failing project, particularly if that project is a high profile one. And so, we can find ourselves in difficult situations in which standing up for Agile principals may be the right thing to do, but is not necessarily the easy thing to do.
As I eluded to earlier, no system of rules (which I count a methodology as being) can actually spare you the complexities of the real world, and so, yes, Scrum can also fail.
So are Agile and Scrum a good idea?
Agile yes, certainly. The principals of the Agile manifesto are good, sound principals. There is nothing wrong with having principals in your work and, if you’re able to stand up for them, I believe the Agile principals are the right ones for software development in practice.
Scrum? Well, that of course depends on management buy in, as well as understanding it’s limitations, and in making it fit with the organizations structure. If used correctly, as I’ve stated twice, it is a fine system for managing and scheduling development work, and there are of course other systems too. What you can’t expect of Scrum however, is for it to fix a poor business structure in an industry which is flooded with poor business structures. So long as you accept that, then I’d say Yes, Scrum is also a good idea.
Is Agile Dead?
I’ve read lots of internet articles suggesting that Agile and/or Scrum are through, and sadly that may be the case. You see, it’s not that they are a bad idea, as I’ve explained above, but that they are often poorly implemented, or that what is implemented is not Agile or Scrum at all, but fall under the same name.
Do you remember popping candy?
There was once a rumor that if you ate popping candy and then drank a carbonated drink, this could lead to sufficient pressure, as to cause internal damage and a visit to the E.R. Such a situation is not really possible – the body contains fail-safes to prevent it, and so as long as those fail-safes aren’t compromised you’re safe to eat and quaff popping candy and soda as much as you wish… if you should wish such a thing … but none the less, sales of popping candy dropped when the rumor first appeared, and have never recovered. The rumor also resurfaces each time there is a sales up-tick for the candy, and so it looks like the high-sales days of popping candy are dead.
Agile and Scrum are the popping candy of the software development world. Their use is in decline because of rumors that they don’t work. Worse still, unlike the popping candy case in which there is no recorded incident, the death of Agile and Scrum is lead by countless failure case studies.
I personally would not much mind if Scrum went away (sorry Jeff and Ken), because another methodology will pop up to take its place, and perhaps the new methodology will have better marketing, or more strict rigid rules to keep the higher-ups happy. Its not that I think that Scrum is a bad thing, I really don’t. I would however think it tragic if Agile principals went down with that same ship.
As always, Thanks for reading!