Can Sustaining Engineering Be Agile and Engaging for Team Members?

Hi all:

I recently talked with a manager whose team's role is changing from main line product development to Sustaining Engineering (also known as Maintenance). He first wanted to understand how to adapt the agile process for his team, but as we talked, it became apparent that the bigger issue was keeping his team happy and engaged, as they were a bit nervous about what the change meant for them and their careers.

I was interested in exploring this area with him, and thought it would be worthwhile to discuss the concerns he had for his team.  I must be transparent and state that I have never served as a manager for a dedicated sustaining engineering team, so many of my thoughts in this area come without direct experience.  Perhaps it would be best to explain why this is the case.

Why Have I Never Had a Sustaining Team? 

It's not that I've never had the opportunity to have one, I simply rejected the notion and made it part of the responsibilities of the product development team.  I firmly believe that the notion of "sustaining engineering" may be indicative of a failure mode*.  Separating the responsibility for how code runs in production can lead to another "throw it over the wall" type of mentality that we as an industry are just starting to make headway on between development and QA.  More critically, it can totally disconnect crucial feedback to the product development team causing years and years of waste that could otherwise be invested in creating customer value.  

Here are some of the arguments I've heard in support of a Sustaining Team:

• "If we could free our development team from all those damn critical situations that come up in production, we could finally get some valuable features out the door and make some money."
• "We can get cheap resources overseas to handle fixing the defects and focus our expensive resources on the things that customers really want.  It just makes good business sense."
• "Development is constantly interrupted by fixing low-priority nits and we keep missing our release dates or drop planned functionality.  This has to end!"
• "Our releases are too long for customers to wait.  A sustaining engineering team will prevent our generally late releases from being even more so."
• "Are you kidding me?  We can't have our developers actually talking to customers, those guys are a bunch of mutants that can barely communicate with each other!"  (OK, I haven't heard this explicitly, but it's certainly implied.)

Each one of these points constitutes a failure mode that I won't go into here, but may be good fodder for another blog post.  As agile is "the art of the possible," I didn't see this as an opportunity to challenge the notion of the sustaining team, but I thought I'd help with ideas to make the team as effective and engaged as possible.

*Note:  one possibility of a non-failure mode could be a distinction between a team that is primarily interrupt-driven, and one that is focused on long-term goals.  This point was brought up by my colleague and may be useful to folks in some situations.  However, it is critical that the whole is being maximized for the benefit of the customer, company, and the individual software engineers.

Can Sustaining Engineering be Agile?

Sure--why not?  If you look at the Agile Manifesto there's nothing in there that that precludes the values and principals that are universal to software engineering and the humans who are engaged in its practice.  Process improvement frameworks geared towards software engineering such as Scrum and Kanban, and the modern engineering practices that support them (e.g. XP, TDD, etc.) are certainly as valid for sustaining a product where the work tends to focus on fix packs, platform certifications, and minor feature improvement.

Some might argue the semantics of "agile" vs. "lean" and say that Sustaining Engineering isn't agile and should be run through a Kanban process.  I would tend to agree that there can be benefits of running sustaining requests (essentially tickets) through a Kanban system, if for no other reason than the fact that near constant reprioritizations happen when dealing with a mature mission-critical product that is loaded with technical debt.  There tends to be a real need to release extremely quickly to production as well and the changes are smaller than large marquee features/epics associated with new product development.

One of the nicest things about Kanban, however, is it tends to visualize the entire value chain, at least when implemented to obtain the greatest customer value in the smallest amount of time.  Seeking to "maximize the whole" may lead to some very beneficial activities, which get to the manager's concern about having an engaged team.

"Re-Imagining" Sustaining Engineering through Engagement

Back to our conversation with the newly minted Sustaining Engineering Manager...He was nervous that his team may not be as engaged in their new work assignments and they would no longer be writing new valuable features and functionality.  As someone whose success is more than partially defined by the innovative products that the teams he serves delivers, I can definitely understand his concern.  It would be mine if I was in his place as well, and I came up with a few ideas.  It's important to keep in mind the context of these ideas--this is a team of product developers who are changing roles into sustaining engineers.  I'm not sure that they would apply to every Sustaining Engineering team.  My basic premise here would be to put yourself "out of a job" by eliminating sustaining requests through the constant improvement of the codebase.  What should be left is platform certifications (browser compatibility, operating system,  JVM version, .NET version, etc.) that are always changing and can be handled by the same team now that they have the capacity to do new feature development.

Note:  This is not a comprehensive list of "best practices" for Sustaining or Support--there are a ton of those.  These are just a few ideas about how to help a team of software engineers be fully engaged while they are part of a sustaining group.

• Create a target of business-specific technical debt reduction.  Identify common root-case problems in the codebase, often a source of Technical Debt to remediate common-occurrence problems.  In general, never have the same type of defect show up more than twice (fix it "right" the first or at worst the second time).  It is possible to eliminate enough of this debt where having a sustaining team just won't make sense.
• Fix the leak first before mopping the floor.  Reducing technical debt by a sustaining team will be useless if there is a product development team continuously creating it and making it worse with each new release.   A sustaining team can be engaged in evaluating new code that is delivered and acting as an advocate to increase code quality, or at least prevent new code from creating the next wave of customer critical situations.  
• Develop and employ modern engineering practices to analyze the code base and ensure that regressions are not "developed" along with fixes.  Many legacy code bases have weak automated testing, automated unit tests, and coverage mechanisms.  Configuration / builds tend to rely on "magic" of people who are long gone, or at least not accessible.
• Act as an alpha tester of new releases from the product development team.  No one knows better what kind of problems are out in the field and how the best features may flop when put into production more than sustaining teams.  Embed yourselves in sprint reviews (demos) and quite possibly act as the owner of final user story acceptance.  How's that for engagement?
• Funnel innovation - because a sustaining team operates so close to the customer, they are many times the closest ones to know what the customer pain points actually are.  Sustaining members can act as powerful contributors to a product advisory board/committee and have the weight of real customer feedback when they speak.
• Delight customers by being able to predictably deliver fixes.  One powerful technique to do this involves measuring average cycle time for defect fixes though a process such as Kanban.  Scrum iterations can also be used effectively here as well.  Most customers don't need their fixes "right now," but have been burned so many times by missed expectations.  If a schedule can be made transparent to customers and that schedule is constantly made good, customers will start properly assigning severity to issues.
• Rotate with the active product development team.  This gets really close to my initial idea of just having one team that does both new product development and sustaining work.  Think of this as a hybrid model where there is a constant trade of learning between new product developers and developers that couldn't be closer to customers.

Additional Reading

Thanks to John Heintz (@jheintz) for some additional great reading around Sustaining Engineering, some of which may contradict or amplify the ideas I've presented:

• Everything I can find about software maintenance /DZone
• David Anderson's "From Worst to Best in 9 Months", which was used as part of the basis of his work developing Kanban.

What do you think?  Do you agree/disagree with these points?  What's worked for you as a Sustaining Engineer to have mastery, autonomy, and purpose in your work as a team member?

Image via Creative Commons, Jose Franco's (jose_franco_) Flickr photostream. (source)

Guest Blog: Are Doc Sprints Really Agile?

Welcome once again to my friend and colleague, Janice Hamrick, with her latest perspective on helping whole development teams attain agility through effective technical writing.  Unfortunately, technical writers, who are crucial members of the product development team, are often siloed for the sake of "efficiency."  I've been serving my teams for years by insisting that stories are not complete until there is adequate documentation, and it has driven dramatic positive changes from the perspective of people, product, process, and tools.   If you like this post, please check out her other two posts (How Agile is Like a Guitar and Tech Info Fits Into Agile Software Development and 3 Ways to Make Documentation as Agile as Development).
~@MulticastMatt

Some development teams choose to separate documentation from the rest of the development work.  In fact, the Society for Technical Communication (STC) recently held a seminar entitled, “Doc Sprints: The Ultimate in Collaborative Doc Development,” which included the following description:

“Many technical writers find themselves in agile environments, where time is at a premium and it can be difficult to pry the developers and other subject matter experts away from their day-to-day deadlines.”

The problem with this practice is that it violates almost every principle of Agile and causes problems for all members of a team.

What is a “doc sprint” as opposed to a sprint?

A doc sprint is just that – a sprint containing only documentation tasks. Often, the only member of the sprinting team is the writer, with nonexistent or at best nominal support from the rest of the team. Dev and Doc are like milk and cookies - they should always go together.  If you are having separate doc sprints that lag behind development sprints, YOUR TEAM IS NOT AGILE.  Documentation is as important to software development as coding and QA - no story is done unless it’s fully coded, tested, and documented. 

Why should it be difficult to pry developers away from their deadlines?

If your team is truly Agile, EVERY team member is responsible for every user story. This means that developers are as responsible for providing content and reviews as they are for coding.  If a writer is blocked and a doc task can’t be completed, then the story can’t be closed.  This provides the means and the motivation for excellent communication and cooperation among all members of the team, and ensures the documentation gets the attention it needs from all concerned.  If a team conducts separate doc sprints, dev and QA have already moved on to something else by the time the writer starts and will have to switch context to provide information. Moreover, dev and QA very likely have not budgeted time into their own sprints to work with the writer – after all, their stories apparently can close without documentation. This leaves the writer in the uncomfortable position of “prying” developers away from their “real work” or resorting to begging (or threats). Plus, the likelihood of missing information altogether (because it’s no longer current) or of being shortchanged during review cycles increases astronomically. The doc quality suffers, team relationships suffer, and ultimately product quality suffers.

Why are writers different from any other team member?

The short answer is that they shouldn’t be! However, in many organizations, tech writers report to a different business unit than development teams, a situation that can leave writers feeling like outsiders because they are treated like outsiders. I know writers who are left off mailing lists, who aren’t included in team-building training, and who are sometimes the last to know about feature changes.  

For real Agile success, writers must be real team members. It might not be possible or even desirable to change the management hierarchy of your organization, but it is possible for teams to ensure that writers are first-class team members.  The first steps include making sure everyone on the team understands this (management support is essential), eliminating doc sprints, ensuring doc tasks are part of the team user stories, and planning for the time needed to provide content, write, and review the doc. 

Is there ever a good reason to have a doc sprint?

Probably not.  There might be a reason occasionally to have a doc-only user story (for example, if your team decides the product needs a new user guide), but even in that case the team is responsible for content and reviews.  No one team member, whether writer or not, ever works in a vacuum. All stories require commitment by the team, and as such should be incorporated into team sprints.

Your thoughts?

Does your team hold separate doc sprints or are your documentation tasks integrated into the development sprints? How does that work for your team and the quality of your product? 

About the Author

Janice Hamrick joined CA Technologies in 2010 as the Senior Technical Information Engineer for the Capacity Management suite of products (formerly Hyperformix).  Before joining Hyperformix, she spent years writing documentation for backup and recovery products for a rival company whose three initials shall remain unnamed. It is a myth that she has worked as a technical writer in the software industry since dinosaurs roamed the earth, but she does remember when “agile” was something you had to be to avoid the saber-toothed tigers.

Outside of work, Janice is the author of the award-winningJocelyn Shore mystery series (DEATH ON TOUR, DEATH MAKES THE CUT, and DEATH RIDES AGAIN, Minotaur Books), and is currently working on a new novel.  She lives in Austin, Texas.

The Most Important Book on Scrum Since the Original

I have just finished reading a book that is, without a doubt, THE most important book on Scrum that has been published since the original.  This book is Tobias Mayer's, The People's Scrum.


Why should you read this?  

You should read this because you are likely in a system that must be improved--I deeply believe that all of us are.  You will likely not agree with everything, but in order to truly realize the value of Scrum or any agile development framework, there are things that you need to know contained in this book.  If you find yourself in agreement with everything contained within, this will be an excellent reference to share with others who may not be so fortunate.  Ultimately, you want to better the lives of your team mates and yourself in ways that are founded on experience and love (how's that for a bold and courageous statement?)  It's also a nice fast-paced read as these are essentially edited blog posts, none of which are more than five pages.

What was my experience?

As I read this book, it reminded me of the first time I read the Agile Manifesto--my head was nodding in accord so many times, I nearly became dizzy.

Based on the foreword, I was convinced that I would find something to disagree with Tobias on and was looking forward to the opportunity to have a great conversation with him and potentially have my mind expanded. But I didn't...his views are so close to my own as what he says could have been written by me (although I couldn't have written them as well). I suppose that comes as no surprise as one of my early strong influences was Kert Peterson as well. We also seem to share strongly-held (as in concrete reinforced with rebar) views on the importance of humanity, decentralization, and co-location in innovative and creative work.

I suppose I didn't learn too many new things in the book, but they were told in a way to convey important meaning to the people who actually do the work and are the main focus for the Scrum framework. Of course executives and managers should also be exposed to these ideas as they are (fortunately or unfortunately) ultimately responsible for maintaining or at least spawning the systems where the workers create value. These are the people that will receive the most benefit of these essays as I share them with the teams I serve and advise others in the community and our industry.

Agile Austin - Here we Come!

I plan on ordering five copies of this immediately and sharing with the Agile Austin community. We will likely conduct a book discussion group on this book in the July time frame and will report back with comments and observations.

A few shout outs.


Thank you Tobias for this important compendium of essays that continue to be sorely needed across our industry to maximize joy and potential of our people, organizations, and technology users.

Also, a big shout out to Bernardo Salinas for catching me in the airport in Austin and recommending (insisting?) that I read this book!

@MulticastMatt

Can Rules Create Engagement?

Here's a quick story about one of our practices to encourage engagement through structured self-organization for our agile software development teams.

- @MulticastMatt

Why Structure Is Important in a Self-Organizing Agile Environment

Like many aspects of agile software development, there are different interpretations of some of the practices that are espoused, especially by novices.  In particular, we tend to encourage team members to volunteer for work versus being "signed up" for it by, in days of yore, a manager.  Some times this is taken to an extreme, and team members may not sign up for any work, or sign up for work that is currently not a high priority.  This can be problematic as it leaves holes in work that must be done.  Therefore, some level of structure is required to get the highest priority (and hopefully valuable) work to be done.  This includes the necessary collaborative meetings that, with appropriate engagement, can be phenomenally productive.

An Example:  The Rules of "Engagement"

Last year, I came up with a simple list of rules for our larger team meetings to encourage participants to focus on the goal and to eliminate wasted time (muda).  This was in response to complaints from the team that larger meetings (backlog prioritization, release planning, sprint planning, etc.) were "painful" due to the propensity to get off track and only be interesting to a few participants at a time.  Key symptoms included heads bobbing behind laptops, bent heads staring down at what we hoped were the individuals' smart phones, and the standard response of "huh?" when an opinion was solicited of a team member in either of the two former states. The rules were designed to create a system where the team could feel comfortable holding themselves, each other, and their manager accountable to achieve the stated goal and maximize the value obtained by all participants.  I refer to these as the "Rules of Engagement," pictured below.

Rules of "Engagement" Poster Used in our Larger Planning Meetings

The ultimate rule, dubbed "The Rule of Two Feet", was repurposed from the Open Space Law of Two Feet, which states: "If at any time during our time together you find yourself in any situation where you are neither learning nor contributing, use your two feet, go someplace else."  This would be the ultimate solution to disengagement, but I thought it made sense to have the departing team member discuss why they were leaving with the rest of the team.  In this system, team members are empowered with the flexibility to leave if they feel the content is off-topic, does not apply to them, or any other reason.  However, it encourages potential continuous improvement (kaizen) moments, including the discontinuation of practices that no longer serve the team.

Team Response to the Rules of "Engagement"

These rules have been put to the test on a few occasions and resulted in the deprecation of a release planning ritual that seemed to work and had been a standard for the team.  One team member was fairly frustrated with this process and stood up to leave but gave his feedback first.  We then went on to have an open dialog in front of the rest of the team.  This led to a much richer discussion about what we were trying to accomplish and the best way to do it, which continued beyond the original meeting.  While the goal had not changed, we learned a more effective way to get there by discontinuing the process, thereby eliminating wasted time.  I feel that if the initial team member had just walked out, we would have missed that critical kaizen moment.  

This is certainly a success story, but it worked within a larger framework of trust that we had developed over close to three years of working together.  As with most practices, your mileage may vary.

Our Standard Rules of "Engagement"

We at CA Technologies host a bevy of Agile Austin meetings, and the poster above tends to be an item of interest.  In fact, many visitors have taken pictures of it and brought it back to their teams.  This is great, but it's important that everyone understand the context in which it was developed.  It is critical that the team buy into this for specific reasons, understanding the why.  Like other standards, rules, process, etc., the team needs to be free to change them as they see fit when there are new kaizen moments.  

Here's a transcription in case my writing is hard to decipher:

• Cell phones down / away (unless taking pictures of story point estimation)
• Laptops closed unless
• You're taking notes for the team
• You're presenting
• You're remote
• You desire employment elsewhere :)
• Break for 10 minutes every 80 minutes
• If you're late, you missed it
• Time boxes will be maintained by the MC
• Raise your hand if you think the discussion is off-track or the Rules of "Engagement" are being violated
• Consider remote team members
• One conversation at a time
• Consider the goal of the meeting
• "Rule of 2 feet" is justified, but should be discussed

An Aside - Freedom and Organization

Many organizations have unpaid volunteers (local political campaigns, open source software projects, Agile Austin, etc.) but that does not imply that everyone is free to do whatever they would like.  There is a mission or a purpose that drives the most effective organization.  As the definition of "organization" is: "The structure or arrangement of related or connected items,"  organization implies structure.  Generally, people come in as novices, learn the system and then are entrusted with some span of control beyond themselves once they have attained some level of mastery.  It's naive to believe that volunteering is equivalent to the ability to do whatever one desires in any type of complex organization.  I see the same thing in organizations trying to create software.  

Allowing for self-organization and self-management is essential for software development teams where agility is critical.  Indeed, it creates an environment where team members are fully engaged in creating usable and valuable software.  However, self-direction may lead away from the goal of the organization.  Striking that balance is something that no "best-practice" can ever dictate--indeed it requires experience and a deep understanding of many dynamics.  For those in entrusted with managerial or technical leadership positions, this can be an area of extreme benefit or dismay and thus should be a focused area of learning.

Don Reinertsen's Video on InfoQ - "Second Generation Lean Product Development: From Cargo Cult to Science"

Mike DuVall turned me on to this video the other day: http://www.infoq.com/presentations/2nd-gen-lean where Don Reinertsen discusses framing many of our engineering problems in light of economic tradeoffs. One of the interesting things to see is how the ideas of lean manufacturing DON’T apply to product development (perhaps this is counter-counter-intuitive? J ). This presentation was made during the Lean and Kanban 2009 conference and I believe a number of folks have been discussing some aspects of this for a while, but I’ve also talked to a number of folks who also may not be aware of it.

I often find myself in conversations with teams who have adopted some of the practices of Scrum but are disappointed with the results that they are seeing. When I ask a few probing questions, I find out that they’re somewhat missing the point and have adopted neither the agile principals nor values on top of which the Scrum framework was based. I’m sure the same can be said for adoption of others’ successes where limited benefits are observed. The term “cargo cult” has been used to refer to this type of behavior and this video has a definitely serviceable explanation of the phenomenon.

It’s definitely a bit more of an advanced topic, but shouldn’t be difficult for most folks with a technical background to understand (although you may need to review a few of the bits a few times to get the full meaning). Not surprisingly, Don has a book, which is called _The Principals of Product Development Flow: Second Generation Lean Product Development_ (http://www.amazon.com/gp/product/1935401009/ref=cm_li_v_cr_self?tag=linkedin-20). I IMMEDIATELY ordered this book after watching the video and am now half way through it. By the way, Don Reinertsen wrote the intro to _Kanban_ by David J. Anderson and he is credited for a great deal of influence on this work.

@MulticastMatt