Can Sustaining Engineering Be Agile and Engaging for Team Members?

Hi all:

I recently talked with a manager whose team's role is changing from main line product development to Sustaining Engineering (also known as Maintenance). He first wanted to understand how to adapt the agile process for his team, but as we talked, it became apparent that the bigger issue was keeping his team happy and engaged, as they were a bit nervous about what the change meant for them and their careers.

I was interested in exploring this area with him, and thought it would be worthwhile to discuss the concerns he had for his team.  I must be transparent and state that I have never served as a manager for a dedicated sustaining engineering team, so many of my thoughts in this area come without direct experience.  Perhaps it would be best to explain why this is the case.

Why Have I Never Had a Sustaining Team? 

It's not that I've never had the opportunity to have one, I simply rejected the notion and made it part of the responsibilities of the product development team.  I firmly believe that the notion of "sustaining engineering" may be indicative of a failure mode*.  Separating the responsibility for how code runs in production can lead to another "throw it over the wall" type of mentality that we as an industry are just starting to make headway on between development and QA.  More critically, it can totally disconnect crucial feedback to the product development team causing years and years of waste that could otherwise be invested in creating customer value.  

Here are some of the arguments I've heard in support of a Sustaining Team:

• "If we could free our development team from all those damn critical situations that come up in production, we could finally get some valuable features out the door and make some money."
• "We can get cheap resources overseas to handle fixing the defects and focus our expensive resources on the things that customers really want.  It just makes good business sense."
• "Development is constantly interrupted by fixing low-priority nits and we keep missing our release dates or drop planned functionality.  This has to end!"
• "Our releases are too long for customers to wait.  A sustaining engineering team will prevent our generally late releases from being even more so."
• "Are you kidding me?  We can't have our developers actually talking to customers, those guys are a bunch of mutants that can barely communicate with each other!"  (OK, I haven't heard this explicitly, but it's certainly implied.)

Each one of these points constitutes a failure mode that I won't go into here, but may be good fodder for another blog post.  As agile is "the art of the possible," I didn't see this as an opportunity to challenge the notion of the sustaining team, but I thought I'd help with ideas to make the team as effective and engaged as possible.

*Note:  one possibility of a non-failure mode could be a distinction between a team that is primarily interrupt-driven, and one that is focused on long-term goals.  This point was brought up by my colleague and may be useful to folks in some situations.  However, it is critical that the whole is being maximized for the benefit of the customer, company, and the individual software engineers.

Can Sustaining Engineering be Agile?

Sure--why not?  If you look at the Agile Manifesto there's nothing in there that that precludes the values and principals that are universal to software engineering and the humans who are engaged in its practice.  Process improvement frameworks geared towards software engineering such as Scrum and Kanban, and the modern engineering practices that support them (e.g. XP, TDD, etc.) are certainly as valid for sustaining a product where the work tends to focus on fix packs, platform certifications, and minor feature improvement.

Some might argue the semantics of "agile" vs. "lean" and say that Sustaining Engineering isn't agile and should be run through a Kanban process.  I would tend to agree that there can be benefits of running sustaining requests (essentially tickets) through a Kanban system, if for no other reason than the fact that near constant reprioritizations happen when dealing with a mature mission-critical product that is loaded with technical debt.  There tends to be a real need to release extremely quickly to production as well and the changes are smaller than large marquee features/epics associated with new product development.

One of the nicest things about Kanban, however, is it tends to visualize the entire value chain, at least when implemented to obtain the greatest customer value in the smallest amount of time.  Seeking to "maximize the whole" may lead to some very beneficial activities, which get to the manager's concern about having an engaged team.

"Re-Imagining" Sustaining Engineering through Engagement

Back to our conversation with the newly minted Sustaining Engineering Manager...He was nervous that his team may not be as engaged in their new work assignments and they would no longer be writing new valuable features and functionality.  As someone whose success is more than partially defined by the innovative products that the teams he serves delivers, I can definitely understand his concern.  It would be mine if I was in his place as well, and I came up with a few ideas.  It's important to keep in mind the context of these ideas--this is a team of product developers who are changing roles into sustaining engineers.  I'm not sure that they would apply to every Sustaining Engineering team.  My basic premise here would be to put yourself "out of a job" by eliminating sustaining requests through the constant improvement of the codebase.  What should be left is platform certifications (browser compatibility, operating system,  JVM version, .NET version, etc.) that are always changing and can be handled by the same team now that they have the capacity to do new feature development.

Note:  This is not a comprehensive list of "best practices" for Sustaining or Support--there are a ton of those.  These are just a few ideas about how to help a team of software engineers be fully engaged while they are part of a sustaining group.

• Create a target of business-specific technical debt reduction.  Identify common root-case problems in the codebase, often a source of Technical Debt to remediate common-occurrence problems.  In general, never have the same type of defect show up more than twice (fix it "right" the first or at worst the second time).  It is possible to eliminate enough of this debt where having a sustaining team just won't make sense.
• Fix the leak first before mopping the floor.  Reducing technical debt by a sustaining team will be useless if there is a product development team continuously creating it and making it worse with each new release.   A sustaining team can be engaged in evaluating new code that is delivered and acting as an advocate to increase code quality, or at least prevent new code from creating the next wave of customer critical situations.  
• Develop and employ modern engineering practices to analyze the code base and ensure that regressions are not "developed" along with fixes.  Many legacy code bases have weak automated testing, automated unit tests, and coverage mechanisms.  Configuration / builds tend to rely on "magic" of people who are long gone, or at least not accessible.
• Act as an alpha tester of new releases from the product development team.  No one knows better what kind of problems are out in the field and how the best features may flop when put into production more than sustaining teams.  Embed yourselves in sprint reviews (demos) and quite possibly act as the owner of final user story acceptance.  How's that for engagement?
• Funnel innovation - because a sustaining team operates so close to the customer, they are many times the closest ones to know what the customer pain points actually are.  Sustaining members can act as powerful contributors to a product advisory board/committee and have the weight of real customer feedback when they speak.
• Delight customers by being able to predictably deliver fixes.  One powerful technique to do this involves measuring average cycle time for defect fixes though a process such as Kanban.  Scrum iterations can also be used effectively here as well.  Most customers don't need their fixes "right now," but have been burned so many times by missed expectations.  If a schedule can be made transparent to customers and that schedule is constantly made good, customers will start properly assigning severity to issues.
• Rotate with the active product development team.  This gets really close to my initial idea of just having one team that does both new product development and sustaining work.  Think of this as a hybrid model where there is a constant trade of learning between new product developers and developers that couldn't be closer to customers.

Additional Reading

Thanks to John Heintz (@jheintz) for some additional great reading around Sustaining Engineering, some of which may contradict or amplify the ideas I've presented:

• Everything I can find about software maintenance /DZone
• David Anderson's "From Worst to Best in 9 Months", which was used as part of the basis of his work developing Kanban.

What do you think?  Do you agree/disagree with these points?  What's worked for you as a Sustaining Engineer to have mastery, autonomy, and purpose in your work as a team member?

Image via Creative Commons, Jose Franco's (jose_franco_) Flickr photostream. (source)