Product Manager versus Quality Engineering Lead

Our concept of agile has a Product Manager (PM) representing the customer. The PM is also active in defining the features needed per release train and in generally working close with the team to define stories and the iteration-by-iteration flow. This thinking embraces Steven Wheelwright’s (http://www.hbs.edu/faculty/swheelwright/index.html) model of the heavyweight product leader and seems necessary ensure we have a train payload that makes market sense. It does present a challenge when combined with the separate notion of a closely-coupled but independent software quality engineering (SQE) team. Each development team (4 to 10 people) generally has two or more engineers charged with the responsibility of ensuring all the features of the train, new and previously developed, have, and continue to have, high quality. This is usually done several different forms of testing with the SQE engineers laying down strategy and developing the tests.

A lead for each train represents the quality side for all product decisions and in specifying work for other SQE engineers attached to the team. These leads hold a fairly powerful position since they can hold up a release if not comfortable with the state of the quality as reflected by test results, defect load, and general readiness of several factors/artifacts. Operating within a strict development timeboxes and having a multi-year history of generally not missing a release (and doing up to 11 a year), the decision to hold up a train release, of course, would be a big decision. Using a sports metaphor, the train goes into a Quality Overtime and means the release takes a day-by-day slip until the lead is ready to sign off. During this time, all members of the team, including all development engineers, report to the lead to ensure the Overtime has the necessary resources. This is consistent with our thinking that all dev engineers report to quality during hardening that may occur before release.

While the heavyweight empowerment for the PM is necessary it can also be a point of conflict if the PM drifts a bit and starts to specify what the SQE engineers’ storycards and tasks. This generally arises from a PM being pushy in wanting to ensure that defects are found as soon as features are completed. We generally meet this situation by reminding the PM that the SQE team has a history of keeping up. So it becomes a bit of reset and enforcement in roles.

There is at least one other, and more complex, reason for this arising. The Lead SQE is generally focused on functional testing necessary to show/ensure feature quality. Other testing, though, may be driven by market needs apparent only to the PM. These may include security testing, stress testing, unique configuration testing apart from standard, and forms of conformance testing. While these are all doable, a PM popping these needs on a SQE team already driven to meet functional quality checks can result in a bit of a breakdown.

Some have observed that this situation is tough on the SQE lead and the gravity is for the SQE to sign-up for more than can be accomplished. A better situation, sometimes requiring management attention from the SQE side to ensure “table balancing” is for the PM to state the market needs and then to come to an agreement of testing and when it might occur: Day 1 before release, Day 2 immediately following release, or Day 3 on customer request. Done early in a train, this type of agreement, sometimes called negotiated quality, is the best situation since the SQE Lead can work to include the additional testing beyond the normal functional test work.

We recently had to work our way through a negotiated quality situation. My default assumption has always been that the SQE Lead is empowered to handle the negotiation from the quality side. Management in Moscow pointed out that that is not always the case as many engineers are good at engineering but not equipped to be good in arriving at a win-win solution. This gets even more so, when the PM is at company headquarters and the SQE Lead is offshore. It probably means that the SQE Site Lead, responsible for ensuring quality activities at the offshore site work well, must now participate in the heavy lifting of getting agreement to the needs. Done right, the SQE Lead for the respective project will participate and become better at negotiation.

An improvement beyond this is to provide training for everyone on negotiation with the emphasis of getting to win-win. I have done this at a previous company and it was helpful in empowering the Leads, giving them approaches and in reducing their being “whipped around” to the PM’s view, but it was not an overnight win. It was, thought, a starting point that did lead to success with lots of re-enforcing the roles by management. It’s important to keep in mind that software development, and especially with agile, is sometimes more about people than technology.

Standish Group research

I found an interesting Standish Group research which was done in 1995. It is about a set of causes why software projects succeed. They ranked all reasons based on score collected from many different vendors.

• Users involvement (score: 19)


• Top management support ( score: 16)


• Requirements accuracy (score: 13)
  

• Right planing (score: 11)


• Realistic expectations (score: 10)


• Iterative development (score: 10)


• Competent stuff (score: 8)


• Stuff involvement (score: 6)


• Clear goal definition (score:3)


• Purposeful and hardworking personal (score:3)

In spite of the age of this report I would agree with this conclusion.
This is good that Agile practices address several causes from this list. For instance: Requirements accuracy, Right planing and Iterative development. But the heaviest reasons from the list: Users involvement and Top management support should be addressed on another level of management.
I thing that nowadays people are more concentrated on bottom causes than firsts ones. One of the PingIdentity advantages is an ultimate close relationships between top management and engineering team including offshore. Probably it is a key factor of a company productivity. Speaking about User involvement seems to be an intensive growing area.

Elaboration and Stuff of the Agile Pipeline

Our experience with agile is that it yields a considerable speed advantage to the market over the traditional waterfall and RUP framework approaches. While just an intuition, the 30% to 40% boosts reported in the press could be very close to the gain we are seeing in our Denver office. Stretching agile to offshore, we also see a boost over a traditional approach with the estimates being 20% or so. I should note that our first offshore relationship, roughly a year, with a large provider in India, we did not see any gain and instead discovered high attrition and agile were a deadly combination leaving us with low velocity/feature output. The situation with Moscow is quite different with agile being very compatible to the team.

Starting with a definition: The agile team “core” is those engineers that take a storycard, task it out, do design, implementation, unit testing, integration, and demonstration over two-week (our choice) iteration. In our model, the core depends on the product manager (PM) to identity and elaborate product features and on the software quality engineering (SQE) members to do feature acceptance/testing. This all sounds good but once a world class core is ramped up the pacing problems across the dev pipeline, feature to accepted code, become focus points. The PM can fail to “feed” the core with enough feature detail to keep them going and the SQE guys can fail to keep up with the feature outlay. Today, I’m focusing on the front-end, the elaboration process.

A simple model for the front-end looks like this:

So this looks fairly straightforward but raises the question of who does the elaboration. Start with the view that the PM is charged with elaboration. This is really only one possibly with the elaboration taking several different paths and the picture looking like:

Let me emphasize that “elaboration” means “written description of requirements” This shows 4 possibilities with the PM taking the elaboration load, a sharing in some way, the dev engineer doing it all, or a null path. The top bubble often becomes the second bubble with the engineer asking questions but, of course, the answer may not be written down. The null approach occurs in at least two ways: 1. the feature is so simple that even my youngest cat could implement it if he knew Java versus Modula or 2. The engineer is well grounded in the product, having maybe built much of it from the ground up, understands the usage market well, and can go directly to design from a brief feature description. Before the SQE manager beats me up, it is important to note that if the null path is taken, it might be necessary for the implementation to be explained to the quality team or they might be left wondering what the feature means when implemented. The same words apply to the tech pubs team that must write about each feature for the user documentation.

Our experience is that a development train payload, composed of many features, may spread features out across all four of the paths. The null path is the fastest path to functionality but honest scoring must account for the time taken for verbal walkthrough/explanation to the SQE team on finishing up the implementation. It seems that as a team grows with product creation and grows in its understanding of the market needs, meaning many releases without much team churn, the null path takes a larger percentage than in the early days. Velocity, feature production for core members, does go up with this being one of several contributing factors. Definitely the manifesto words “Written software over comprehensive documentation” come to mind.

Now, with this simple model, we can look at what happens when we go global with the team split across between Denver and Moscow. The PM is in Denver close to the market. The dev team is offshore. The challenge: Don’t bottleneck with elaboration.

The cautious approach is for the PM to elaborate everything. They are close to the domestic market and definitely should understand it. A high aggregate velocity, total feature product across all core members, means more than one PM might be needed not to slow things down or the PM might have to be several weeks in front of the current iteration cranking out requirements grist for the core to chew on. That sounds very much like waterfall-like agile. (The oxymoron light just went on.) One alternative is to have another PM in Moscow with who shares the elaboration and question answering from the team. We’ve done that and it works. We have also done a variation where the solution is a daily verbal exchange to elaborate features. It means lots of hours on Skype and it also is somewhat like the null approach in that somewhere the SQE team must catch up/understand the functionality behind a feature. There is also the approach where the developer takes the SQE team through the functionality, they take notes, develop an attack plan and hit the features with tests. This might also mean the SQE team has someone whose job is to interview the developer and render the functionality to written requirements. Some places see this as the role of a requirements engineer.

Just thinking about what works and could work better, the front end comes to mind and I have not even touched on the all the craziness that happens in prioritizing the requirements and keeping that priority dynamic as the train flies along through the many iterations.

Global Agile Team: USA and Russia

SUMMARY
The development of software products using the agile methodology across global teams can yield significant advantages. It is not without challenges that require diligence in day-to-day management and agile adaptations to achieve high velocity. Certain patterns of structure, iteration cadence, and trust co-evolution are necessary in bridging differences between members separated by distance, time, culture, management style and backgrounds. This report details the experience gained over a two-year period using agile with offshoring and a team with members in Denver and Moscow. It compares discovered patterns for success against discoveries with both a successful co-located agile team and with a less successful effort in agile offshoring to India.

PROJECT BACKGROUND
Ping Identity develops enterprise-level, server-based software products that operate as middleware for user authentication and identity federation. The agile efforts considered for this report include the implementation of a specialized test system, the creation of a security token service (STS) used in web service security, and the development of a software engine for user provisioning among business partners. These products are characterized by a great deal of XML handling, encryption, browser-based user interfaces, and operation in application-server settings. Most work was done in Java and used a combination of open source packages. The development environment included both commercial and open-source tools for engineer IDEs, testing, source management and defect tracking. Rally was used to manage features and tasks across the team. Development cadence revolved around daily scrums, iterations of two weeks and development trains, yielding significant releases with documentation of two to four months.

STORY
This experience is noteworthy as a dynamic combination of ramping up a global team while evolving agile methodology to meet situational needs. Moving away from a faulty, year-long relationship with a large Bangalore offshore provider, Ping Identity® evaluated eight companies with multi-day, on-site due diligence. Two of those companies were actively using agile with mixed performance possibly linked to factors that we later identified as critical to offshore agile success. Luxoft, while not one of those practicing agile at the time, scored high on 13 out of 17 criteria and was selected. It is important to note that Ping Identity had, at that point, practiced agile since the major staffing of the Denver-based engineering team, a period of roughly a year and a quarter. Drawing on consulting assistance, the Denver team had evolved to an agile team composed of a product manager, tech lead, development engineers, co-located but separately reporting quality engineers, and technical publications. The development process was best described as rigorous scrum laid on top of a framework of a tight development timebox and a hardening tail for each development train. The challenge was the mapping of this baseline process to an offshore team.
Development during the first year and half of the relationship with Luxoft was based on dividing the global team so that the product management, architecture guidance work was done by a dedicated Denver team and the implementation was largely done by the Moscow team. Quality engineering was accomplished by a split team with Moscow doing the bulk of the test design and testing and the Denver team doing follow-on audits and shared testing. The Moscow team quickly accepted the agile processes as something fresh, and at 6 weeks into the relationship an acceptable velocity was attained and continued to grow from that point. Measurements across purely Denver-based and the Denver-Moscow combo team showed task attainment per iteration—one aspect of velocity—to be similar, with the combo team typically having iteration-to-iteration rates 80 to 86%. During this time the Moscow team totally refactored a test system, contributed to two other product lines, and developed a whole new product from blank paper. After the year-and-half point, the team structure shifted to have architectural roadmapping done in Moscow and a different daily communications format.
Two observations complete the story: communications and indexed responsibility. Successful agile hinges on solid verbal communications. This can present a challenge when viewed across a global operation. However, we were fortunate in that many Moscow engineers customarily work from 11:00 AM to 9:00 PM. This shift matched against an early morning start in Denver supported daily global stand-up scrums, requirements questions, joint problem resolution, and, on iteration kickoff days, five to six hours of interaction for retrospectives, story and task reviews, and estimation. Trips between the two teams, roughly 16-person trips per year, further strengthen the communications bridge. The overlay of offshore with growth and agile was successfully done with staff growth in Moscow and responsibility for business-critical software indexed on successful performance. That is, agile operation was successfully demonstrated at a core staffing size of four before the addition of more team members. Working along another dimension, business criticality (the assignment of importance to tasks) was carefully indexed to performance in each train. First the split team demonstrated high payload delivery on test-system development before contributing to the flagship product. Success was shown there before they were permitted to take on a full-blown product from blank paper. Then success was shown there, before the shift of architectural responsibility from Denver to Moscow. This indexing was matched by the Moscow’s team’s desire for more and more technically challenging efforts.

LESSONS LEARNED
Bottom-line, agile requires attention to many factors to yield the 30% to 40% increase in team yield reported in the press. A global team complicates this but is achievable. The Ping Identity and Luxoft teams discovered many key factors over the last two years. A few of these include the following:

CLOCK SPEED COUNTS
We tend to describe agile as a continuous marathon. Being so strongly team linked, it demands that each team member must have a fast “clock speeds”. Both Denver and Moscow lost smart individuals, some with PhDs, whose thinking, production, unit testing, and demonstration capacity failed to fit into a two-week iteration. Attention to quickness in moving from an idea to a tangible is now a key factor considered in hiring.

ROLE TRAPS EXIST
Individuals with broad development and problem-solving skills work the best in keeping agile attainment high. This means that individuals may have to do a number of things to make the team successful. Individuals whose scope of work or performance of assignments is in some way tied to their title, such as “architect,” typically are less productive in the agile setting and eventually self exit. Similar to this but not as destructive, the will of development engineers to participate in product hardening varies with individuals. Since our form of agile has a hardening tail, all must operate in this phase for what is roughly 20% of the train length. Drops in productivity from role rigor appear to be tied to individuals and are not country specific.

STRUCTURES GET IN THE WAY
Agile flattens out the organization. The test of this is the scrum with each person noting his or her status and work situation. Organizations and, perhaps cultures, with strong hierarchical thinking, find it difficult to let go of the leader representative. While a hierarchical organization, the Moscow team learned this notion and is careful to honor this flattening. This continued to be an issue in spite of a year of experience with the Indian agile team. It is thought that this “stickiness” of role is largely linked to strong hierarchical management patterns inherit and re-enforced daily in the offshore organization.

TIME OVERLAP NECESSARY FOR EFFICIENCY
The core of all agile teams is the code-production “engine.” Feature attainment suffers when this core is starved for elaborated requirements or is incapable of self-elaborating to have requirements. Global agile teams are faced with the task of efficient communication of elaborated requirements. Comparing Moscow to Bangalore, with overlapping workdays as opposed to little or no overlap, it is clear that the overlap situation is better, supporting light elaboration on the part of engineers with direct accessing to the product manager to answer questions on a daily basis. Otherwise, the product manager has to richly elaborate requirements in hopes of not starving the core team. In the spirit of doing only as much “in-flight” documentation as needed, this rich elaboration is less efficient.

CONTEXT-RELATED COMMUNICATIONS
Agile requires engineers to focus and operate in what is often called a “heads-down” mode. This simply means that each individual is focused on achievement necessary for the current iteration. Awareness about this level, a higher context, is necessary for good individual motivation and team momentum. Often missed is the fact that engineers co-located with the executive team, marketing, and sales receive contextual information through a variety of channels. These channels, hall chit-chat, discussions, and other things, do not extend beyond the facility. An important and originally overlooked need of global agile teams operating on a minimum of documentation is for a steady feed of context information about the destination of the product and the company commitment. This can be through visits by executives, ensuring presentations given to headquarters engineers are also presented offshore, and, in some way, through “gateway” individuals that funnel information to their counterpart and buddies offshore.

CONCLUSION
Agile with a global team can yield significant benefits. It requires the leadership to be aware of many added dimensions necessary for high efficiency. Existing patterns of work and attitude across the team require time and attention to co-evolution so that high velocity is achieved. It can be done as we have shown with the Denver/Moscow team.

First bullet

We (DL and Mr. Bill Wood) would like to start blogging about Agile practices we experienced.
I guess, the plan is following:
1) Start pushing ideas and thoughts about topics we are going to talk
2) Invite other readers for criticism and sharing experience
3) Aggregate all valuable information to one Power Point presentation