Category: Information_Systems

Challenges in Agile Product Development

Agile methods are used in product development for their benefits – efficiency and effectiveness. However, practitioners face a variety of challenges in successfully transitioning into agile mode of thinking and working. This article attempts to summarize the various challenges that agile teams face leaning on the author’s experience and referring to few published articles in this regard. The challenges are grouped into three areas – Alignment, Enablement and Operationalization.

1. Lack of Alignment within the organization and with external stakeholders on the agile transformation initiative

Transition to agile requires alignment across the organization (across levels & functions) and with stakeholders outside the organization as well. Top management commitment and support is to be obtained right at the start. While it is an advantage if the trigger for agile transformation comes from top management, the risk with that is that the organization may be forced into Agile transformation even if it is not suitable for the organization (or) without adequate enablement (or) with agile approaches chosen without adequate consideration of the organization’s business context. Adopting Agile methods should not be seen as the end in itself but as a means to achieve better business results. Leaders need to cascade the vision down to the teams clearly stating the benefits to the organization and employees, the strategic objectives, goals and the enablement plan. This will avoid the lack of ownership within the team members. Alignment with customers is crucial as agile methods heavily depend on the customer participation. Even in the case of internal IT projects, alignment with internal customers is required. Lack of alignment across other functions in the organization will lead to operational issues. This needs to be thought through and discussed well in advance.

2. Lack of Enablement through training & coaching for translating the agile philosophy into practice with appropriate changes in systems, processes and tools

Agile approach is basically a bunch of guiding principles and rest is to be derived and applied considering the context. Without training and coaching by professionals who have good experience in applying agile approach, teams will not be able to successfully apply agile approach. People playing critical roles like Agile Evangelist, Agile Coach and specifically in the case of SCRUM method the roles of Product Owner and SCRUM Master should be carefully chosen with required mindset, expertise and experience. Team members tend to overcommit without experienced people guiding them and this potentially leads to burnout. The project and program governance structure dashboards that are required for different levels are to be worked out carefully maintaining the spirit of the Agile philosophy. This needs the involvement of a good Agile coach. The agile transformation needs a cross functional team to guide in updating the systems and processes to enable the transformation and the teams. Specific tools are also to be introduced with the guidance of the Agile Coach. Even business models need to be worked out for the Agile context as discussed in detail in the post Agile product development: Sharing risks and rewards. In Agile methods, people important and hence the HR policies & processes need to be fine tuned.

3. Operationalization

Besides Alignment and Enablement, as they say ‘when the rubber meets the road’ there are several critical challenges during Operationalization. The challenges are specifically in:

Bringing about the changes especially in the Mindset

Many transformations fail to achieve desired results due to the gap in good change management. While Alignment and Enablement set the foundation for the changes to happen, special attention should be given to this aspect during the operational phase. The critical part in this is in bringing about the changes in mindset across all levels especially due to the heavy dependency of the Agile methods on a different mindset. Changes will definitely be possible if the concerned individuals understand what is in it for them and if there is a win-win situation. It is the responsibility of the concerned leaders and task force to do this through effective communication.

  • Ensuring good communication within team and with stakeholders

Communication starts from the beginning as early as the Alignment phase. During the operations, relevant and sufficient communication is to be ensured with all the parties involved. Of course, Agile methods rely heavily on the close interactions with customers and between team members and hence communication and collaboration is part of the approach. It needs to be ensured it happens and is effectively done.

  • Managing Scope, Cost, Schedule, Quality and other issues

Typical challenges associated with any project management activity involves Scope, Cost, Schedule and Quality.  In agile projects, since change is welcome as a philosophy, as mentioned earlier, the business model needs to be worked out to handle changes in scope well to avoid effort overruns. As evident these parameters are interlinked and they need to be managed in totality. From the cost perspective, budgeting for the total project is a challenge and some approaches are discussed in Agile product development: Sharing risks and rewards. Planning the budget, timeline and scope and adjusting is enabled well by agile methods but only if they are managed with right approach and mindset with good cooperation between the stakeholders and collaboration between team members.

Specifically, agile is not easy to be adopted in large projects as they are invariably distributed across geographies with many teams. Agile methods have evolved to address these challenges for such large and distributed projects. With support of the experienced Agile coach such projects can be accomplished successfully. Quality in agile methods stress on product quality with the emphasis on the process quality not so visible – to the extent that sometimes processes tend to be ignored in projects leading to different set of challenges. Team needs to ensure that value adding process steps – focused on product quality and other project objectives – are identified and followed.  

A partial list of articles referred is provided below for additional reading – this list is by no means exhaustive!

Agile product development: Sharing risks and rewards

Product development is mostly in Agile mode where the development is done incrementally prioritizing the features, while being flexible to changes in them and ensuring product quality in each iteration. Product companies develop the product internally and outsource a portion of the work to vendors for a variety of reasons.

Internal product development is budgeted as per business plans associated with the product and managed within the same. Features are prioritized, planned and implemented with product quality in every iteration in Agile mode. This ensures that the product launch can be done at any stage when either the timeline is reached or the budget is exhausted.

For the outsourced portion of the product development, it is a challenge to finalize the budget as the pricing has to be worked out with the external vendor. Pricing in software engineering industry is done in one or a combination of the well-known Fixed Pricing (FP), Time and Materials (TNM), Output based and Outcome based pricing models. Each of these come with their own advantages and disadvantages for the client and the vendor. A suitable pricing model with some variations from the standard models or a hybrid variant is typically worked out as per the context. Agile mode of execution poses additional challenges as requirements are constantly evolving and changes are welcome without the conventional change management clauses. The clients (Product companies) prefer FP so that there is a sense of the cost upfront and in turn the budget for the product development. However, the vendor prefers TNM pricing model as this reduces the chances of cost overruns due to scope uncertainty associated with the agile mode of execution.

In order to address this situation, clients and vendors need to work as partners sharing the risks and rewards by coming to a middle ground between the two extreme positions of TNM and FP. A few ideas the public domain are shared below with references for additional reading.

Clients can accept a TNM pricing initially to refine the specifications, benchmark the sizing of the requirements in terms of story points and such units and understand the stable velocity that the team is able to reach. Subsequently, the vendors can accept a FP per unit of requirements (typically story points) delivered. Because the sizing of the requirements have been benchmarked earlier, efforts per unit output of requirement is expected to be constant. However, there is a chance of effort deviations (overrun / savings). This deviation can be a risk for one party and an opportunity for another. A good approach to address this can be sharing of the risks and the rewards associated with effort deviations since this is not a FP engagement with change request option to address deviations. After each iteration, vendors can be permitted to charge a portion of the effort overrun with a threshold though FP has been agreed per unit of requirement delivered. To be fair to the client, this should be coupled with an offer by vendor to reduce the price (upto a certain threshold) if there was effort savings in an iteration.

Another concern that clients have with TNM is that of the capability and the associated productivity of the team. I have highlighted the challenges in defining and measuring productivity in another post. Assuming that the client and vendor agreed to define and measure productivity, vendor can commit a certain productivity target with an associated reward for doing better than that and an associated penalty if not meeting the productivity target. This would enable clients to choose TNM pricing model with an assurance that the vendor would deliver value for the cost expended and that the priority features would be implemented before the budget or timeline is forcing the project completion due to agile mode of execution.

To manage changes in requirements when FP model is chosen, the total number of story points to be delivered can be maintained as per agreed contract while swapping in priority new requirements and swapping out current requirements which are not priority.

Hybrid pricing is yet another option. Here TNM can be chosen for the portion of software where requirements are still evolving and FP can be chosen for the portion of the software where requirements are clear and technical complexity/uncertainty is minimum. Other combinations of pricing are possible as well in hybrid pricing mode.

In outsourced agile projects, clients and vendors working as partners and sharing the risks & rewards is the way to go!

Few references for additional reading

  1. Book, M., Gruhn, V., & Striemer, R. (2012, May). adVANTAGE: A fair pricing model for agile software development contracting. In International Conference on Agile Software Development (pp. 193-200). Springer, Berlin, Heidelberg.
  2. Sutherland, J.: Money for Nothing and Your Change for Free: Agile Contracts. Agile 2008 talk, http://www.slideshare.net/gerrykirk/money-for-nothing-agile-2008-presentation
  3. https://www.cognizant.com/InsightsWhitepapers/An-Agile-Twist-Fixed-Bid-Pricing.pdf

Software Productivity – Is it still relevant?

Much has been talked about how productivity is different in software industry, the challenges in not only measuring and interpreting but in defining productivity itself. Productivity is just about efficiency – the ratio of the output to input. Input is typically the human effort involved. Though it appears easy to get this through logs done by team members, there are challenges in getting the actual efforts spent due to variety of reasons. Besides, the team members do a variety of tasks which may not be directly related to the work on hand and the challenge is in dealing with efforts on such work. On measuring output the issue is in determining what is to be considered as output – Lines of Code, Function Points, Story Points, variants of these, Business Value delivered etc. Martin Fowler has discussed in detail and concludes that productivity cannot be measured.

Is productivity still relevant?

Productivity is defined and measured at different levels – individual, team and organization. It is measured to compare the efficiency in the production of software with respect to other organizations in the industry, across teams and between individuals. This efficiency determines the cost and the profitability at project and organization level. Hence, productivity is a critical input to budgeting of internal product development projects and estimation of costs for bids in fixed pricing model projects for customers (of course, the underlying big assumption is that the software requirements have been sized correctly!). At an organization level it is about profitability. However, in order to analyze organization’s profitability and improve the same, there is a need to define and measure productivity at a lifecycle level. At the same time going to individual level productivity measurement can impact team work and collaboration amongst team members leading to in the worst case pockets of high efficiency and overall degradation in productivity. Agile methodology appears to be not/less focused on productivity. The focus there is on delivering value incrementally in every release with product quality. However, the velocity with which value is delivered assuming a stable team exists is the focus – this comes with its own similar challenges. Treating software work as a flow is another possibility so that methodologies like Kanban and Theory of Constraints can be applied effectively. For some lifecycle steps, it is very straight forward to apply Kanban approach and measure productivity as well as other throughput measures like inventory and cycle time. Steve McConnell author of ‘Code Complete’ – the popular book on best practices in software production – has discussed various aspects of productivity in his talk. He has proposed a scorecard approach for productivity in which the involvement of stakeholders and their perspective on what is business value is considered. Other alternatives are to break down the work in the software industry into its lifecycle steps and deal with productivity in each step. Organizations and teams do have their own variants in this regards as per context.

Thus, organizations and teams need to identify why they want to measure productivity given their context (nature of work done, project lifecycles, execution methodologies, management policies and organization culture), define it appropriately along with other alternative metrics to productivity, measure them to the extent required, analyze the measured performance carefully to draw insights and use the same!