Improving flow in large software product development organizations: A sensemaking and complex adaptive systems perspective

Abstract

Software development organizations operate in an environment of ever-increasing volatility, uncertainty, complexity, and ambiguity. The pace of change is accelerating, business and technology complexity is growing, and organizations are struggling to keep pace. The software development industry has a $3 trillion productivity problem, according to one study. Value is not flowing as it should. Flow-based software development is part of the continued evolution of contemporary software development approaches contributing to addressing this problem. It builds on agile and lean software development approaches and incorporates lessons from Deming’s management method, the Toyota Production System, Lean Product Development, Theory of Constraints, Operations Management, and other influences. Flow-based development is foundational to modern systems approaches, including DevOps, Continuous Delivery, Site Reliability Engineering, and more. Creating and sustaining flow in organizations is a challenging problem. Despite this, there is a lack of rigorous research on the topic of impediments to flow.

Stephen Hawking famously predicted that the 21st century would be the century of complexity science. Software development organizations are complex adaptive systems (CAS). Sensemaking provides a means of exploring and understanding complex systems through making sense of people’s lived experiences in organizations. This study provides a sensemaking and CAS theory-informed understanding of product development flow in large software development organizations. To achieve its objective, this research studies ten large, globally distributed, software development organizations. Using the SenseMaker® software suite this study collects and analyzes micronarratives from people in these organizations, thereby providing insights into the patterns of organization culture that influence flow and impediments. The research methods for this qualitative study include a combination of focus group research, narrative research, and case study research. The unit of analysis is the software development organization.

The main contribution of this study is the development of an impediment management framework, related to which this study presents four sets of findings. First, this study provides an analysis of the system patterns that contribute to impediments to flow in organizations. In particular, this study analyzes system patterns identified from value streams, flow metrics, and organization culture. Second, this study explores factors related to how organizations identify impediments to flow. This study identifies eight common types of impediments that affect flow. The study further identifies nine common contributing factors to impediments and six common effects of impediments. Third, this study analyzes how people in organizations make sense of impediments. This research explores the use of the Cynefin sensemaking framework to make sense of the context of impediments in order to inform how to resolve them. Fourth, this study explores four approaches used by organizations to resolve impediments. In particular, this study explores the use of continuous improvement, A3 problem solving, experiment design, and system improvement goals in the context of resolving impediments and improving flow in organizations. This study employs the CDE (containers, differences, exchanges) framework to understand better the interventions that organizations make in order to resolve impediments and improve flow. The analysis shows how these interventions relate to the sensemaking domains of the Cynefin framework.

Ultimately, this study shows impediments continually emerge to affect flow in organizations. Some impediments are problems to be solved; others are indicators of patterns that need to shift in the organization. The quality of flow and the nature of the organization’s response to impediments reflects the organization’s culture. Improving flow requires attending to the patterns, identifying impediments, making sense of the impediments in the context of the system patterns, and taking context-appropriate action in the form of interventions to resolve impediments. This study adds to the body of knowledge of contemporary software development approaches, flow-based development, management, leadership, and sensemaking in organizations. The framework can guide practical approaches to improving flow in organizations and serves as a foundation for future research.