decision making

The New Generalist: OODA, Machine Learning and Decision Making Languages

Machine learning operationalization has come to be seen as the industrial holy grail of practical machine learning. And that isn’t a bad thing – for data to eventually be useful to organizations, we need a way to bring models to business users and decision makers. I’ve come to understand the value of machine learning for decision making from the specific context of tactical decision making as fitting into the common OODA loop for taking decisions. At first sight, these seem like specific ideas meant for a business and enterprise context, but further exploration below will reveal why this could be an important pattern to observe in ML-enabled decision making.

Lazy Leaders of a Specific Kind

I recently listened to a Simon Sinek talk in which he made make a rather bold statement (among the many others he’s made) – “Rules are for lazy leaders”. I see this phrase from the specific lens of using machine learning solutions as part of a decision making task – building which is almost existential to the machine learning practitioner’s workflow today. What I mean here, is that leaders who think in terms of systems rarely rely on one rule, but usually think in terms of how a system evolves and changes, and what is required for them to execute their OODA (“Observe-Orient-Decide-Act”) loop. I want to suggest that machine learning can create a generation of lazy leaders, and I mean “lazy” here in a very specific way.

Observe-Orient-Decide-Act

A less-well-known sister of the popular Deming PDCA (Plan-Do-Check-Act) cycle, OODA was first developed for the battlefield, where there is (a) a fog of war – characterized by high information asymmetry, (b) a constantly changing and evolving decision making environment, (c) strong penalties for bad decisions, (d) risk minimization as a default expectation.

These four criteria and other criteria make battlefield decision making extremely tactical. Strategy is for generals and politicians who have the luxury of having a window into the battlefield (whence they collect data) and have the experience to make sense of all this in the context of the broader war beyond the one front or battle. Organizations are much the same for decision makers.

OODA
OODA – Observe-Orient-Decide-Act – Courtesy Opex Society

As of 2021, decision making that uses machine learning systems has largely impacted the tactical roles, where there are decisions made on a routine basis with a days-long, hours-long or minutes-long decision window. Andrew Ng famously summarized this in his “AI is the new electricity” talk – where he discusses the (unreasonable) effectiveness of machine learning for tasks that require complex decision making in a short time frame.

Sometimes, strategic and tactical decisions are made on the same times scales, but this is rarely the case outside of smaller organizations. Any organization at scale naturally sees this schism in decision making scales develop. OODA is ideal for decision makers that are domain experts, in this limited tactical decision making setting. Such decision makers are rarely domain experts, but are adept functional experts who understand the approach to and the implications of success in the environment they’re in.

OODA Loop is an actual street named for the Observe-Orient-Decide-Act context

So where does machine learning operationalization fit into all this? Today’s decision making environment is data-centric and information-centric in all enterprises. What this means for the typical decision maker or manager, is that they are often faced with information overload and high cognitive load. When you’re making decisions that have dollar-value impact on the enterprise, cognitive load can be a bane beyond a point. Being cognitively loaded is absolutely necessary for any activity up until a certain point, when the tedium of decision making under uncertainty can affect decision makers emotionally, causing the fight-or-flight response, or can trigger other set behaviours or conditioned responses.

This brings us to the information-rich nature of today’s decision making environment. When we’re dealing as decision makers with lower level metrics of a system’s performance and are expected to build an intervention or take a decision based on these lower level metrics, we are rarely able to reason well in terms of more than a few such variables. The best decision makers still end up thinking in terms of simple patterns most of the time, and rarely broach complex patterns of metrics in their decision making processes.

Machine learning enables us to model systems by transforming the underlying low-level metrics of their performance into metrics of higher level abstractions. In essence, this is simplification of complex behaviour that requires high cognitive load to process. Crucially, we are changing the language with which we take decisions, thanks to the development of machine learning models. For instance, when we are looking at a stock price ticker and are able to reason about it in terms of confidence interval estimates, we’re doing something a little more sophisticated than thinking in simplistic terms such as “Will the stock go up tomorrow?”, and are probably dealing with a bounded forecast spanning several time periods. When we’re analyzing video feeds manually to look for specific individuals in a perimeter security setting, we ask the question “Have I seen this person elsewhere” – but when doing this with machine learning, we ask the question of whether similar or the same people are being identified at different time stamps in that video feed. We’re consequently able to reason about the decision we ought to make in terms of higher level metrics of the underlying system, and in terms of more sophisticated patterns. This has significant implications in terms of reducing cognitive load, allowing us to do more complex work with less time, and crucially, with less specialized skill or intelligence for executing that task, even if we possess a complex understanding of the decisions we take.

The Limits of our Decision Making Language

I want to argue here in a rather Wittgensteinian vein, that humans are great at picking up the fundamentals of complex systems rather intuitively if the language they use to represent ideas about these systems can be conveyed in a simplistic manner. Take a ubiquitous but well-known complex system – the water cycle. Most kids can explain it reasonably accurately (even if they don’t possess intricate deep knowledge of the underlying processes), because they intuitively understand phase changes in the state of water and the roles of components in the cycle such as trees, clouds and water bodies. They don’t need to understand, say, the anomalous expansion of water, or the concept of latent heat, in order to understand the overall process of evaporation, the formation of clouds and the production of rain and water bodies as a consequence of these.

OODA and other decision making cycles can be amplified by operationalized machine learning systems. ML systems are capable of modeling complex system behaviour in terms of higher level abstractions of systems. This has significant implications for reducing cognitive load in decision making systems. Machine learning operationalization done through MLOps can therefore have significant implications for decision making effectiveness on a tactical basis for data-driven organizations.

Implications and Concluding Remarks

Machine learning and the development of sophisticated reasoning about systems could lead to the resurgence of the generalist and the AI-enabled decision-making savant with broad capabilities and deep impact. For centuries, greater specialization has been the way capitalism and free markets have evolved and been able to add value. This has had both advantages and disadvantages – while it allowed developing societies and countries to rise out of poverty by the development of specialized skill, it also meant decelerating returns for innovative products, and more seriously, led to exploitative business practices that sustained free markets. Machine learning systems if applied well can have far reaching positive implications and free up large numbers of specialists from tedium, enable them to tackle broader and more sophisticated problems, while simultaneously improving their productivity. As a consequence, ML and smart decision enablers such as this may be able to bring even more people from developing nations in Asia, Africa and elsewhere into the industrial and information age.

Some Ideas on Combining Design Thinking and Data Science

Recently, I had the opportunity to finish Stanford SCPD’s XINE 217 “Empathize and Prototype” course, as part of the Stanford Innovation and Entrepreneurship Certificate, which emphasizes the use of design thinking ideas to develop product and solution ideas. It is during this course, that I wrote down a few ideas around the use of data in improving design decisions. Design thinking is a modern approach to system and product design which puts the customers and their interactions at the center of the design process. The design process has been characterized over decades by many scholars and practitioners in diverse ways, but a few aspects are perhaps unchanged. Three of these are as follows:

  1. The essential nature of design processes is to be iterative, and to constantly evolve over time
  2. The design process always oversimplifies a problem – and introduces side effects into the customer-product or customer-process interactions
  3. The design process is only as good as the diversity of ideas we use for “flaring” and “focusing” (which roughly translate to “exploring ideas” and “choosing few out of many ideas” respectively).

Overall, the essential idea conveyed in the design thinking process as explained in XINE 217, is “Empathize and Prototype” – and that phrase conveys a sense of deep customer understanding and focus. Coming to the process of integrating data into the design process – by no means is this idea new, since engineers starting from Genichi Taguchi, and perhaps even engineers a generation before Taguchi, have been developing systems models of processes or products in their designs. These systems models are modeled as factor-response models at some level, because they are converted to prototypes via parameter models and tolerance design processes.

Statistically speaking, these are analogues of the overall designed experiment practice, where a range of parameter variables may be considered as factors to a response, and are together modeled as orthogonal arrays. There’s more detail here.

Although described above in a simplified way, data-driven design approaches, grouped under the broad gamut of “statistical engineering” are used in one or other form to validate designs of mechanical and electrical systems in well-known manufacturing organizations. However, when you look at the design thinking processes in specific ways, the benefits of data science techniques at certain stages become apparent.

The design thinking process could perhaps be summarised as follows:

  1. Observe, empathise and understand the customer’s behaviour or interaction
  2. Develop theories about their behaviour, including those that account for motivations – spoken and unspoken aspects of their behaviour, explicit and implicit needs, and the like
  3. Based on these theories, develop a slew of potential solutions that could address the problem they face (“flare”)
  4. Qualify some of these solutions based on various kinds of criteria (feasibility, scope, technology, cost, to name some) (“focus”)
  5. Arrive at a prototype, which can then be developed into a product idea

While this summary of the design thinking approach may appear very generic and rudimentary, it may be applicable to a wide range of situations, and is therefore worth considering. More involved versions of this same process could take on different levels of detail, whether domain-specific detail, or process-wise rich. They could also add more fine-grained steps, to enable the designer to “flare” and “focus” better. As I’ve discussed in a post on using principles of agility in doing data science, it is also possible to iterate the “focus” and “flare” steps, to get better and better results.

Looking more closely at this five-step process, we can identify some ways in which data science tools or methods may be used in it:

  1. Observing consumer behaviour and interactions, and understanding them, has become a science unto itself, and with the advent of video instrumentation, accelerometers and behavioural analysis, a number of activities in this first step of the design thinking process can be improved, merely by better instrumentation and measurement. I’ve stressed the importance of measurement on this blog before – for one, fewer samples of useful data can be more valuable for building certain kinds of models. The capabilities of new sensors also make it possible to expand the kinds of data collected.
  2. Developing theories of behaviour (hypotheses) may be validated using various Bayesian (or even Frequentist) methods of data science. As more and more data gets collected, our understanding of the consumer’s behaviour can be updated, and Bayesian behavioural models could help us validate such hypotheses as a result.
  3. In steps 3 and 4 of the design thinking process I’ve outlined above, the “focusing and flaring” routine, is at one level, the core experimental design practice described by statistical pioneers including Taguchi. Using some of the tools of data science, such as significance testing, effect size determination and factor-response modeling, we could come up with interesting designs and validate them based on relevant factors.
  4. Finally, the process of prototyping and development would involve a verification and validation step, which tends to be data-intensive. From reliability and durability models (based on Frequentist statistics and PDF/CDF functions), to key life testing and analysis of data in that context, there are numerous tools in the data science toolbox, that could potentially be used to improve the prototyping process.

I realize that a short blog post such as this one is probably too short to explore this broad an intersection between the two domains of design thinking and data science – there’s the added matter of exploring work already done in the space, in research and industry. The intersection of these two spaces lends itself to much discussion, and I will cover related ideas in future posts.

“Small Data”and Being Data-Driven

Being data-driven in organizations is a bigger challenge than it is made out to be. For managers to suspend judgement and make decisions that are informed by facts and data is hard, even in this age of Big Data. I was spurred by a set of tweets I posted, to think through this subject.

Decision Making Culture

A lot of organizations have jumped into the Big Data era having bypassed widespread use of data-driven decision making in their management ranks altogether. And this is, for many organizations, an inconvenient truth. In many organizations, even well known ones, experienced managers often made decision on gut feeling or based on reasons other than data that they collected. Analytics and business intelligence hoped to change that, and in some ways, it has. Many organizations and managers have changed their work styles. Examples abound of companies adopting techniques like Six Sigma in the 1980s and 1990s, a trend that continues to this day in the manufacturing industry.

Three Contrasts

With the explosion in technologies and methods that have enabled Big Data to be collected and stored as “data lakes” and for data to be collected in real time as streaming data using technologies like Spark and NiFi, we’re at the advent of a new era of decision making characterised by the  3 Vs of Big Data, and data science at scale.

To see three contrasts between old and new management decision making styles:

  1. Spending and buying decisions (for resources, infrastructure, technology and projects) are made after competitive evaluation based on data now more than ever. In the past, the lack of communication and analysis engines, and limited globalization enabled managers to spend less time evaluating even critical decisions, because the options were limited. Spending and buying decisions make up a lot of the executive decision making and a lot of it is informed by small data. The new trends of connected economies to networks, data mining and data analysis is bound to impact this positively. A flood of information enabled by the digital age exposed them to possibilities but without the tools to do better at such competitive analysis. The advent of advanced analytics will upend this paradigm, and will result in a better visibility for decision alternatives.
  2. Operational excellence decisions are based more on real-time data now more than ever. Operational excellence and process efficiency is a key focus area for many manufacturing organizations, and increasingly concerns service oriented organizations as well. While “small data” were being collected at regular intervals, to get a sense of the business operations, these were not fully effective in capturing the wide range of process modes and didn’t represent the full possibilities one could leverage with such data. The number of practitioners of advanced methods, who used such methods in a verifiable way, were also limited and rarely formed the management strata or informed them. The proliferation of the new classes of data scientists and data engineers will affect the way decisions will be taken in future, in addition to the advent of real-time analytics.
  3. Small data as a stepping stone to Big Data. Small Data, which is data collected as samples that may be slices of sensor information or representative samples of population data (such as Big Data), may increasingly be used to formulate the “cultural business case” for doing Big Data in companies. Many companies that do not have the culture of data driven decision making in their managerial ranks, are experimenting on a grand scale, with Big Data. Such organizations have taken to Big Data technologies such as Hadoop and Spark, and are collecting more data than they usefully analyze, often times. There is definitely scope to evaluate the business value with such implementations. There is also an opportunity to improve the cost effectiveness of the data science initiatives in companies, by evaluating the real need for a Big Data implementation, by using “small data” – data that does not have the same volume, velocity, variety and veracity criteria that what’s now accepted to be Big Data does have.

Data Driven Decision Making Behaviours

Decision making is strongly influenced by behaviours. Daniel Kahnemann’s book Thinking Fast and Slow provides a psychological framework for thinking about fast and slow decision making, the former being gut-driven, and the latter being driven by careful, plodding analysis. Humans have the tendency to decisiveness, especially in organizations, and executives are often rewarded for fast decision making that is also effective. Naturally, this means that decision making as a habit flourishes in organizations.

Such fast decision making, however, comes at a price. A lot of decisions that aren’t well thought-through, could influence a large organization’s functioning, because the decision could be fundamental to the organization and may be relevant to all employees. Some organizations do reward behaviours in their managerial cadres that facilitate looking at the data that supports decisions. However, the vast majority of managers have a tax on the time they spend on decisions and would be rewarded for acting quickly and influencing a wide ranging array of decisions instead.

Enabling fast decision making has obvious benefits in a market economy. The more time managers spend in decision making, or delay a decision, the less competitive companies tend to look. Data driven decision making can be enabled by providing access to data, in a quick and painless way. And this means building intelligence into our interfaces, and into the machines that help us make and record decisions. It also means being able to delegate the mundane tasks well and easily.

Concluding Remarks

A lot of organizations that have Big Data initiatives may not have the appropriate management or decision making culture that can fully utilize the investment in Big Data, which can sometimes be considerable. By using “Small Data” and the insights from analysis of such data, there is an opportunity to invest less and build the behaviours and organizational systems and habits that will make a Big Data implementation effective.

Data and Strategy for Small and Medium Organizations

Data analytics and statistics aren’t historically associated with the strategic decisions that leaders take in small and medium sized businesses. Data analytics has for some years been used in larger organizations and organizations with larger user bases are also benefiting from this, thanks to the use of big data to drive consumer and business insight in business decision making. However, even such businesses can benefit from the large volumes of data that are being collected, including from public data bases. Most decisions in traditional businesses and in small and medium businesses are still taken by leaders who at best have a pulse of the market and a domain knowledge of the business they’re in, but aren’t using the data at their disposal to create mathematical models and strategies derived from them.

When does data fit into strategy?

To answer this, we may need to understand the purpose of strategy and strategic initiatives themselves. In small and medium organizations, the purpose of strategic initiatives, especially the mid- and short-term strategies, is to enable growth. Larger organizations have the benefit of extensive user bases, consumer bases or resources, which they can use to develop, test, validate and release new products and services. However, smaller organizations and medium sized organizations make these strategic initiatives, because their focus tends to be limited to the near term, and in maintaining a good financial performance. Small and medium organizations in modern economies will also seek to maintain leverage and a consumer base that is dedicated and loyal to their product or service journey. The latter is especially true of niche product companies, because they sell lifestyles, and not merely products.

In this context, data fits into strategy in the following key ways:

  1. Descriptive data analytics allows strategists and leaders to question underlying assumptions of existing strategies
  2. Data visualizations allow strategists to classify and rank opportunities and have more cost and time efficient strategies
  3. Inferential data analytics, predictive analytics and simulations allow strategists to play out scenarios, and take a peek into the future of the business

Descriptive data analytics may work with public data, or data already available with the organization. It could be composed of statistical reports, illustrating the growth in demand, or market size, or certain broader trends and patterns in consumption, or demand, for a certain product, service, or opportunity. Descriptive analytics is easy enough to do, and doesn’t involve complex modeling usually. It is a good entry point for strategists that hope to become more data driven in the development of their strategies.

Data visualizations, in addition to being communication tools that provide strategists leverage, could also throw some light on the functional aspects of what opportunities to seek out, and what strategies to develop. They could also help strategists make connections and see relationships that would otherwise not have been apparent. Data visualization has been made easier and more affordable because of powerful and free software such as R and R-Studio. Visualizations are extremely effective as communication and ideation tools. For strategists who look to mature beyond just using descriptive statistics in developing their strategies, visualizations can be valuable.

Inferential data analytics leverage the predictive power of mathematical and statistical models. By representing what is common knowledge as a mathematical model, we can apply it to diverse situations, and throw new light on problems that we haven’t evaluated before in a scientific or data driven manner. Inferential data analytics generally requires individuals with experience as data scientists. Inferential statistical models require a good understanding of basic and inferential statistical models, and therefore, can be more complex to incorporate into data based strategy models. While descriptive and visualizations may not be driven by advanced algorithms such as neural networks or machine learning, advanced and inferential analytics can certainly be so driven.

Data for Short and Medium Term Strategy

Data analysis that informs short term strategy and medium term strategy are fundamentally different. Short term strategy, that focuses on the immediate near term of a business, generally seeks to inform the operational teams on how they should act. This may be a set of simple rules, which are used to run the rudiments of the business on a day-to-day basis. Why use data to drive the regular activities of businesses for which extensive procedures may already be in place? Because keeping one’s ear to the ground – and collecting customer and market information on an ongoing basis – is extremely important for most businesses today in a competitive business world.  Continual improvement and quality are fundamental and important to a wide variety of businesses, and data that informs the short term is therefore extremely important.

Data analytics in the short term doesn’t rely on extensive analysis, but keeping abreast of information and the trends and patterns we see in them on a day-to-day basis. Approaches relevant to short term strategy may be:

  1. Dashboards and real time information streams
  2. Automatically generated reports that give operations leaders or general managers a pulse of the market, or a pulse of the business
  3. Sample data analysis (small data, as opposed to big data), that informs managers and teams about the ongoing status of a specific process or product – this is similar to quality management systems in use in various companies small and large

Data analytics in the mid term strategy space is quite a different situation, being required to inform strategists about the impact of changing market scenarios on a future product or service launch. The data analysis here should seek to serve the strategists’ need to be informed about served and total addressable markets, competitive space, penetration and market share expectations, and such business-specific criteria that help fund, finance or prioritize the development of new products or services.

Accordingly, data analytics in a mid-term strategy space (also called Horizon 2 strategy) may involve more involved analysis, typically by data scientists. Tools and themes of analysis may be things like:

  1. Consumer sentiment analysis to determine the relevance of a particular product or service
  2. Patents and intellectual property data munging, classification and text mining for category analysis
  3. Competitor analysis by automated searches, classification algorithms, risk analysis by dynamic analytical hierarchy processes
  4. Scenario analysis and simulation, driven by methods such as Markov Chain Monte Carlo analysis

Observe how the analyses above are distinct from the more ready information that’s shared with operational teams. The data analytics activities here generally require analysis of data in a rigorous manner, not merely the collection and presentation of available data that fit a certain definition. When data is unstructured and when data science requires the cleaning and visualization of data, the creation of models from a starting point, such as public data, is much more challenging. This is where the skills of well trained data scientists and data analysts is essential.

Data for Long Term Strategy

One narrative that has made itself known through data in the world of business, is that the long term as it was traditionally known, is shrinking. Even S&P 500 companies are conspicuous these days by how short lived they are, and small and medium companies, therefore, are no exception. Successful tech companies boast product and service development cycles of a few months up to a year, and the technology world is therefore unrecognizable from what it was every few months, thanks to innovation. However, there is probably a method to even this madness. The scale and openness of access has made the consumer and end user powerful, and the consumer these days has opportunities to do things with free resources and tools, that could only be imagined a few years ago.

Data informs strategists in such longer term strategic scenarios, typically, five years or more, by helping construct scenarios. Data analytics in scenario planning should account for the following:

  1. Dynamic trends in the increase of velocity in information/data being collected (Velocity, out of the four Vs)
  2. Dynamic changes in the type of information being collected (Variety, out of the four Vs)
  3. Dynamic changes in the reliability of information being collected (Veracity out of the four Vs)

Volume, the other V out of the four Vs, is a static measure of the data being collected at specific points in time, but these above are more than just volume, and they represent the growing size, variety and unreliability of available data.

Data analysis of a more simple nature can be used for some of the analysis above, while for specific approaches such as scenario analysis, sophisticated mathematical models can be used. In small and medium organizations, where the focus is usually on the short term, and at best on the mid term, data analytics can help inform executives about the long term and keep that conversation going. It is easy in smaller organizations to fall into the trap of not preparing for the long term. In the mid and long term, more advanced methods can be used to guide and inform the organization’s vision.

Concluding Remarks

Data analytics as applied to strategy is not entirely new, with many mature organizations already working on it. For small and medium businesses, which are mushrooming in a big way around the developed and developing world these days, data analytics is a force multiplier for strategic decision making and for leaders. Data analytics can reveal information we have hitherto believed to only be the preserve of large organizations who can collect data on an unprecedented scale and hire expert teams to analyze them. What makes analytics relevant to small and medium businesses today is that in our changing business landscape, we can expect analytics driven companies to respond in more agile ways to the needs of customers, and to excite customers in new ways, that traditional, less agile and larger organizations are not likely to do. The surfeit of mature data analysis tools and approaches available, combined with public data, can therefore make leaders and strategists in small and medium organizations more competitive.