Metrics: How to Improve Key Business Results Part 14

When I developed the metrics to answer these questions, I knew it was critical to produce an answer that would be accepted as accurate. Besides the need to have controls around the collection and a.n.a.lysis, I needed a basic construct for the metric.

I knew the quadrant. I knew the possible measures that would fit (or at least a starting point). I also knew how to test the measures for alignment with the Product/Service Health quadrant.

But to ensure we gained a comprehensive picture, I fell upon triangulation-the use of three or more measures to answer the question. In the case of effectiveness, we identified the primary ones-Delivery, Usage, and Customer Satisfaction-from the Answer Key, as shown in Figure 7-2.

Figure 7-2. The Product/Service Health quadrant.

Rather than select one or two, I determined that we should use all three, which would provide a fuller picture. Each measure had different characteristics in their sources and methods of collection.

Delivery addressed the need for objective measures of how well we provided our services and products. This measure would capture data on our effectiveness without customer involvement. The best source would be the trouble-tracking and reporting tools used by the IT department. One of the benefits of metrics is that it highlights tools you are already using.

By using the data from these tools, it drove the units to capture the data more accurately and frequently than they had been. Before the metrics program, the tools were not being used to their best levels and the resulting data was not useful-it was GIGO (garbage in, garbage out).

Usage was a means of capturing the customers' viewpoint un.o.btrusively, in this case watching how they "voted with their feet." This could be represented by unique customers (how many customers do you really serve) or repeat customers.

Customer satisfaction surveys allow you to ask how well you served the customer and understand what's important to them, but you can also ask open-ended questions to get direct input on how you can improve. While the IT department already had satisfaction surveys for each trouble case, these only gathered information from customers who expressed a problem with our service or product and were already using our product or service.

We also wanted to hear from those who hadn't had any issues with our service and perhaps even more importantly, those who had not yet tried our services. We needed to know why potential customers weren't turning to us to help address their needs. An organization can only grow so much from marketing to current customers. At some point, it needs to bring in new customers.

So, besides the trouble ticket follow up surveys, the IT department conducted annual surveys to the entire customer base. Depending on your business, you may need to partake in a sampling of potential customers.

Each area investigated had a customer viewpoint and addressed a different type of feedback-objective and independent customer feedback, indirect customer feedback, and direct customer feedback.

It would be nice if you could introduce the concept of triangulation and it would be accepted. It would be nice if you could introduce any of the concepts I've offered so far-from root questions, to the metrics framework and taxonomy, to the use of doc.u.mentation, how to use and not use metrics, the Answer Key, starting with Effectiveness-without having to fight for their acceptance.

When I attempted to create a metrics program for my own organization, I ran into a lot of resistance. I believed it stemmed from the "no prophet is accepted in her own village" syndrome. But after helping others develop and implement metrics programs, I now believe it's deeper than that. While the syndrome does make it difficult, even if you are an outside consultant, the refrain will be raised-"Just ask the customer! Our customer satisfaction surveys are enough!" Not only will it be shouted, a chorus will rise up in strong harmony.

Even today, I have to fight for a multi-measure metric. Not just using more than two measures, but in using ones from different views within the same quadrant.

Triangulation of Collection Methods and Sources.

Triangulation also requires different collection methods, as follows: Delivery is objectively collected and without customer involvement. In other words, the customer won't know you're collecting the data. You won't be using any opinions (all quant.i.tative data). Most times I also try to use automated collection methods for these measures (like trouble call tracking systems, monitoring systems, or time accounting systems). It is important to note that these do not measure customers, but how well the organization delivers the products and services. I will explain shortly.

Usage is a measurement based on customer behaviors. What do they buy? Who do they call? How often do they use our services? How did they find out about our services and products? How many one-time customers do we have vs. how many repeat customers?

Customer Satisfaction is the most customer-centric measurement group. Here, we directly ask the customer for their opinion. A better t.i.tle for this item would be "customer direct feedback." You ask the customer what they thought of the service and product, but you also ask for ideas for improvement. These questions can be asked in a survey, through focus groups, or through individual interviews. There are pros and cons to each, from varying costs to differing volumes of data collected. You should pick the methods that work best for you. Many times the customer base will dictate the best feedback tools.

Remember our three blind mice? When they "observed" the elephant, they had a hard time figuring out what it was until they were allowed to communicate what they observed (triangulation). Since they were able to identify the animals correctly, it would seem that there was no need to vary this strategy or methodology-only the sampling and source. A more accurate form of triangulation uses multiple methods and strategies. If the elephant were alive rather than a museum exhibit, one mouse could use touch, another smell, and the third could ask the elephant a well-thought-out set of questions.

Triangulation dictates that you use different sources, methodologies, and types of measures. Table 7-1 reflects the possibilities.

You can use other groupings for triangulation. I offer these because I know they work, and they are simple to implement. The idea is to address at least three different viewpoints, sources, and methods of collection.

The concept of triangulation can be used at each level of the metric. I don't suggest you go too deeply or you may find that you are collecting thousands of different measures. I want you to use triangulation at the top level-in the case of effectiveness metrics, at the Delivery, Usage, and Customer Satisfaction level. But, you can use the same concept at the next level.

For Customer Satisfaction, you could use the following three different methods of collection: surveys, focus groups, and interviews. This can be very expensive, especially interviews. But you could also use two different surveys-the annual survey given to a large portion of your customer base, and the trouble call survey. You can also use work-order surveys-questionnaires provided to customers when you deliver your service/product. This is another survey of your active customer base, but it doesn't require a problem. You can also administer satisfaction surveys three to six months after delivery to see if the customer: Continues to be satisfied with the service or product now that the initial excitement may have worn off Has new insights to the strengths or weaknesses of your product or service Has recommended your service or products to others Usage also allows for multiple measures. You can measure the number of unique customers, repeat customers, and the frequency of customer "purchases" of your services and products. I recently bought a new laptop from Best Buy and the experience was so enjoyable that I took my adult daughter there the next week. She bought a laptop for herself (nothing at all like the one I bought) and I couldn't help but buy a wireless keyboard and mouse (which I'm using to type this). Not only was I a repeat customer, but I referred someone else the next week (actually I drove her to the store, helped her pick out a computer, and offered to carry it to the car for her). All of this could be used to measure usage.

I saved Delivery for last because it is the one most easily triangulated. In the program I developed for my organization, I broke delivery into the following three major factors: Availability.

Speed.

Accuracy.

Availability is straightforward and not always applicable. Is the service or product available when the customer wants or needs it? In the case of telephone services, when you pick up the handset, do you get a dial tone? I just suffered through five days without my home service (I still have a landline for our home) due to a malfunction to the phone line outside the house. The service (product) was unavailable for five days. My wife and I both found this unacceptable. Availability can also be triangulated (remember, I warned about going too deep)-in this case, total outages, partial outages, and degradation of service are three viable measures.

Speed is one of the most powerful of the delivery measures, especially in our present society, even the global society-we all want things faster. Patience is mostly a lost virtue. So in my project, we measured speed to resolve, speed to restore, and speed to complete work orders. We also measured speed to deliver projects, software contracts, and new products. Another favorite at many companies is speed to respond-how fast do you return messages, answer the phone, or respond in person? Speed can easily be related to availability. For example, we not only wanted to know the number of outages there were and how long they lasted, but how fast did we respond, resolve the problem, and restore the service?

Patience is mostly a lost virtue, so expect many of your Product/Service Health metrics to be related to speed.

In the case of my phone outage, I was unhappy with the time to respond (five days to have the technician show up at my door), but I was pleased with the speed to resolve. The technician fixed the problem in less than 30 minutes.

The last area I used for Delivery was accuracy. This was normally captured in the form of rework. How many cases were reopened after the customer thought it was resolved? Others could include defects per products delivered. There are also errors in service. Did we deliver the right things? Did we deliver to the right people? At the right time? At the right location? There are many areas in which an organization can make mistakes. Accuracy measures the quality of your products or services from the customer's viewpoint.

Triangulation of Perspectives.

Triangulation is a simple concept. Use multiple sources of data when you can. Use more than one type of measure. More than one collection method. More than one perspective. In the case of the quadrants you are addressing, the differing perspectives is not referring to the following four viewpoints Customers'

Business's (managers') Workers'

Leaders.h.i.+p's.

Instead, I'm offering the following perspectives:.

Objective, non-customer involvement.

Customer behaviors (observation) and.

Direct customer feedback.

In the case of Effectiveness metrics (the place I recommend you begin), you are only dealing with the customer's viewpoint.

"The simultaneous consideration of intrinsic and extrinsic test factors forces every researcher to be self-consciously aware of how his every action can influence subsequent observations. From this perspective research becomes a social act."2 What if metrics became a "social act?" What if metrics, through triangulation coupled with open sharing of findings, were to become a collaborative event? A social act within the organization's society? It would help eliminate misuse and abuse of data, allow for multiple viewpoints to ensure accuracy of the data and, more importantly, lead to more meaningful interpretations of the metrics (answers). This social act may also lead the organization to ask better questions.

Demographics Don't Count.

You may mistakenly choose to triangulate your metrics based on demographics. This is not a true triangulation of the data, but decomposition. For example, at inst.i.tutes of higher learning, we frequently look at the data based on a demographic of our customer base faculty, staff, and students. Not every customer fits cleanly into one of these three categories, so we also look at non-inst.i.tute affiliates, officers, and family. While this provides multiple and different data, they are not different views, sources, and methods. They are just a breakout of the data already gathered. This decomposition could be very useful (especially when wanting to know a specific const.i.tuent's feedback) but doesn't const.i.tute triangulation.

These breakouts are very good for a.n.a.lyzing a subset of your customer population and allows you to address specific needs of each demographic. But it doesn't satisfy the requirement for triangulation.

______________________________.

2 Norman K. Denzin, Sociological Methods: A Sourcebook (Chicago: Aldine Publis.h.i.+ng Co., 1970), pp. 471-472.

According to Denzin, using multiple forms of triangulation within a single investigation leads to higher confidence in the (observed) findings. He also offers that the more diversity in the measures, the methods of collection, the people interpreting the data, and the sources directly correlates to the level of confidence. The higher the level of triangulation, the higher the level of confidence. The lower the level of triangulation, the lower the confidence.

I agree with this concept and would happily argue that you can have higher confidence in any metric made up of multiple measures than one made up of less. Answers derived from other metrics encourage an even higher level of confidence.

I don't propose that we seek out this level of confidence by demanding meta-metrics (metrics made up of other metrics) at every turn. Remember, some questions may not dictate even a single metric. Some may simply need a yes or no answer. Others may need a measure (or two) to provide the necessary insight to make a better decision. But, the more complex and critical the question, the more likely a higher level of confidence is called for. If your root question will be used to help decide on the future direction of the entire organization along with a considerable investment, you might demand a higher level of confidence in individual data used to build the metric and in the metric's overall conclusions.

Conflicting Results.

Because we use varying methods and data sources, we run the real risk of obtaining conflicting results. But, rather than see this as a negative, you should see it as a positive.

Let's look at our restaurant example. If our restaurant's effectiveness metric is made up of Delivery, Usage, and Customer Satisfaction, we may expect that the results of each of these measurement areas should always coincide. If we have good service (Delivery) we should have high Customer Satisfaction ratings. And if we have high Customer Satisfaction, we could a.s.sume that we should have high levels of repeat customers, and high usage. We also expect the opposite. If we have poor Customer Satisfaction, we expect that customers won't come back. If we don't deliver well (too slow, wrong items delivered, or the menu items are unavailable) we would expect poor ratings and less usage.

These are logical a.s.sumptions, but many times incorrect ones. Each of the permutations tells us something different. In Table 7-2, let's look at each measure with a simple high vs. low result. Of course, the real results of your measures may be much more complicated-especially when you remember that each can also utilize triangulation. Delivery could have high availability and speed to deliver, but poor accuracy. You could have high usage for one type of clientele and low for another. Customer satisfaction could have high marks for some areas (courtesy of staff) and low for others (efficiency of staff). Rather than complicate it further, let's look at the measures at the higher level, keeping in mind the complexity possible when taking into account lower levels of triangulation.

Besides the interpretation of each of the individual measurement areas, the triangulation itself offers information you would lack if you only collected one or two areas.

Using our restaurant business as an example, let's interpret some possible measures. In Table 7.2, you can see how different permutations of the results of the measures can tell a different story. While each measure provides some basic insights, it is more meaningful to look at them in relation to each other.

You may argue that triangulation seems to make the results more confusing, not clearer. But in actuality, triangulation a.s.sures that you have more data and more views of that data. The more information you have the better your answers will be. But in all cases the next step should be the same. Investigate, investigate, investigate. The beauty of triangulation is that you already have so many inputs that your investigation can be much more focused and reap greater benefits with less additional work.

Imagine if all you measured was Customer Satisfaction. If you ratings in this area were high, what could you determine? You could think life was good. But if you're not making enough money to keep your business open, you'll wonder what happened.

Triangulation not only allows you to use disparate data to answer a single question, it actually encourages you to do so.

Recap.

Triangulation is a principle foundation for a strong metric program. Triangulation has many benefits. The more triangulation used, the stronger the benefits. But the saying "all things in moderation" is also true with triangulation. You can overdo anything. You'll need to find the happy medium for you. Let's look at some of the following benefits: Higher levels of confidence in the accuracy of the measures used to form the metric Higher levels of confidence in the methods used to collect, a.n.a.lyze, and report the measures A broader perspective of the answer-increasing the likelihood of an accurate interpretation of the metric Satisfaction in knowing that you are "hearing the voice of all your customers"

A more robust metric (if you lose a measure, data source, or a.n.a.lysis tool you will have other measures to fall back on) Confidence that you are "seeing" the big picture as well as you can It is important to use triangulation in more than one aspect of the measurement collection and a.n.a.lysis, including the following: Multiple sources of data Multiple collection methods Multiple a.n.a.lysis methods (across measures and the willingness to apply different a.n.a.lytics to the same measures) Multiple areas (like Delivery, Usage, and Customer Satisfaction) or categories With all of this diversity it is important to stay focused. Collecting data from different quadrants in the Answer Key would not fit the principle of triangulation. If you dilute your answers by mixing the core viewpoint, you will run the risk of becoming lost in the data. If you lose focus and collect data from disparate parts of the Answer Key, it is probable that you are trying to answer multiple questions with only one answer. While meta-metrics use other metrics as part of their input, they must still stay within the context defined by the root question.

A solid metric can lead you in time to metrics in other areas of the Answer Key, but only after you've done your due diligence in answering the questions at hand.

Conclusion.

The concept of triangulation is not unique to the development of metrics and has been proven to be an effective guiding principle for using data.

Triangulation is a major principle in creating a viable metric program. It offers many benefits in return for the effort it takes to collect and a.n.a.lyze more data, from more sources, in multiple ways. Using triangulation on the Product/Service Health quadrant of the Answer Key garners you a robust set of measures to build your metric.

By concentrating on using a blend of sources, measures, and methods you end up with a comprehensive metric. You will have to still a.s.sess the metric for completeness to the question, but triangulation helps you get there with much less fuss and frustration.

Figure 7-3. The Product/Service Health quadrant A quick review of the Product/Service Health quadrant (Figure 7-3) demonstrates this, as follows: Product/Service Health Delivery Availability Speed Accuracy Usage Unique users Repeat customers Referrals Customer Satisfaction Annual surveys Trouble resolution feedback Interviews You can have more than three components for any of these. For example, you could include "reliability" in Delivery or "frequency of use" under Usage. Many times, the measures themselves dictate different methods of collection and a.n.a.lysis. Surveys are inherently different than interviews. While trouble resolution feedback is normally in the form of a survey, the types of questions asked are drastically different from the general survey administered annually.

Each piece of information (Delivery, Usage, and Customer Satisfaction) has a different collection/a.n.a.lysis methodology, but within each set you can further vary the tools, processes, and methods of collection/a.n.a.lysis.

When you combine the focus of the Answer Key with the comprehensiveness of triangulation, you will find yourself ready to promote a practical metric program to the organization.

Expectations.

How to View Metrics in a Meaningful Way.

If the Answer Key is the secret to unlocking the development of a useful metric program, the use of expectations is the keychain. Expectations allow you to change many of the common negatives toward measurement into productive improvement.

Don't Choose Poorly.

I love the line in Indiana Jones and the Last Crusade (1989) when the Grail Knight matter-of-factly announces, "he chose poorly." There were dozens of cups to choose from and the villain chose the wrong one. It was understandable, though. He chose the most ornate, jewel-bespeckled chalice on the table.