How do you identify a measure for an attribute that is hard to measure?

When a management goal is specific, the relevant measure will be obvious. For example, suppose management set the following goal:

  “Customer calls are processed within four hours,”

 A relevant measure would be:

             The percent of customer calls processed within four hours.

But what if the performance goal is not about cycle time or some physical attribute that is easy to measure?  Suppose management wants to improve customer service and customer satisfaction by giving front line staff greater responsibility for day-to-day decisions when serving customers? Assume that management stated the following goal in its strategic plan:

“Empower employees to make decisions that best serve the needs of customers.”

How can “employee empowerment” be measured?  To identify a relevant measure, management would have to define what empowerment means and how an empowered employee behaves.  For example, management could define empowerment as follows:

Empowerment is defined as taking a non-standard but appropriate action to satisfy a customer need without management approval. The action is non-standard if it is not specified as a standard operating procedure for the situation.

To describe the behavior of an empowered employee, management could develop a performance logic model for the work process that an empowered employee uses to meet a customer need.  For example:

1. Employee receives a call from a customer with a concern.
2. Employee assesses the customer need.
3. Employee reviews the standard procedure for meeting customer need.
4. If the standard procedure meets the need, employee provides it, and the call is closed successfully.
5. If the standard procedure does not meet the need, employee creates a procedure and provides it.
6. If the employee-created procedure satisfies the customer, the call is closed successfully.

Using this logic model, management could measure progress toward the empowerment goal with this measure:

 A. Percent of customer calls resolved by employee-created procedures

Now suppose management determined that there was an organizational policy that is a barrier to employee empowerment such as requiring several levels of management approval before taking a non-standard action. Management might state the following goal in support of empowerment:

 ”Reduce the number of management reviews required for a non-standard procedure.”

To measure progress toward this goal supporting employee empowerment, management could use the following measure:

B. Number of management review required for an employee to use a non-standard procedure to meet a customer need.

Note that B is an upstream (leading) measure and A is a downstream (lagging) measure, but both are relevant to increasing employee empowerment. The ultimate lagging measure, however, is C below:

 C. Percent of satisfied customers who called with a concern

What is challenging about measuring employee empowerment is that empowerment has the nature of a capability, a potential for action, which may or may not result in observable behavior.  If no customer has a need that requires a non-standard procedure, measure A will not indicate the amount of empowerment in the organization. To get around this measurement difficulty, management would have to measure an employee’s state of mind with the following:

D. Percent of employees who say they feel empowered to use a non-standard procedure without management approval if one was needed to meet a customer need.

Employee empowerment is hard to measure and so it has to be surrounded.

As always, your comments are welcome by clicking on Comments below.

What are the good uses of organizational performance measures and data?

I am writing this post at the end of 2011.  In this season of making a list and checking it twice, I thought I would offer a list of the many uses of data that come from measuring organizational performance.  Here is my list, organized into three categories of use:

A.    Measuring Organizational Performance for Managing Performance [These uses are guided by management's stated expectations for success.]

•  Measures make clear management’s performance goals and expectations.

•  Measurement data inform managers about progress toward performance goals.

•  Measurement data inform employees of how well their work unit is doing.

•  Measurement data can suggest the need for preventive action before a problem occurs.

•  Measurement data can be used for organizational learning and improvement.

•  Measurement data document successful accomplishments.

•  Measurement data can be used to hold managers accountable.

B. Measuring Organizational Performance for Planning, Evaluation, and Decision-making [These uses are guided by specific problematic situation requiring study.]

•  Measurement data can be used to set organizational priorities.

•  Measurement data can be used to study organizational problems.

•  Measurement data can be used for allocating resources.

•  Measurement data can be used to make policy and program decisions.

•  Measurement data can be used to describe, evaluate, or audit programs.

C.    Measuring Organizational Performance for Scientific Research [These uses are guided by relevant bodies of scientific knowledge.]

•  Measures are used to test theories and hypotheses about organizational behavior.

• Measures are used to answer questions about a field of organizational policy or practice.

Here at the Managing with Measures website, I am primarily interested in category A, in particular, how to develop a system of measures that provide the right data about organizational performance on a timely and on-going basis.

I also have an interest category B which involves designing empirical studies to address a specific questions or problems.

 Wishing you all a happy, prosperous, and high-performing 2012!

 As always, your comments are invited by clicking on “Comments” below this post.

A question about using a Yes-No scale

A reader of this blog sent the following questions to Managing with Measures.

(1)   Is a Yes-No scale used to quantify by rating or is it used to quantify by counting?

(2)   I see a Yes-No scale being used to record two things—it exists or it does not exist; it works or it does not work. An example is a lamp in a hotel room. Are there other possibilities?

These questions go back to a post on November 2010 which discussed the three ways to measure anything—using a measuring device, counting, and rating. To review briefly:

• Quantifying With a Measuring Device

Time is measured with a chronometer; weight is measured with a scale, temperature is measured with a thermometer, etc. These devices each have a well-defined unit of measure which are counted to determine the observed amount of an attribute.  The device records the number of units observed.

•  Quantifying By Counting

Instances are counted in which a defined attribute is present.  An example is the number of ball bearings in a sample that fail to meet a specification.

•  Quantifying by Rating

A rating scale does not have a unit of measure that indicates an amount.  Instead human judgment is used to estimate the amount of an attribute. Examples are customer satisfaction and employee morale.

To answer the reader’s first question, Yes-No is a categorical scale with just two categories. It is an example of quantifying by counting, because it is used to determine the presence or absence of an attribute, not the amount of an attribute (a rating).

To answer the reader’s second question, you can use a Yes-No scale (or any type of categorical scale that has only two values that apply to all observations) to count the presence or absence of all sorts of attributes.  For example:

•  Number of employees who like their boss
• Number of travel vouchers that did not comply with company policy
• Number of rainy days in a month
• Number of registered voters who will vote in the next election
• Number of participants who rode the bus

Now, if you push on these examples, you could argue that each has to do with either the “existence” of an attribute  (likes boss, compliance with policy, rain) or whether the thing observed “works” or “does not work” (will vote, rode the bus).  So the reader may have a point in his lamp example.

What is a performance attribute?

I was teaching a seminar on stating and measuring strategic goals recently.  We discussed the different types of goals and I explained the “measurability” of each type.  (Types of goals and their measurability may be the topic of a future posting to this blog!)  When we moved on to the topic of selecting specific measures, one participant shared the thought that customer service can’t be measured.

I always enjoy it when someone says you can’t measure something.  It can be a “teaching moment.”  I personally think that customer service is easy to measure, and so I asked why he thought it was hard to measure.  His answer was that customer service is very complex.  I invited the class to imagine themselves in a customer service department and to identify specific characteristics of customer services performance that could be measured. 

They came up with the following list:

• Percent of satisfied customers
• Number of customer complaints received
• Number of new orders from customer referrals
• Number of customer calls for information or help
• Percent of on-time deliveries to customers
• Average time to deliver products to customers
• Average time to resolve a customer concern
• Average time that customers waited for a response after calling
• Cost of servicing a customer problem
• Backlog of unresolved customer problems

It became clear to the class that it is not hard to identify measures for customer service.  The key is to identify specific attributes of performance.  A performance attribute is a single characteristic of performance.  (Some authorities use other terms—dimension, property, aspect, factor—I prefer attribute.)  Each individual attribute will suggest a measure.  To measure anything—the size of a room, the talent of an Olympic figure skater, organizational performance—you have to identify each specific attribute you want to measure.  A single measure can only be applied to a single attribute.  If what you want to measure has several attributes, you will need a different measure for each.  A performance attribute is what a performance measure measures.

The person who thought that customer service could not be measured was making the mistake of assuming that the performance of the customer service department had to be measured by a single comprehensive measure that represented the overall performance of the department.  It is possible to represent the overall performance of the department in a single number but this is done by creating an index made up of individual measures of specific attributes.

I invite questions or comments by clicking on “Comments” immediately below this post, or you can send an e-mail.

What is a metric owner?

Like an employee, a measure of organizational performance needs someone to supervise it to make sure it is doing its job. That person is called a “metric owner.” The two primary responsibilities of a metric owner are:

• To regularly review the data produced by a measure to ensure that they  are trustworthy and useful for managing with measures.
• To look for and advocate ways that a measure can be improved or a better measure substituted if necessary.

Owning a performance measure does not mean that the owner is responsible for achieving the performance that is being measured. The responsibility for achieving higher organizational performance should be shared jointly by all members of the leadership team. Assigning ownership of measures to members of the leadership team is simply a way to ensure that the measures are functioning properly over time.

I facilitated the strategic plan and performance measures for a large corporation using the balanced scorecard methodology.  Here is how they shared measure ownership.

 The CFO owned the three measures in the Financial perspective:

• Number of sales quotes issued with total dollar value
• Number of orders received as a percent of orders received + lost
• Warranty costs from repairing or replacing as a percent of sales

 The V. P. Marketing owned the two measures in the Customer perspective:.

• Percent of surveyed customers who are very satisfied
• Percent of orders delivered on the original day of commitment

 The V. P. Production owned the three measures in the Internal Process perspective:

• Cycle time index of average percent improvement in critical processes
• Cost reduction index of average percent improvement of critical products
• Fixed costs as a percent of sales

 The V. P. Human Resources owned the three measures in the Learning and Growth perspective:

• Percent of employees who are very satisfied
• Number of employees leaving voluntarily
• Employee time lost due to accidents

It was understood by these leaders that they were mutually accountable for achieving the goals that were being monitored with these measures. For example, the CFO was not solely responsible for reducing the costs of honoring warranties, only making sure that the cost data were properly collected and reported.

What is performance benchmarking?

Once I attended a presentation on how an organization developed a system of performance measures.   The speaker said that they “benchmarked” their organization.  To do this, they visited several other organizations and reviewed their performance measures to get ideas for their own measures.

 What they did was make “site visits.”  They did not benchmark.

The notion of a benchmark comes from the field of surveying. A surveyor will measure the altitude of a selected position and mark it with a metal plate that documents the altitude at that position.  The surveyor then can measure the altitude at other positions using the metal plate as a reference point.  Essentially, a benchmark is a data point used as a basis for comparison.  In the field of organizational performance, a benchmark is a measured level of performance in one organization that is shared with another organization in order to compare performances.  To make this comparison, both organizations need to have measured their own performance.  Benchmarking is a way for two organizations to analyze and improve performance by sharing data on comparable organizational processes with each other.

To conduct a benchmarking study, one organization needs to find another organization that is willing to be a benchmarking partner. The partners then exchange performance data on a selected work process to learn how well each one is doing, which organization is performing better, and why.  An internal benchmarking study is carried out across divisions within a single organization.  An external benchmarking study is carried out between different organizations in the same or even different industries.  Sometimes a benchmarking study is conducted by an independent consulting firm that invites a number of organizations to participate in sharing data and best practices.  A benchmarking study needs to be carefully planned so that the data that is shared is comparable and useful to all participants.

Carl Thor in his book, The Measures of Success: Creating a High Performing Organization, describes the steps in a benchmarking study:  (1) Select a function or process to benchmark.  (2) Choose a benchmarking team.  (3) Gather data on organizational performance on the selected function or process.  (4) Select a benchmarking partner.  (5) Agree on the ground rules for confidentiality, data sharing, and schedule.  (6) Teams meet to compare data, discussing how each performs its work, and discuss the possibility of mutual site visits.  (7) In each organization, the teams implement performance improvements learned from the study.

On the distinction between measuring and evaluating performance

Recently I was involved in a disagreement about whether or not a strategic goal can be measured without a performance target.  The answer depends on what is meant by “measured.”  What I want to do in this blog post is clarify a very basic distinction that lies at the core of how performance measures are used to manage performance.

What complicates discussions of measuring organizational performance is the unfortunate reality that the word “measure” has many meanings in our everyday language.  The Oxford English Dictionary lists more than 30 different meanings for the word “measure” as a noun and as a verb.

• Among its many meanings as a noun, “measure” can refer to a definite quantity of something, a standard for evaluation, a measuring device, and a course of action to accomplish something.
•  Among its many meanings as a verb, “measure” can refer to estimating, marking off, judging, and weighing one’s thinking carefully.

So the word “measure” can refer to the act of determining the quantity of something (which is its technical meaning) and to the act of judging or evaluating something (which is not its technical meaning).  When discussing measures of organizational performance, the word measure should be used only in its technical meaning to avoid confusion and unproductive disagreements.

If we understand measurement as a process of determining the quantity of something, the job of a performance measure is to provide data that describes the current level of performance on some operational characteristic or attribute being measured.  Measuring performance is simply a systematic and objective method for observing the level of performance.  What measuring does not do is judge the observed level of performance.  Judging performance is the job of the manager and to do this, a manager needs to compare the measured level of performance to the desired level of performance (which is stated in the manager’s performance expectation).

The person who argued that you can’t measure a strategic goal without a performance target was using the word “measure” to mean evaluate.  The person who argued that you can measure without a target was using the word measure in its technical sense of determining the quantity or level of performance.  Using the same word with different meanings caused an unproductive discussion.

In summary, when reviewing and evaluating organizational performance, the manager considers four things.

• A performance attribute:  The specific aspect or dimension of organizational performance that is being reviewed and evaluated.
• A performance expectation:  What the manager thinks is adequate, satisfactory, or acceptable performance for the specific performance attribute at the time of the review.
• Data from a performance measure:  A quantitative description of the observed level of performance on the attribute being reviewed.
• A performance evaluation.  The manager’s judgment of how satisfactory the observed level of performance is in comparison to expected performance.

Why do employees resist measuring organizational performance?

This blog post is an excerpt from my forthcoming book, Managing with Measures:  How to Select the Right Performance Measures to Manage Your Organization.

When managers are implementing a system of performance measures, they will likely encounter resistance to the measures from the workforce (managers as well as employees).  I have observed two types of resistance to measuring organizational performance.

Resistance to the possibility of measurement

The first resistance has to do with whether it is possible to measure organizational performance, especially the important attributes of performance. This is resistance about the very idea of measuring something as complex as an organization.  It is expressed in different ways, such as “numbers don’t tell the whole story,” or “what is really important can’t be measured,” or “what we do can’t be measured.”  Behind these concerns is a fear that measuring will misrepresent, not accurately represent organizational performance.

To address this resistance, personnel need to be given opportunities to understand measurement and express their concerns.  For example, it is true that numbers don’t tell the whole story, but this is no reason to avoid measuring altogether.  Numbers tell part of the story and the challenge is to measure what is important.  It is not true that what is “really” important can’t be measured or that the collective work of employees can’t be measured.

Ways to address these concerns include distributing data reports to all personnel and by having employees participate in developing and using measures.  Also, being members of a team that uses measures to improve team performance and whose improvements are recognized by management can make converts of all but the most incurable skeptics.  Resistance to the possibility of measurement is overcome through learning.

Resistance to management’s use of measurement

This resistance has to do with how managers will use measurement data to manage organizational performance.  Employees who understand and accept the value of measuring performance could at the same time be concerned that a manager might use measurement data to find fault with or embarrass employees, or that a manager could manipulate data about poor performance to make management “look good.”  These  concerns reflect a lack of trust in management.

To build trust, the use of performance measures needs to be visible and transparent so that skeptics can  observe what managers do with the data. One way to be transparent is to involve employees in reviewing and interpreting performance data.  Also,  when data suggest that performance is unsatisfactory, management should let employees look at the data and find a way to fix the problem.  Finally, when data show that management’s expectations are met, this success should be celebrated with the employees.  Resistance to management’s use of measurement is overcome by building trust in management.  It’s not really about measurement.

I invite questions or comments by clicking on “Comments” immediately below this post, or you can send an e-mail.

What motivates a manager to measure organizational performance?

In the professional literature, the popular way to promote the measurement of organizational performance is to list its many benefits for managing performance.  When I review these lists, I keep coming back to the three benefits that are most important.  I discussed these three benefits in an earlier post to this blog in December 2009.  To remind you, performance measures help to (1) clarify management expectations for performance, (2) monitor progress in achieving performance expectations, and (3) document performance success.

The question addressed in this post is not about the benefits of measurement but about what would motivate a manager to seek these benefits.  Hearing about a benefit is not in itself a sufficient motivation to take action.  A motive is an inner drive that causes each of us to engage in action—to make an effort to achieve a desired result.  A motive provides the energy that sustains us in our purpose, especially when we encounter difficulties.  Our motive keeps us going.

The following is an excerpt from my forthcoming book, Managing with Measures: How to Select the Right Measures to Manage Your Organization.  There are two motives for measuring organizational performance.

Motive 1:  Measuring to comply with accountability

Some higher-level authority may require you to report on performance periodically.  Or the authority may “call you to account” after your performance has gone wrong and an explanation is needed.  It is reasonable for you to comply with accountability, because it is reasonable for an oversight authority to require it.  Having a system of performance measures in place makes you ready to report when required.  Measuring provides data for reporting to higher-ups.

Motive 2:  Measuring in the pursuit of excellence

Most of us would like to be better at what we do and we would like our organization to be better at what it does.  It is the responsibility of a manager to see that the organization gets its work done day by day, week by week, and month by month.  But a manager should also see to it that the organization gets better at getting the work done—this is the leadership component of managing.  Measuring provides you with data to manage performance more successfully.

The motives compared

Is on motive to be preferred over the other?  From the point of view of performance measurement, the answer is Yes. The motive to comply with accountability and the motive to pursue organizational excellence have an asymmetrical relationship.  When you measure for accountability, you get data only for compliance. When you measure for excellence, you get data on a continuing basis to improve performance.  The data for improvement will also be the data for compliance, and the data for both will always be within reach.

Part 1: How can measurement data be combined?

There are three ways to think about combining measurement data—aggregation, indexing, and aligning. This post discusses data aggregation. The other two ways will be discussed in future posts.

When a performance measure is being used in several organizational units that do the same work and measure the work in the same way, you can consider combining the data from all units, called “data aggregation,” to measure the performance of the units as a single work organization. For example, think of the sales division of an organization that has regional offices and collects weekly sales data at each office. The sales data can be analyzed separately for each office, but they can also be aggregated to report on sales for the sales division as a whole.

Just because the same measure is being used to collect data in different work units does not require that the data be aggregated. A wise principle in managing with measures is to let each work unit analyze its own performance data. On the other hand, if the manager of the sales division wants to know how the sales division as a whole is doing, the data from all the regional offices have to be aggregated. This is sometimes called “data roll up.”

If you want to aggregate or roll up several sets of data from the same measure, there are two important considerations.

• The operational definition of the measure should be the same in all work units being aggregated.
• Data sets must be aggregated as raw data before they are analyzed.

Consider the case of a sales division with two regional offices. One week, the first office makes 50 sales totaling $5,000 for an average weekly sale of $100. In the same week, the second office makes 100 sales totaling $5,000 for an average weekly sale of $50. The average sale for the division is not the average of the average sale for the regional offices. To compute the average sale for the division, the two data sets from the regional offices must be aggregated into a single data set. The sales division as a whole made 150 sales totaling $10,000 which is a weekly average sale of $66.

The advantage in aggregating data is being able to report organizational performance at a higher level higher in the organization. The disadvantage is that the higher level performance hides the variation in lower level performance. Once the data are aggregated, you can no longer tell which units are doing well and which are not. To not lose this capacity to see the variation in lower level performance, the process that “rolls up” the data should also be able to a “drill down” to see each lower level unit separately.

As always, I invite questions or comments. Submit a comment  in the space below.