Engagement surveys - 5 problems, 4 solutions

It seems that every medium sized to large organisation now conducts some form of engagement survey annually.

On first glance, the reason seems clear enough. Research shows a statistically significant connection between organisation performance and employee engagement . And over 70% of leaders surveyed by the Harvard Business Review ranked employee engagement as “very important” to an organisation’s success. But are engagement surveys a great way to help boost an organisation’s prospects? 

Here are five problems with engagement surveys and four possible solutions.

Problem 1 – sampling error

Not every employee completes the engagement survey. In larger organisations (over 500 employees) completion rates of 70% to 80% are common.

While some of the reasons why people do not complete the survey are obvious – such as staff being on leave during the survey period – others are not.

The challenge is, what you take from fact that 20% to 30% of the population did not take part? If they had completed the survey would their responses have shown them to be more, or less, engaged that the participating group? What would their responses have done to the overall scores?

Unfortunately, it’s impossible to tell. But the impact of the non-responders on the overall result could be significant at an aggregate level or in a particular segment.

At an aggregate level, the non-responders may have a very different level of engagement from the responders. So, if their hypothetical scores could somehow be included in the overall result, that may move the overall outcome significantly.

The potential effect at the segment level is more concerning.

Perhaps response rates in a particular division were much lower than the overall group - so the aggregate data might not tell you very much about engagement in that division.

Or perhaps the organisation has an unconscious bias against – for example – LGBTQIA employees. If those employees feel that their views are not heard, then they may be more likely to decide there is no point to completing the survey. Accordingly, that group’s valid concerns will not be reflected in the survey results and the organisation’s unconscious bias problem will continue.

Problem 2 – influencing the results

The second problem with engagement surveys is that organisations become fixated on the results. If that happens, it is almost inevitable that people will start to game the process and influence the outcome.

Leaders in the organisation can, consciously or unconsciously, suggest that there are “right” and “wrong” ways to complete the survey.  If the result of the engagement survey becomes a KPI that leaders are measured against, it is likely that they will start suggesting that the “right” way to complete the survey is to score every question “highly engaged”.

In the worst cases, employees can become concerned that the promised anonymity of the survey will not eventuate. They can be worried that efforts will be made to identify and punish individuals who gave poor scores. Accordingly, they may survey as “highly engaged” when their true engagement is the opposite.

Problem 3 – point in time data

Engagement surveys only measure engagement at the point in time when they are conducted - they do not reflect the level of engagement over a period.

This creates two risks.

• Leaders may ignore declining engagement for a significant time. Assume changes occur in January that lead to a fall in engagement, but a survey is not conducted until October. Leaders may have spent nine months ignoring an issue they could have been doing something about. And they may now have a problem that is harder to fix than it would have been had they acted sooner.

• The timing of the survey may influence decision making. Assume the engagement survey is scheduled for October. In September, management has the chance to make a change that will have long term benefits but could lead to short term dissatisfaction. If leaders delay the change because they fear an adverse survey result, the organisation might lose the opportunity.

Problem 4 – what engagement means

There is a broad consensus that high employee engagement is a good thing. But there is no consensus about what employee engagement actually is, much less what questions should be included in a survey to measure it.

Here are some of the perspectives around what an engagement survey should measure - 

• Employees’ loyalty to the organisation
• Employees’ advocacy for the organisation
• Employees’ enjoyment of their work
• Employees’ focus on, and commitment to, their work
• Employees’ alignment to the organisation’s purpose and values
• Employees’ willingness to apply discretionary effort
• Some combination of the above.

It is understandable that different people have different views about what engagement means. And it makes sense that engagement in one organisation might be measured differently from engagement in another. For example, assessing employees’ support for an organisation’s innovation program might be more relevant to measuring engagement at a cancer research facility than it is at a national supermarket chain.

However, the fact that there is so much variation between different approaches to what engagement is and how it should be measured does suggest that engagement surveys cannot be the silver bullet – and perhaps not even the copper or tin bullet – of assessing employee engagement.

Problem 5 – what happens next

The biggest problem with engagement surveys is what happens after the survey.
Where the survey becomes an end in itself, there are three common responses to that question. None of them are positive –

• Some organisations do nothing. And many take some steps in the short term but do not sustain that commitment over the longer term. Shockingly, according to Gallup, only 8% of employees strongly agree that their employer takes action based on the survey outcomes.
  

• One of the common responses to a poor survey outcome is to look for people to blame. One division might blame another. One layer of management might blame another. Team members might blame each other. Repeated finger pointing sessions will not do anything to improve actual engagement.
  

• A third response is to ask employees to develop and implement action plans to improve engagement scores themselves without any support. In some respects, that seems logical. But when the sales team identifies problems with their IT systems in a survey and the management response is to ask the sales team to fix the IT system, the chance of a positive outcome is remote.

Four solutions

While engagement surveys have many flaws, they remain valuable tools and they can be useful if they are used the right way. Here are four tips to make engagement surveys more valuable.

• Make sure the engagement survey is a tool and not an outcome. Leaders should not be measured by the survey results and communications about the survey should highlight its role as only part of the organisation’s engagement program.

• Have a clear plan about how you intend to respond to the survey results. If there is no plan to invest real time and resources in addressing any issues the survey raises, then it may be best not to conduct a survey at all.

• Invest more time communicating what you do in response to the survey than you do communicating about the survey or its outcome.
  

• Use other means to measure engagement as well as a survey. Interviewing a relatively small number of randomly selected staff can provide a better data sample than a survey because it includes the views of people who do not respond to the survey. Plus, interviewing people will provide much more nuanced information about how engaged they are and what their key drivers of, or barriers to, engagement are.