This article was first published in the ILTA Knowledge Management Community and authored by Holly Hanna, KM Firm Solutions Manager at Perkins Coie, and Paul Henry, Chief Executive at NGAGE Intelligence.
Whether you’re using the Objective & Key Result (OKR) framework, Key Performance Indicators (KPIs), or some other performance measurement framework, identifying how you will define success is an integral part of proving the value of your program or solution. When rolling out a new solution, or an update to an existing solution, many solution owners focus on user adoption as the key way they will measure the success of their work.
Each application deployed within your firm likely has a different set of adoption metrics. For example, for your Document Management System (DMS), it wouldn’t make sense to look at the total number of users who are accessing it, since usage of the DMS is mandatory in most firms. Instead, you might look at the usage of the Email Filing feature specifically. In the context of an intranet or other portal there will be several stakeholders, each one measuring success in their own ways. For example, a Practice Group Team Site Owner might want to see growth in the number of Content Contributors and Contributions, while the owner of the firmwide intranet is probably looking at growth in the length and depth of site visits. For the Governance Team, it could be all about highlighting Inactive Sites and taking action as needed.
We suggest there are two ‘universal’ KPIs that you will want to be able to report against, whatever your specific change management/adoption challenge: adoption rate and behavior rate. These KPIs define both the target user group and the behavior that you want to change.
The Universal KPIs
The value of any technology investment within an organization is a function of how well and how widely it is adopted by its target user population.
Adoption Rate = #Active Users/#Target User Population
The definition of your target population is hugely important. It might be the entire firm, but it could also be only attorneys, or only junior associates, or only paralegals in a specific practice group. You need to know (and keep track of) the precise size of your target population, so that you can move towards 100% adoption within that population.
But you can have problems with relying on this KPI in isolation. For example, if someone is automatically logged on to a SharePoint portal every morning they would count as an ‘Active User’ and superficial evidence of ‘adoption’. But if they don’t then otherwise interact during their working day, it’s hardly meaningful.
Behavior Rate = #Behaviors Count/#Active Users
You also need to be counting and seeing growth in the Behavior Rate (average activity levels) of Active Users. Even then you could be measuring high usage of a small number of enthusiastic early adopters, or the less enthusiastic who have no choice. And you might be getting zero or negligible activity among users who do have options and are choosing to stick with their ‘old’ way of doing things. As a result, on its own this, too, can be misleading.
So, while you should be tracking these individual metrics week by week, month by month, the most valuable insight comes from combining these metrics as a definition of ‘Engagement’. Because that’s what defines a successful implementation.
Reporting on KPIs
At Perkins Cole, we use an analytics platform called NGAGE, which allows us to track adoption of our SharePoint portal; it can also be extended to report on virtually any third-party product (e.g. Microsoft Teams, Yammer, Handshake, Confluence, iManage) and home-grown IIS and Angular apps.
NGAGE promotes the notion of continuous improvement rather than hard targets for KPIs. Each KPI visualization displays the value of the described Measure for the selected time period and the % change relative to the value for the immediately prior equivalent period. In this example the Adoption Rate of the target population over the last 30 days had fallen (and hence highlighted in red) by 20% relative to the preceding 30 day period.
As well as a configurable ‘traffic light’ indicator, each KPI visualization is accompanied by a Trend Line representing the daily value of the Measure during the selected period.
The Value of User Segmentation – Measuring Engagement
An enterprise class analytics tool will integrate with Active Directory to maintain a rich model of your organization. You can then use those user attributes to filter the KPIs to the precise Target User Population (i.e., certain departments or practices). The key here is that it should be easy to quickly ‘drill down’ to the user group you’re interested in and report on the underlying metric(s). Reports should also be intuitive and easily understood. These are powerful when used to track a single segment in relation to a single KPI. Even more so when used in combination to compare and contrast multiple segments.
For example, in the Engagement report below, you can filter by Department, Location and/or Job Title and see each segment represented by a ‘bubble’. Its size reflects the size of its population, while its position is a function of both its Adoption Rate and its Behavior Rate. The most engaged segment will tend to the top right-hand corner, while segments where you have low adoption but high activity from a small group of ‘power users’ will tend to the bottom right. The top left-hand corner is where you’ll find segments that are adopting but at minimalist levels of activity.
And note that the bubble report can also be scoped to an important application feature (like the Email Filing example mentioned above) or, in the context of a portal, to understand engagement with Search or a specific Site. In all scenarios this will help you intervene early and accurately where there are the most severe adoption problems. You will be able to measure the effectiveness of those interventions and learn what works. Equally, you can use it to identify the ‘hot spots’ of engagement so you can understand and share their success stories.
Identifying the right metrics to track and measure adoption is a key part of change management success, but in order to do so, you need to have the right tools to accurately measure the targeted behavior. The resulting reports should provide both actionable insight and be easily understood by management as part of the overall change management story while correctly identifying the key components of who, what, where, and when. With a combination of thoughtful KPIs and a user-friendly reporting tool your next change management initiative is sure to be a success.