Understanding how clients use software has always been important information for SSI to know because we use this information to help improve our products. We have thought carefully about how we collect information about user behavior because different strategies have different shortcomings and benefits.
Surveys
Surveys are a common way companies attempt to better understand their clients‘ current and future needs. However, I have always been skeptical of the value of surveys because I think to get real benefit from them, a company needs to ask really specific questions that do not leave much room for interpretation. For instance, using terms that are not clearly defined is problematic. Examples of that problem would be referring to PLM, PDM, integration, initial design, or basic design because those terms can be defined in many different ways. This can skew a survey’s results in a specific direction which may not match reality.
Another problem I see with surveys is that sometimes they do not have a truly representative sample. For instance, some companies entice users to participate by offering the possibility of winning a prize such as an iPad, Amazon gift card or free software. I know this works to get more participants but the question I ask myself is, “Are those people really a good cross section of the user base?” That thought brings me back to one of my statistics courses where the professor gave an example of how he could create a survey that showed that 100% of the people surveyed said that they shop at Superstore (it is a big Canadian grocery store). All he would have to do is stand out in front of the doors of Superstore and ask people when they were leaving Superstore if they had ever shopped at Superstore before. His point was that surveys are only useful when conducted properly. This has really made me mindful when I read survey results that even though the information it provides may be useful, it is only a piece of a bigger whole.
Monitoring (not like “Big Brother”)
Another more common strategy for understanding user behavior today, especially with cloud/off-site products, is recording absolutely every user action. This is relatively easy for cloud/SaaS companies to do and is actually integrated into their company processes. For example, many companies make small changes such as moving a button or changing the font of text and then analyze the changes in user behavior. I know this might not sound like a big deal, but some of these small changes can have big effects on user behavior. I think we all have heard about the Facebooks Emotion Study that may give the impression that tracking user behavior is “big brotherish” but as with anything that is good, there is always a dark side.
Fortunately, desktop applications have a different strategy which is usually voluntary for clients to participate in. This means the user needs to explicitly agree before the application can send any information back to the software vendor for any particular reason. You probably have seen this with several applications you use such as:
Autodesk
Microsoft
Apple
SSI
Why would you as a user care about sharing your user behavior with a software vendor?
Before I give an explanation, I would like to mention I am assuming the information being sent back to the software vendor does not contain any IP (Intellectual property) and only contains very basic information such as when and how often a feature is used, information to diagnose crashes of the software, etc. It does not track user passwords, individually identify users, or contain details to recreate information, company data, etc.
The more information you provide to a software company about how you use their product, the better they can allocate their resources to solve the problems users are actually having. I know this might sound a bit unbelievable to some of you, but the reality is that software companies struggle every day to determine what to fix, improve, remove and add to their solution. There is no shortage of work to be done and having actually quantifiable user information can help make decisions as to where a company should concentrate their efforts and that ultimately
I know at SSI we use our user voluntary CIP (Client Involvement Program) data to make many decisions. These decisions include:
- Let’s improve feature ‘A’ since many of our clients are using it and we feel that we can improve its performance.
- Let’s improve ‘X’ workflow because we have noticed a pattern of commands which can be streamlined.
- Our clients are not using feature ‘B’:
- So there is no need to spend resources on it.
- Which they may not be using because of lack of knowledge and documentation. Let’s educate them.
- Which might indicate that it was not designed properly.
- And many more.
AutoCAD has the same type of mechanism as ShipConstructor to monitor user commands which is 100% voluntary and is by default off. I talked to several Product Managers at Autodesk about their CIP data before SSI decided to include a similar type of client participation. They could not say enough good things about the data they were collecting and how often they use it to drive decisions within the product development. One use case a Product Manager told me about was that they found out that the most common command after the user executed the HATCH command was UNDO. The analysis of this information led to a dynamic hatch command which allowed the user to change the hatch before it was applied. Without the CIP data, Autodesk would never have prioritized this feature high enough to include it in a releasable product, especially because Autodesk had virtually no user requests to improve the HATCH command in this way. However, after they made the HATCH command more dynamic, they received many good reviews regarding this improvement.
Below is an image from Autodesk Research which shows one of the tools they use to analyze the data received from their CIP.
Closing remarks
From a software company’s perspective, understanding how users are actually using their software can help in analyzing trends and patterns to make much more informed product development decisions. This allows us to spend time on the right things or, as the saying goes, “Doing the right things.”
From a user’s perspective, providing feedback on how you use a software product can ensure that the product is developed with your current and future needs at hand. Most of the time there are mechanisms built into the software that enable you to share only relevant information to the software vendor. This information will be used to optimize the current product and make positive changes your company can benefit from.