In part 1 of this article, we started our retrospective journey of understanding customer behavior that eventually helped improve the customer experience and halve the normalized number of support questions raised by customers. In the previous part of this article, we looked at communicating with customers about the self-service portal in non-intrusive ways including a how-to article and a web banner. The non-intrusive constraint is usual in the case of platform enhancements as they do not get a press release or a launch event.
In this part of the article, we will measure the success of the how-to article by analyzing page views using a new software. We’ll also increase the page views by referring to the self-service feature and to the explanatory how-to article at various customer touch points. Lastly, to continue the data-driven analytical approach, we will get page views per referral source by identifying the referrer links.
These steps together will enable us to build a data-informed process in boosting customer experience by reducing the friction for a customer in discovering a feature that they would value.
Thumbnail credits to PR-Distribution, Workana, and Design vector created by freepik.
Pointing to the FAQs and Changelog
To drive an increase in viewership of the FAQ article, I considered referencing it at multiple places. I was hoping that by adding the links in more places, more customers were likely to read the article. This meant a higher percentage of customers would be aware of the self-service feature and not need to wait for help from customer support.
The company’s changelog tool had the ability to alert customers whenever there was a changelog entry via email and used to post every update to Twitter. I added a changelog so that I can explain the “when” of the launch to customers and use that to point back to the FAQ.
I added the link at the top of the self-service feature in the web portal. I also added the link in the PDF report sent to customers and in the email that sends the report. I’ll leave out the details of the three more places that I added the link to. See the mockup below for the four places we mentioned above. Mockup thanks to API Changelog on Twitter and Hugo’s Changelog Theme.
Examples of software you could use for building a changelog are Tandora, Wagtail, Django, Drupal, Beamer, Launchnotes, Changelogfy, and Sleekplan.
Problem — I don’t know whether adding the links increased views of the how-to article or the self-service feature. If these actions did increase page views, by how much? To optimize future launches, what sources contributed the most?
Analyzing Page Views
I got help from the marketing team and technical writers about another tool that would analyze the page views of any article. I used this software to see the views of the how-to articles that I had created or updated. The software showed the number of page views for a given URL across a given period of time. The views for the support article I’d written had grown from 0 to non-zero, but I did not know whether it was good enough.
Examples of software you could use for analyzing page views on your website are Matomo, Smartlook, Mixpanel, Google Analytics, Clicky, Heap, Leadfeeder, Optimizely, Marketo, Adobe Analytics, FoxMetrics, and Woopra.
I started using the tool at around week 15-20 so I saw the recent and gradual increase in page views. A later view of the graph below showed that the number of views stabilized later on.
Several months later, I could use this tool to understand the overall increase in views to support articles. Since one of the gaps I filled through the year was to document tribal knowledge or FAQs, my hypothesis was that an increase in page views would be indicative of a decrease in potential support tickets.
Going back from the high-level post-analysis to the chronological discussion.
Problem — although I could see total views, I could not see the source of these views. How are customers discovering the article?
Another Problem — although I can see an increase in page views, how do I identify correlation vs causation? What was the increase in page views due to articles edited/created vs otherwise?
Enabling Source-specific CTR
How does our website parse URLs? How does our website know which page to open? What format or structure do we use to create URLs for new pages on our website? Learning from the technical writers, I realized that only a part of the URL is used by the website to determine which page to open. For example:
All these links lead to the same article. Using a similar insight, I renamed the URLs to the article from different sources. Revisiting the list of redirects I’d mentioned above:
Article itself: support.company.com/1029384756-how-to-use-this-self-service-feature
Created a changelog and linked it back to the article: support.company.com/1029384756-changelog-how-to-use-this-self-service-feature
Added the link to the PDF report the customers often use: support.company.com/1029384756-PDFreportFrontPage-how-to-use-this-self-service-feature
Added the link to the how-to at top of the feature page in the web portal: support.company.com/1029384756-web portalPageTop-how-to-use-this-self-service-feature
Added the link to the email that sends the report to the customers: support.company.com/1029384756-reportEmail-how-to-use-this-self-service-feature
Now, the analytics software is aware of the page views of each of these 5 links, and that helps me understand the source of the click on the article. After using the analytics software to see the data, I could see the number of views per source. Three takeaways from the graph:
The majority of the views are from unknown sources. Which is hard to utilize. This is possibly due to some browser or security features of users removing the redirects for CTR tracking.
Some of the views are from the links we had customizations for Click Through Rate (CTR) tracking above.
The views from some sources are seasonal in nature.
Based on the attributed sources, we know we are able to inform customers at the right touch points about a feature they need. You might wonder why did we choose this complicated approach to understand the link sources vs using built-in tools. Built-in referral tracking tools are easy and great to differentiate between online sources of traffic to your webpage. The advantage of using URL customizations over built-in tools is to additionally differentiate between traffic coming from an offline PDF or an email; we expect offline sources like these to also be significant sources of traffic and hence want to measure them.
However, the number of page views from the email still is very small which means it may not be a reliable indicator of how many people open the emails. Also, although we can see the page views of the support article from the email, the email also pointed to the self-service feature itself and if a lot of customers didn’t need to click on the how-to article and just clicked on the feature link itself, I won’t know the feature’s usage influx due to the email.
Problem — even though I did not see a lot of clicks from the emails I sent, I did not know whether the emails were opened. I did not know which other links were clicked in the email.
Takeaways
As a Product Manager or Product Marketing Manager, you need to know what tracking software do you have and what are its capabilities. Accordingly, how can you differentiate between referrers of traffic? If your referrers are other web pages within your web portal, it would be straightforward to use the funnel or path analysis tools in your software. If not, what approaches can you use to get the insights you want in customer behavior?
My other takeaway from retrospecting is that I should’ve persuaded and written a company blog post or PR regardless of the convention of not writing blogs for platform or settings-level changes. There are a few other “I should have” from the retrospectives that I’ll cover in an upcoming article.
Next Up…
In part 5, we will look at how to understand the engagement level of our customers with the links in the email, and what to make of the CTR report from this source. We will do experiments to test customer behavior when reading a document, and reduce the time spent by customer support in answering customer requests.
Originally published at https://harshalpatil.substack.com on May 12, 2021