With many business units, web domains, and campaigns, large companies face far more challenges performing conversion testing than smaller companies.
I recently sat down with James Niehaus, Director of Optimization and Web Analytics at Symantec to talk shop about the challenges of testing in a large organization, and how Symantec has established a thriving testing program across multiple business units and functions. Symantec currently runs a handful of tests each month, ranging from simple headline experiments to complex merchandising strategy tests, and is driving towards having over one hundred tests per month.
Says Niehaus: “Conversion testing is the most critical strategic lever we have. If properly implemented, it allows you to optimize all your other optimization tactics (like SEO, paid search etc.). Conversion testing should be part of every customer touch point or business function.”
This post is a redux of 10 practical takeaways from that interview that you can apply to your optimization process. (Though most applicable to large companies, many of the tips are helpful to businesses of any size).
Organizational Structure and Support
1. Ensure you have executive support
Niehaus believes executive or revenue owner buy-in is the “single most critical piece to any organization big or small. If your leadership is on board, they can make things happen for you.”
Unfortunately, for many organizations, convincing the HiPPO (highest paid person in the organization) to back testing is an uphill battle. Stay tuned, in our next post we’ll explore strategies for gaining buy-in from key decision makers.
2. Create a conversion center of excellence
A center of excellence is “a formally appointed, and informally accepted, body of knowledge and experience on the subject area. It is a place where the highest standards of achievement are aimed for in a particular sphere of activity” (thank you http://en.wiktionary.org/wiki/center_of_excellence Wiktionary, I couldn’t have said it better myself).
Although Symantec has always believed in the importance of optimization, it has not always been a core competency of the company. Previously, its ecommerce operations were outsourced. As Symantec has brought those activities in-house, a renewed focus on making optimization testing part of the DNA of the company emerged. Symantec’s center of excellence is a 7-person team with a diverse skill set, weighted heavily towards optimization and web analytics, but also including technical support (e.g. experience with APIs and JavaScript).
According to Niehaus, it helps to speak the language of IT if you want your voice heard.
“You must be able to speak in terms of the IT folks to get your code on your site. Once it’s on, you can do a lot yourself. That’s the power of the tool. But it can’t happen until you get the code implemented.”
The center of excellence is responsible for test design, execution, and measurement. In order to deliver effective tests in an efficient fashion this team must thoroughly understand every unit of the business and ‘how they deliver’. An end-to-end view is critical says Niehaus because “I have to propose tests and ideas that can get done.”
The center of excellence also works to help others in the company apply best practices around conversion optimization to their particular focus.
Centralize or decentralize?
Having led optimization for four companies, Niehaus believes that businesses should not all operate in the same way.
“Developing the structure that’s right for you is dependent on your organizational structure as a whole. My division is a reflection of my company.”
Symantec started off with a very centralized conversion program and has now started decentralizing.
“To centralize or decentralize comes down to two things: organizational resources and your approach to global. What is the organizational strategy around globally supporting the business – to centralize or decentralize? Your optimization program will align with that.
If your business lines are standalone or siloed, then it makes sense to have an optimization expert take the lead in each business unit. It depends on how much autonomy each group has and how much their activities need to be coordinated with those of other groups.”
3. Get cross-functional buy-in
By necessity, multiple departments are involved in any conversion testing program. In addition to executive sponsorship, Marketing provides strategic input and business guidance. User Experience designs the concepts. QA validates the code. Development, Merchandising, IT, Program Management, and the platform team are also all involved.
“Until optimization is understood by most business units and embraced, they will see it as more work. And it is more work. If they have not bought into the vision, they will present challenges. Whoever drives the optimization project has to paint the vision and appreciate the challenges of each business unit, otherwise, they’ll be caught up in roadblocks and red tape.”
Technology
4. Aim for platform flexibility
You can only move as fast as your business moves. The larger the organization, the more technological barriers will pose a challenge. Ideally, optimization would be built into the actual platform but a close second is a flexible platform. And the longer the lead cycles, the more difficult testing becomes because of the processes and structures in place. Conversion testing relies on rapid release cycles (daily, not yearly) to roll out tests in a timely manner.
Process
5. Use an enterprise testing tool
Symantec’s tool of choice is Test&Target, part of the Adobe Omniture suite, a market leader in the enterprise space. While free or low cost tools like Google Website Optimizer are sufficient for many small and medium sized businesses, larger organizations that are serious about testing and have a need for advanced functionality should consider enterprise tools.
6. Crawl before you walk
When you begin conversion testing, don’t jump head first into complex multivariate tests. Year one is about finding low hanging fruit and showing value. Year two is about refining execution and doing more complex tests.
“You don’t do too much segmentation or profiling at first. As your program matures, it becomes more sophisticated, and you start doing more advanced tests to find those hidden gems and constantly innovate.”
7. Consider both sides of the optimization coin
Before you launch a test, it’s important to consider the potential upside and downside – there is always the risk that a test might perform worse than your control. Understand your traffic, conversion rate etc. before the test.
“You plug in the numbers to get a realistic understanding of the impact and potential risk. You need to provide four or five scenarios to the business so they understand in advance and are prepared for the unexpected. Experience helps here and you want to watch closely so you can take advantage of good and bad results quickly.”
8. Prioritize tests appropriately
“Test ideas come from everywhere. Each group has different goals and a different view of the business that’s valuable.”
With many sites and pages to test, how do you determine where to start?
Niehaus prioritizes tests based on their potential value to the business in terms of bookings and end units, the level of effort needed to execute a given test, and what areas are available to test. Since you need to wait for test results materialize, you can’t launch a new test on the same page or flow while another test is running.
9. Gain insight from tests methodically
Symantec’s methodology for picking a winner varies depending on test complexity. Single page tests are different from multi-page tests. A/B and multivariate tests are different. Price tests require modeling to predict optimal pricing.
Test&Target is leveraged to get a real-time view of the results, and depending on the test they examine KPIs, average dollar value, product mix and/or how purchase behavior is impacted. Decisions are made based on high-level metrics but take into account the user experience.
Because Symantec sells subscription software, they look at multi-year behavior to understand how changes might affect future business. Consider the behavioral changes you’re generating when you modify the purchase process, messaging, pricing, etc.
10. Roll out intelligently
We’ve written before on Get Elastic about the benefits of rolling out a test winner slowly, rather than flipping a switch suddenly.
Symantec tests regularly across their e-stores in 10 to 15 countries, but mostly in the US and UK as these regions generate the lion’s share of revenue for the business. Once they are confident they have a winner, Symantec switches on the variation in Test&Target first and then will work with the business units to implement the change as a permanent part of the platform.
However, when they find a statistically significant winner they don’t simply roll it out globally. A winner in one channel or region doesn’t necessarily imply a winner in the other channels or regions.
Big thanks to James for sharing his secret sauce for site optimization with us. Stay tuned for our next post, where we’ll tackle the challenge of getting executive buy-in for your testing program.
Looking for help with A/B and multivariate testing? Contact the Elastic Path consulting team at consulting@elasticpath.com to learn how our conversion optimization services can improve your business results.