Digital Lab. Orchestra of the Swan. Blog 2. A/B testing and The Learning Circle

With my first experiment for the Digital Lab and the implementation of Google Tag Manager, I thought I had reached the peak of my tech knowledge…but I was wrong. 

For my second Digital Lab experiment, as always spurred by my coach Tom Beardshaw, I decided to go further. Strangely enough, everything started with a 1-2-1 session where Tom and I discussed a lot about the learning circle which, by coincidence, is partly the basis of the Digital Lab philosophy. 

The learning circle is a practical model of effective learning in four steps: 

• Reflection
• Learning
• Planning
• Action 

We realised how difficult it could be, especially in arts organisations, to stop and spend time on Reflection and Learning, because arts marketers tend to go to and fro between Planning and Action. 

We thought that a good way to apply the learning circle would be a split test. 

A/B testing  (or split testing) is the practice of showing, in this case, the process of comparing two versions of a web page with just one varying element to determine which of the two variants is more effective.  

I used the A/B testing with some audio material I was working on. With the sound clip, I produced a podcast (audio only, which I uploaded on our SoundCloud channel) and a video (I found interesting free stock videos online which worked very well with the audio that I had and I uploaded it on Youtube). 

My initial thought was to send 2 different emails to our database (almost 4000 contacts): 50% would receive the version with the podcast, 50% the video. But I wasn’t entirely sure this would work as I wanted: what if I split the database incorrectly? What if I wasn’t able to split the database at all? I was pretty sure that with our current system for sending marketing massemails that could prove difficult. 

So I reflected on this (first step of the Learning Circle) to find the best way to manage this experiment and I decided to use another tool by Google, called Google Optimize, which was mentioned during one of the webinars by AMA with Devon Smith. I was aware of its existence, but I had never used it before. Google Optimize is a free product that allows you to get started with experimentation and offers A/B testing, website testing and personalisation tools for small businesses to help deliver engaging customer experiences. 

After a short period of learning (second step of the Learning Circle), I set up an account, configured it with the necessary data and linked it with Google Analytics.  

In the planning step I created the plan of action. I learnt that it would have been much more effective to send out a link to a webpage which showed, thanks to Optimize, the video or the podcast, randomly, instead of sending out a link to the video OR a link to the podcast. I could also share the same link on our social media.  

In order to do this effectively (action, the last step of the learning circle), I created a page on our website with the podcast and with Optimize I created an alternative version of the same page with the video: 50% of the visitors would see the version with the video, 50% the version with thepodcast. I shared the URL on our social media and I used it on our monthly e-newsletter which was sent to our entire database (4000 contacts).  

After a few weeks, I looked into the results of my experiment, which, I now realise, proved to be an experiment (A/B test) within an experiment (the integration of Google Optimize).Unfortunately, we don’t have enough traffic coming to our website to get feedback as Optimize needs a minimum of one experiment session per day, per variant.  

However, I could see that the total session durations were very similar (there is a difference of a few seconds between the time our visitors spent on the page with the video and the page with the audio) and that led me to think that, with our audience, podcasts and videos can work equally well.

I found the process of setting up a webpage which can have an alternative version that only half of our visitors can see extremely interesting. In the future I might focus my time on podcasts as they are easier to produce than videos. 

Paolo Pezzangora, Marketing Manager at Orchestra of the Swan