Experiment: Post-Incorporation Services Package

Will the use of nudges and improved visual design increase the click-through rate (CTR) of links within Corporations Canada’s post incorporation email? These links lead clients towards beneficial next steps following incorporation to help support their business growth in the Canadian economy, but typically, many new business owners disregard these next steps.

This experiment was a quasi Randomized Controlled Trial, in which newly incorporated business owners received one of following five post incorporation emails: (1) the original email, (2) a newly formatted ‘checklist’ design, (3) a newly formatted ‘checklist’ design plus an authoritatively toned behavioral insights ‘nudge’, (4) a newly formatted ‘checklist’ design plus an urgent toned behavioral insights ‘nudge’, or (5) a newly formatted ‘checklist’ design plus a friendly toned behavioral insights ‘nudge’. The experiment ran between March 1st 2021 and April 7th 2021, during which approximately 300 emails were sent per day. Of those emails, around 70% were opened, and only around 5% of people clicked on the links provided in the original email. For one of the links, we found that compared to the original email, up to 30% of recipients now respond to some of the links when they receive the ‘checklist’ design plus an authoritatively toned ‘nudge’. The effect of each of each type of email varied by link (Odds Ratios ranged from 1.98 to 9.29), and were all statistically significant at the .001 level. Our results suggest that for some links, changing the look and feel of the email, and adding a nudge can improve click-through rate by up to 20%.

The experiment was a quasi Randomized Controlled Trial. Each link (total of 8) was analysed separately, using Logistic Regression, and using the original email as a reference condition in SPSS. The click-through ratio was calculated by dividing the unique clicks for each link (numerator), by the total unique opens of the email (denominator). The emails were distributed using a randomized block design, as individual level randomization was not technically possible. During the trial, each email type was distributed so that each email would have the chance to appear once, for everyday of the work week. For example, during the first week, the original email appeared on Tuesday, and during the second week, it appeared on Thursday, and during the third week, a Monday, etc.

The redesign of the emails involved arranging the text from the original email into a checklist format, which allowed it to be anchored to the nudge sentence, and also made the text appear less cluttered and more discrete. The nudge sentences were as follows: 1. Original email (no nudge) 2. Checklist format only (no nudge, to parse out the effect of the checklist only) 3. Checklist plus Authoritative Tone (‘Complete these next steps to keep your corporation on track’) 4. Checklist plus Urgent Tone (‘Complete these steps as soon as possible’) 5. Checklist plus Friendly Tone (‘Just a few more steps and you’ll be up and running!’)

Each link (8 total) in the email was examined using a separate logistic regression analysis, using email type as a predictor, the original email as a reference group, and CTR as the outcome variable.

It appeared that the type of email had an effect on click-through rate (CTR) for the first, third, and seventh link. Specifically, for the first link, it seemed that the new checklist formatted emails had around a 15% higher CTR than the Original email, and that by including an Authoritative nudge, it increased the CTR by an additional 5%. The third link, showed similar results, in that the new checklist formatted emails had around a 5% higher CTR compared to the Original email, although all of the new checklist designs seemed to perform equally higher in this case. The seventh link, indicated similar results to the last two links in that the checklist design outperformed the original email by about 5%, however this time, the Sense of Urgency had the strongest effect (see Figure 3). These findings aside, we did not see any effect of the type of email on CTR for the remaining links.

Randomized

Program

Sep 7, 2021

Centre of Expertise on Experimentation and Design Centre of Expertise

ISED-2021-00001

Innovation, Science and Economic Development Canada