Press enter to see results or esc to cancel.

15 Surprising Email Subject Line A/B Test Results You Need to Know

How much thought do you give to the subject lines of your emails? Are you A/B testing them on a subsection of your subscribers before rolling the winner out to everyone, or using them as a way to improve or refine your subject lines each time you schedule in a new campaign?

If not, you should be. A/B tests are simple to execute (and even simpler if you use an email marketing platform with the ability to split test built in). They’re also invaluable in helping marketers get more from their emails, and critical if you want to ensure you’re not only doing what works, but what works best.

To get you started, and give you some ideas for subject lines you might want to try out yourself, here are 15 A/B tests that show what works best isn’t always what you expect (which if you needed it, is one more great reason to test everything).

1. Including the word “free” in your subject line probably isn’t sending emails to spam (and can significantly increase open rates)

The Tumbleweed Tiny House Company used AWeber to split test two versions of a subject line – one containing the word “free,” and one without.

Subject line A: “It’s FREE. All the tiny houses on our site and more.”
Subject line B: “It features all the houses on our website plus more…”

The winner: Subject line A (26% more opens)

What’s going on?

The idea that using the word “free” in email subject lines can trigger spam filters is nothing new, and in some cases, may be true (if you’ve got a history of sending spammy emails for instance, placing “free” in your subject line probably isn’t doing you any favors).

If, on the other hand, you’ve got a clear track record with your email marketing platform, using “free” in your subject line is not only unlikely to affect the deliverability of your emails (because the sender’s reputation now plays a much bigger part in where an email ends up than its contents) but as the test above shows, could significantly increase open rates.

The reason why is simple – we really, really like free things.

2. And a synonym of “free” might increase open rates even more

A test using email marketing platform informz found that while the word “free” can boost open rates, using a synonym of it could increase them even further.

Subject line A: “Complimentary Online CME Courses”
Subject line B: “Free Online CME Courses”

The winner: Subject line A (3% more opens)

What’s going on?

There’s only a small discrepancy in open rates, which indicates that the same premise seen in test one must still be at play (basically, people like free things). The word “free,” however, can be linked to “low value” or “low quality.” “Complimentary” doesn’t share these same connotations, which allows you to tap into the consumer’s desire for a freebie, without devaluing your product.

3. Personalizing your subject line could be as simple as adding a “you”

Wishpond used A/B testing to help increase the effectiveness of an email campaign they used to boost sign-ups to their VIP demo. They found that some very simple personalization – adding the word “you” to the subject line – could boost open rates.

Subject line A: “Social Media Stats You Need to Know for 2014”
Subject line B: “Social Media Stats for 2014”

The winner: Subject line A (11% more opens)

What’s going on?

Personalizing an email subject line has long been endorsed as an effective way to increase open rates, but there is some evidence to suggest that consumers have become desensitized to the impact of seeing their name in their inbox.

This alternative addresses the recipient directly and feels personalized, without using the recipient’s name.

4. Money isn’t everything

A test by Optimizely set out to establish whether their audience was more motivated by discounts or by benefits.

Subject line A: “Get Optimizely Certified for 50% Off”
Subject line B: “Get Optimizely Certified & Advance Your Career”

The winner: Subject line B (13.3% more opens)

What’s going on?

Some audiences will be more motivated by benefit-led subject lines than those that push discounts. Try this out for yourself by writing subject lines that touch on how your product or service can make a positive difference to your recipients’ lives or businesses.

5. But context is everything

This is another test from Optimizely, from an email designed to promote an ebook. They made one small change to variant B – adding in the type of content they were promoting.

Subject line A: “Why You’re Crazy to Spend on SEM But Not A/B Testing”
Subject line B: “[Ebook] Why You’re Crazy to Spend on SEM But Not A/B Testing”

The winner: Subject line B (13.3% more opens)

What’s going on?

Ambiguous subject lines drive curiosity, which can increase open rates. I also have very little doubt that they decrease click-through rates and conversions. This is because a vague subject line lacks any targeting. Open rates might be high, but that’s because your recipients couldn’t resist finding out what’s inside the email.

It doesn’t mean they have any interest in what it contains.

The antithesis to an ambiguous subject line is one that details exactly what the email contains. Not only should this increase click-through rates, but as seen in the test above, it could increase open rates too (at least when compared to a similar, but slightly less descriptive version of the subject line).

6. Emails sent from real people instead of a company could see higher open rates and click-throughs

HubSpot conducted an A/B test to find out whether their audience responded better to emails that featured the company or a real person as the sender.

Subject line A featured “HubSpot” as the sender’s name
Subject line B featured a real person as the sender’s name

The winner: Subject line B (0.53% more opens and 0.23% more clicks – it also resulted in 131 more leads)

What’s going on?

We know an email that features a company as the sender’s name is part of an automated email campaign that’s probably trying to sell to us. Using a real person’s name feels more personal (even if it’s not).

7. Clarity trumps creativity

We’ve already seen evidence to indicate that subject lines which are upfront about the contents of an email are more effective than those that try to create “mystery” by covering it up, but this series of tests from AWeber proves the theory pretty much beyond all doubt.

They ran a simple A/B test across a range of emails. Each one pitted a “clear” subject line against something more creative.

Here’s a sample of some of the subject lines they tested.

The clear subject lines performed better than their creative counterparts across all metrics, and led to an average increase in responses of 541%.

What’s going on?

As we’ve seen above, consumers want to know what to expect when they open an email. Even if you can successfully increase open rates with a creative subject line, response rates will drop when your subscribers realize the content of the email isn’t relevant or interesting to them.

8. Emojis might help your subject line stand out, but they could be hurting open rates…

Emojis are commonly used to draw attention to emails in busy inboxes, but do they actually increase open rates? Creative consultancy agency YellowBall ran a series of tests to find out. Overall, subject lines with emojis were opened an average of 2% less often than those without.

This is in stark contrast to the results of the inspiration for YellowBall’s test – research by Experian – that showed a 50% increase in open rates for subject lines that included emojis.

What’s going on?

There are at least a couple of factors at work here. Demographics are going to play a part in how responsive an audience is to emojis. Older or corporate audiences are more likely to be turned off by the presence of emojis in subject lines than their younger counterparts, who are likely to be more accustomed to the role emojis play in digital communication.

They can also render incorrectly (appearing like this: □) and potentially flag spam filters (though as we touched on earlier, this is less of a concern than it used to be).

9. …Unless it’s a clock, used alongside a limited-time offer

Creative agency Brandwidth conducted an A/B test on the subject line of an email for their client, Royal Caribbean International.

Both subject lines included the text “Hurry – Offer ends in 3 days!” but variation B included an emoji of a clock alongside it.

Variation B saw a 3% increase in open rates and a 15% increase in click-throughs.

What’s going on?

Another factor playing into the effectiveness of emojis in subject lines is their contextual relevance. Don’t use them purely to draw your recipients’ attention to your email – use them as a means of reinforcing the message you want to get across.

10. Frequent emails won’t necessarily decrease open rates

This isn’t strictly about subject lines – although they did play a key part in the tests – something I’ll touch on below.

If you email your subscribers too often, they will unsubscribe from your list and that will in turn decrease your open rates.

Makes sense, right?

Tests carried out by Dave Huffman showed that this isn’t necessarily the case.

Huffman’s research involved sending one email a week for six weeks. He expected open rates to fall slightly each week, but they actually remained decidedly similar throughout the campaign.

What’s going on?

While there’s undoubtedly a limit to how often you can email your subscriber list before they’ll hit the “unsubscribe” button, if you’re consistently sending content worth seeing, and using direct, descriptive subject lines (another clear winner from Huffman’s research) this limit may be much higher than you think.

11. Putting yourself ahead of the customer can get you what you want – if you word it right

An A/B test for Money Dashboard set out to uncover whether making the recipient or their business the focus of the subject line would get better results.

Subject line A: “What’s your opinion here [first name]”
Subject line B: “Please put us out of our misery”

The winner: Subject line B (104.5% increase in opens for inactive subscribers and a 30.5% increase in inactive subscribers, as well as a 228.5% increase in clicks for inactive subscribers and a 103.3% increase in clicks for active subscribers)

What’s going on?

Subject line B actually goes against some of the takeaways listed above (not to mention best practices for writing subject lines in general). It’s very ambiguous, and the way it puts the sender ahead of the recipient is bordering on narcissistic.

This alone teaches us something – that there are few definitive lessons to take away from A/B testing emails. Every company, audience and email is different, and as such, what works for someone else might not work for you, and what works best once may not work best the next time.

However, the use of the word “misery” here is unusual. It helps attract attention and from there, invokes an emotional, empathetic response in the audience.

This shows that content which is all about you can work as well as, if not better, than content which focuses on your audience – at least if you word it in a way that taps into your audience’s emotions.

12. Dropping the corporate-speak is effective, even when the subject matter is serious

Obama must have one of the most-talked-about email campaigns of all time. His subject lines scorned all best practices yet got great results. Being Obama would have helped here. A lot. But relentless A/B testing played a big part too.

Specifically, his marketing team found that casual, personal and friendly subject lines worked best – even when the content of the email itself was somewhat more serious.

Surprisingly, their best performing subject line overall was just “hey.”

Image Source

What’s going on?

Simple – these subject lines replicate what you might receive from a friend or family member.

13. Using all caps isn’t always bad

Another test from The Tumbleweed Tiny House Company pitted a lower-case subject line against the exact same wording in upper-case.

Subject line A: “SOLAR LIVING GOES TINY”
Subject line B: “Solar living goes tiny”

The winner: Subject line A (18% more opens)

What’s going on?

The winning subject line is bold and stands out. Just bear in mind that using all caps in a subject line can trigger spam filters. Then again, as we’ve already seen, this risk is mitigated if you’re consistently sending quality emails.

To be absolutely sure, it might be worth trying this one out on a small subset of your best subscribers (i.e. those who read your emails most regularly).

14. Separating the themes of your email could be more effective than trying to summarize them into a single sentence

Estate agency Spicerhaart tested two very similar versions of the same subject line. One described the contents of the email in a single sentence; the other split it into three distinct parts.

Subject line A: “Mill View | Join us for our pre-launch weekend to find out more about these exciting new homes”
Subject line B: “Mill View | Pre-launch weekend | Quality homes affordably priced”

The winner: Subject line B (74% more opens)

What’s going on?

This is another really simple one to answer – subject line B is easier to skim read and as such, is much less likely to be overlooked and ignored.

15. Plain, boring and descriptive might be the most effective subject line style of all

This last one isn’t from a single A/B test – it’s the result of an analysis of 40 million emails sent through MailChimp. I wanted to wrap up with it, however, for two reasons:

  1. The sample size of the data
  2. The fact that the results surprised me more than anything else on this list

So, out of 40 million emails, the best performing subject line was simply…

“[COMPANY NAME] Sales & Marketing Newsletter.”

What’s going on?

We have to consider the specific circumstances surrounding these emails – they may have been sent to a very small, but highly-targeted list of recipients who always open emails from the sender in question.

That said, we can’t ignore the fact that they do something that’s helped many winning subject lines on this list get the edge: they clearly describe the subject of the email.

This goes to prove once again that simply being upfront and honest is all your audience really wants (a lesson that can coincidentally be applied to all elements of running and marketing a business).

Do you run A/B tests on your email subject lines? Have the results ever surprised you? I’d love to hear what happened, if you have a minute to share details in the comments below.

Image: Unsplash

Comments

6 Comments

Ali Mirza

Pretty solid post. Its good to see people are finally getting off the spammy subject lines, strictly going for open rates. While it is nice that your email got opened, but so do most “50% off Viagara emails”. Did the email turn into a lead or convert? Looking forward to the next post.

Jan Kuzel

I have a suggestion for a new blog post (or just question). I tried to research what are common open / reply rates for sales cold email outreach and it’s tough to find something of essence. I understand everything is pretty individual, but to at least get some general sense about how I am doing. In my last 260 recipients campaign, somewhat manual lead gen + creative service sales email outreach, I had 80% open rate and 26% reply rate and I have some idea about that being good, but how good? Are there some sources for this? Data, charts, etc.? On the internet I mostly find what seem to be mailchimp campaigns data, with leads collected from website or something like that and stats like average 25% open rate, that’s clearly something different from what I do with Mailshake and I can’t find data for that.

Nathan

Awesome post Sujan!

Just ran an experiment using MailShake to deliver 450 cold promo emails.
Subject line: article you might enjoy | Open Rate: 53%

Really enjoyed this, and love the actionable data and analytics you’ve provided.

Sujan Patel

Nathan,

Thanks! Great to hear about the 53% open rate 🙂

Victor

++ Agree with Jan Kuzel

Tim

I’m not surprised that Clarity won in #7. What’s also cool about the #7 examples is they provoke curiosity about the details, while being very clear about the subject. In fact, I bet Clarity was the champ for conversion rates, not just CTR.

In fact, the email that got me to this post was a great example of it

“15 Surprising Email Subject Line A/B Test Results You Need to Know ”

Very clear about what I’m getting if I open or click while also making me curious.

I’m sure I would have clicked fora #15-type headline too.

“Sujan Patel’s Latest Mailshake Blog Post”

I would have definitely opened it and clicked through but only because I’ve been conditioned to find the posts interesting and worth reading.


Leave a Comment