Behavioural Insights Pilot Project – Roofing In the Underground Economy

Executive summary

It is estimated that $40.9 billion of economic activity goes unreported annually within Canada’s underground economy. The largest sector in the underground economy is construction, accounting for 28% of total underground economic value^{footnote 1[1]}. According to the Canada Revenue Agency’s most recent estimates, as much as a fifth of private-residence contract work (home renovations) in Canada take place in the underground economy. This economic activity not only deprives the government of tax revenue on which public services rely, it also places homeowners and workers at risk.

Without recording large home renovations projects, such as reroofing a home, in a contract, homeowners expose themselves to common scams, legal liability, and poor workmanship. Furthermore, roofing is dangerous for workers who do not receive proper training, equipment, and insurance, something which illegitimate businesses may be unlikely to provide. Unfortunately, despite these substantial costs and risks, surveys reveal that many Canadians perceive that paying cash in order to evade service tax is socially acceptable. To address this issue, the Ministries of Labour, Government and Consumer Services and Finance partnered with the Behavioural Insights Unit on an online advertising campaign designed to make homeowners aware of the risks inherent in the underground economy by directing them from their online searches for roofers to the Ministry of Labour’s website.

Behavioural insights is a tool that leverages research from behavioural economics, psychology, and marketing in the work of government. By running randomized controlled trials, behavioural insights practitioners are able to better understand how people respond to different contexts and incentives. Through this understanding, decision-makers can design and implement better policies and services.

Together with academic collaborators from the University of Toronto’s Rotman School of Management and Queen’s University Business School, the Behavioural Insights Unit designed 16 advertisements for this campaign. In partnership with the Ministry of Labour’s web development team these advertisements were submitted to two randomized controlled trials, one on Google and the other on Kijiji, in order to assess which were the most effective at increasing traffic to the Ministry of Labour’s Buying a Roof webpage.

Overall the pilot was a success and showed that well-designed online advertising campaigns can help increase Ontarians’ awareness about the pitfalls of engaging in the underground economy. Together the advertisements on Google and Kijiji increased traffic to the Ministry of Labour’s website by 144%. Furthermore, conservative calculations show that if only 1.2% of this traffic resulted in homeowners who decided not to reroof their homes in the underground economy, the campaign would have paid for itself in terms of sales tax alone.

The results of the randomized controlled trials show that two of the Google advertisements and four of those tested on Kijiji performed significantly better than the other advertisements and well within industry performance standards. The best performing advertisements attracted two to three times the number of visitors than did the less effective advertisements.

This report provides an overview of the project, including the method used and results. It also highlights the benefits of applying the principles and methods of behavioural insights to develop, test, and deliver effective messaging.

Introduction

The Underground Economy’s Substantial Cost

According to the Canada Revenue Agency’s most recent estimates, $40.9 billion of economic activity goes unreported annually in the underground economy. It is estimated that the size of the underground economy has remained stable over the past decade at approximately 2.3% of the country’s gross domestic product^{footnote 2[2]}. Although Canada’s tax-system enjoys a high level of integrity and has among the highest compliance rates in the Organization for Economic Co-operation and Development (OECD) membership^{footnote 3[3]}, this unreported economic activity represents a significant loss to both the Federal as well as the Provincial governments’ revenue streams because it is not subject to sales tax or personal and corporate income taxes. In 2012, the Commission on the Reform of Ontario’s Public Service, led by Don Drummond, recognized that curtailing the underground economy was an important means to eliminating the government’s deficit ^{footnote 4[4]}.

The underground economy is any business activity that is unreported or under-reported for tax purposes^{footnote 5[5]}, for example paying for services in cash—such as home renovations or auto body repair—in order to avoid paying harmonized sales tax (HST). It is estimated that as much as a fifth (20%) of all home renovations in Canada are conducted in the underground economy^{footnote 6[6]}.

Consumers who participate in the underground economy, either intentionally or unwittingly, put themselves and others at risk by transacting outside the realm of legal protections. Without a written contract, consumers have no warranties, risk being unable to seek legal remedy for poor workmanship, are often subject to financial scams, and risk liability for injuries or damages that occur from the work. Consequently, home renovations and repairs have consistently been among the top three consumer complaints to the Ministry of Government and Consumer Services over the past decade^{footnote 7[7]}.

Others who may be negatively impacted by the underground economy include workers and legitimate businesses in the economy. Businesses who do not comply with their tax obligations may also be less likely to comply with other legal obligations, such as training, protections, and access to benefits.

The most common underground economic activity is paying for goods or services in cash to avoid HST. In one recent survey 35% of Canadians admitted that they were somewhat or very likely to engage in this behaviour in the coming year^{footnote 8[8]}. Conversely, only seven% admitted that they would likely leave any of their income unreported on their tax return. Given that about a third of Canadians view sales tax evasion as socially acceptable^{footnote 9[9]}, participating in the underground economy poses substantial costs and risks to our public services, consumers, workers, and the economy as a whole.

The Ministry of Labour’s focus on roofing

The scale of underground economic activity varies by sector. By far the biggest sector in the underground economy is construction, accounting for 28% of total underground economic value^{footnote 10[10]}. In this sector, a focus on roof replacement and repair was particularly important to the Ministry of Labour because of its concern for worker safety. Falls are the number one cause of critical injuries and deaths on construction sites in Ontario^{footnote 11[11]}. Many of these falls could be prevented with proper training and safety equipment, which are required under Sections 26 to 26.9 of the Regulation for Construction Projects. Unfortunately there are many compliance challenges when it comes to enforcement of this regulation, especially with regard to the roofing contractors which tend to operate in the underground economy. Therefore the Ministry of Labour’s decision to concentrate its efforts on tackling underground economic activity in the roofing business supports tax revenue recovery, consumer protection, and worker safety.

Online advertising as a means of alerting homeowners

Consistent with the recommendation of the Commission on the Reform of Ontario’s Public Service, the Ministry of Labour decided to run a public awareness campaign advising homeowners about the risks of participating in the underground economy for roof repairs and replacements. Several factors lead the Ministry of Labour to conclude that an online advertising campaign would be the most appropriate channel for this campaign.

Recommendation 18-7

Develop a public awareness campaign on the impact of the underground economy. For example, by using unregistered contractors or contractors who do not issue receipts, there are risks of not obtaining a warranty for repairs, risks of not being able to seek legal remedy for poor workmanship, and risks of liability for injuries or damages that occur on a customer’s premises.

Commission on the Reform of Ontario’s Public Service

1. Online campaigns can be targeted: In contrast to broadcast or print advertising, online advertising can be targeted to only those consumers who are currently in need of roofing services, which makes it more cost-effective. Additionally, most online advertising is paid on a ‘cost-per-click’ model, so that the government only pays when consumers exhibit interest in the ad and end up visiting the government’s website.
2. Online ads are timely: Online advertising can be delivered at the very moment when a consumer is in the process of making the decision of which roofing contractor to hire. Importantly, decisions are influenced by the information we remember at the time we make a decision, and we are more likely to remember information we have encountered recently. So, the timely delivery of information just prior to, or during a decision is more likely to influence that decision^{footnote 12[12]}
3. Anyone can advertise online: People offering roofing services can advertise on online classified websites, such as Kijiji, for free and provide nothing more than a telephone number or an email address. This makes enforcement on these sites very challenging and potentially allows underground operators to reach a large audience of consumers. Placing the Ministry of Labour’s advertising in such locations targets those consumers who may be at greater risk of participating in the underground economy.
4. The effectiveness of online campaigns can be measured: Online advertising allows consumers to be directly hyperlinked to the Ministry of Labour’s website^{footnote 13[13]}. This also allows the government to track the effectiveness of online ads.

The Pilot

Designing the messaging

In order to design advertisements that were likely to be the most effective at attracting consumers’ attention and encouraging them to engage with the Ministry of Labour’s website content, the Behavioural Insights Unit drew upon established marketing and behavioural science research. For example, although increasing tax compliance is a major motivation for the government when it comes to underground economy participation, it was assumed that consumers would be less motivated by this rationale. Research has shown that people are often unmotivated to contribute to public goods and the above mentioned public opinion research showed that many Canadians view HST evasion as socially acceptable^{footnote 14[14]}. Instead, each of the advertisements in the two trials spoke to one of the following consumer motivations.

1. Homeowners’ immediate motivation— choosing an appropriate roofer (Kijiji ad #1 and Google ads #1-3): A traditional advertising strategy is to identify and address consumers’ goals and then explain how a given service offers a unique value proposition to satisfy those goals. In the case of residential roofing, the immediate goal of homeowners searching online is to choose from among the available roofers. The unique value proposition of the Ministry of Labour’s website is the provision of impartial information which can help homeowners make an optimal choice.
2. Homeowners’ financial motivation (Kijiji ads #2 and #3): As with all economic exchanges, the financial motive of the buyer is to find a good or service for the lowest possible price. Unfortunately, this financial motive may drive some homeowners into the underground economy because many roofers will offer thousands of dollars in discounts for cash deals. Thus the team created two advertisements which recognize this financial motivation and then illustrate how cash deals actually undermine it.
3. Homeowners’ ultimate motivation in a positive frame—protecting their home (Kijiji ad #4 and Google ad #4): In addition to appealing to individuals’ goal to protect their home, the term protection also implies a danger from which protection is needed. Fear of danger powerfully motivates action, especially when actions to avoid the danger are provided. As opposed to the word “house,” the word “home” is used in these advertisements because it has emotional connotations of family, helping to make this goal of protection more meaningful.
4. Homeowners’ ultimate motivation in a loss frame–avoiding risks or damage to one’s home (Kijiji ads #5-8 and Google ads #5 and #6): The flip side to protecting one’s home is avoid risks and damage to one’s home, and past research has shown that messaging focussing on these risks more explicitly tends to be more motivating. People are risk averse, which means that they pay more attention to and are highly motivated to avoid potential dangers in their environments. Google ads 5 and 6 leverage the negative stereotypes that many homeowners have towards contractors to highlight the risks associated with the underground roofing economy. The pre-existing stereotypes make the risk seem more real and concretely associating that risk with contractors will hopefully increase the attention that homeowners give to the process of hiring a roofer. In contrast, Kijiji ads 5-8 and Google ads 7 and 8 leave the potential risks unspecified. Both Kijiji and Google’s advertising formats do not leave much space to specify all of the risks that homeowners face, so the intention with these advertisements is to allow their imaginations to fill in the gaps. The term “roofing scams” is intended to cover such risks as intentionally low-balled prices for customers who are not protected by written contracts and thieves who take deposit payments but never return to complete the promised work. The phrasing “don’t be the next victim” also conveys that there are many victims because this is a pervasive problem.
5. Conveying source legitimacy: In addition to addressing the various motivations which may be at play in homeowners’ roofing decisions, it was also thought that conveying the trustworthiness of the messaging was important. Consumers may be wary of false information, scams, and malware online, so establishing that the messaging came from a legitimate source that could be trusted was thought to be important. Therefore all of the advertisements explicitly mention that the messaging comes from the Ontario government. In addition, Kijiji advertisements 5, 6, 7, and 8 tested the importance of communicating this legitimacy visually using the government’s logo. The government’s logo was compared against an internationally recognized warning sign as it was expected that this sign would both attract attention and communicate the need to beware of danger.

The advertisements designed for this trial are presented in the Table 1. Both Google and Kijiji have strict formatting requirements for advertisements that appear on their services. Google advertisements are text only and are a maximum of three lines of text, with a 25 character limit on the first line and 35 character limit on the second and third lines. Kijiji advertisements recommended for the Ministry of Labour’s budget by the advertising agency feature an 86 by 86 pixel image and up to four lines of text, each with a 30 character limit.

Table 1

Serial number	Google Ads	Kijiji Ads
1.	Find the Right Roofer! www.Ontario.ca/beforeyoubuyaroof Info on roofers and consumer rights from the Government of Ontario.	Want help choosing a roofer? Need roofing advice? Get helpful info & advice from the Ontario government here.
2.	Need roofing advice? www.Ontario.ca/beforeyoubuyaroof Advice and information on roofing from the Government of Ontario.	No-contract roofing’s NO deal. Cutting corners, no warranty, the RISKS can cost you $1000s! Get helpful info & advice here.
3.	Unbiased roofing advice www.Ontario.ca/beforeyoubuyaroof Advice and information on roofing from the Government of Ontario	Don’t choose the wrong roofer. The hidden costs can be $1000s. Get helpful info & advice from the Ontario government here.
4.	Protect your home! www.Ontario.ca/beforeyoubuyaroof Protect your home with advice from the Government of Ontario.	Your roof protects your home. So does a roofing contract. Get helpful info & advice from the Ontario government here.
5.	Trustworthy roofer?? www.Ontario.ca/beforeyoubuyaroof Info on roofers and consumer rights from the Government of Ontario.	WARNING: Beware roofing scams! Don’t be the next victim. Get helpful info & advice here.
6.	Don’t hire a bad roofer! www.Ontario.ca/beforeyoubuyaroof Info on roofers and consumer rights from the Government of Ontario.	WARNING: Beware roofing scams! Don’t be the next victim. Get helpful info & advice from the Ontario government here.
7.	Avoid roofing disasters! www.Ontario.ca/beforeyoubuyaroof Protect your home with advice from the Government of Ontario.	WARNING: Beware roofing scams! Don’t be the next victim. Get helpful info & advice here.
8.	WARNING: roofing scams! www.Ontario.ca/beforeyoubuyaroof Protect your home with advice from the Government of Ontario.	WARNING: Beware roofing scams! Don’t be the next victim. Get helpful info & advice from the Ontario government here.

Testing the Conditions

Evaluating the messaging

To evaluate the effectiveness of the advertising messaging two randomized controlled trials were conducted, one on Google and one on Kijiji. The trials ran for 34 days at the start of the 2015 roofing season from February 23 to March 31, 2015. During this time the advertisements received 4,297,485 impressions (were displayed this many times) and the Ministry of Labour’s web development team tracked the resulting 5,970 visitors to their web page. All advertisements in both trials were also run in French, but these advertisements were displayed too few times to permit statistical analyses so they are not discussed further.

Google trial

The Ministry of Labour paid to advertise on Google for a set list of keywords which might be used by homeowners searching for roofing services. When someone within Ontario entered those keywords into the Google search engine, Google’s proprietary algorithm determined whether and where to display a randomly selected advertisement from this trial. This determination is based on a number of factors including the amount bid by the Ministry of Labour, the match between the advertisement, the keywords, and the Ministry of Labour’s website, and competition from other advertisers bidding on the same keywords. In total, the advertisements that were part of this trial were shown 89% of the time someone searched for one of the keywords on which the Ministry of Labour had bid, which resulted in 22,980 impressions (the number of times an advertisement was displayed).

When a consumer clicked on the advertisement, they were brought to the Ministry of Labour’s website and the government was billed the amount they bid for the advertisement.

Kijiji trial

The Ministry of Labour identified an advertising format and placement on the Kijiji website that accommodated their budget. Consumers browsing Kijiji’s classifieds for roofers would randomly see either the Ministry of Labour’s advertisements or ones from private businesses.

The eight Ministry of Labour advertisements were displayed randomly. In order to achieve random display of the advertisements, the Ministry of Labour paid to have each shown approximately 535,000 times, for a total of 4,274,505 impressions.

Results

Overall the pilot was a success and showed that well designed online advertising campaigns can help increase Ontarians’ awareness about the pitfalls of engaging in the underground economy. During the course of the trial the advertisements on Google and Kijiji resulted in an additional 3,299 visits to the MOL website, which is a 144% increase in traffic to the site relative to traffic from all other sources (e.g., traffic from other pages within the Ministry of Labour’s website or from the Ministry of Labour’s social media campaign, etc.). The advertisements on Google contributed 656 additional visits (a 37% increase relative to traffic from other sources) and the advertisements on Kijiji contributed 2,643 additional visits (a 107% increase in traffic).

If only 1.2% of those people who visited the Ministry’s website because of these advertisements decided not to participate in the underground economy when reroofing their home, the campaign would have paid for itself in terms of additional revenue captured in the Province’s portion of HST (8%)^{footnote a[a]}. This estimate is based on the Ministry of Labour’s subject matter experts’ estimate of the average cost of reroofing a house in Ontario at $10,000, and does not include any additional revenue which would accrue through corporate and personal income taxes.

Google results

37% increase in traffic attributable to the Google advertisements is the result of all eight advertisements combined. However, the purpose of the randomized controlled trial was to determine which of these advertisements performed the best.

Analyses of the trial results reveal that two advertisements significantly outperformed the rest. ^{footnote b[b]}. The best performing advertisements were “Don’t hire a bad roofer” and “Warning: roofing scams,” which indicates that homeowners are most motivated by the desire to avoid being taken advantage of by unscrupulous roofers.

Best Performing Google Ads

• Don’t hire a bad roofer!
  • www.Ontario.ca/beforeyoubuyaroof Info on roofers and consumer rights from the Government of Ontario.
• WARNING: roofing scams!
  • www.Ontario.ca/beforeyoubuyaroof Protect your home with advice from the Government of Ontario.

These advertisements had a ‘click-through-rate’ (the% of times the advertisement was clicked for the number of times it was shown) of between 3.88 and 4.85%. By running only these two ads during the same time period, we estimate that this would have resulted in approximately 1,003 visitors to the Ministry of Labour’s website, increasing traffic by 41%. In contrast, if the worst performing advertisements, which had click-through-rates of 1.39%, had been run exclusively during this period, only 319 visitors would have resulted.

Kijiji results

As with the Google trial, the purpose of running a randomized controlled trial with Kijiji advertisements was to determine which of eight ads drew the most visitors to the Ministry of Labour’s website. Analyses reveal that the four advertisements which contained the line “WARNING: Beware roofing scams!” performed significantly better than the other four advertisements. This is consistent with the finding from the Google trial, again suggesting that avoiding harm at the hands of companies that would conduct business in the underground economy is the most effective motivation for attracting the attention of homeowners in need of a roofer.

Best Performing Kijiji Ads

WARNING: Beware roofing scams!
Don’t be the next victim.
Get helpful info & advice here.

WARNING: Beware roofing scams!
Don’t be the next victim.
Get helpful info & advice here.

WARNING: Beware roofing scams!
Don’t be the next victim.
Get helpful info & advice from the Ontario government here.

WARNING: Beware roofing scams!
Don’t be the next victim.
Get helpful info & advice from the Ontario government here.

The click-through-rate for these best performing advertisements was 0.08%, which is within the industry standard performance range of 0.06-0.13% for government advertisements on Kijiji. By running only the best performing ads during the same time period, we estimate that this would have resulted in approximately 3,438 visitors to the Ministry of Labour’s website, increasing traffic by 139%. In contrast, if the worst performing advertisement, “No-contract roofing’s NO deal!” had been run exclusively during this time, only 1,599 visits would have resulted.

A direct comparison between the performance of advertisements on Google and Kijiji cannot be made on the basis of data collected in these CTRs alone. For example, although the Google advertisements exhibited a much higher CTR than did the Kijiji advertisements, this is because the Kijiji advertisements were shown many more times (Google: 22,980; Kijiji: 4,274,505). As a result, although the Kijiji advertisements had lower CTRs in our trials, they also drove hundreds more (Google: 656; Kijiji: 2,643) visitors to the MOL’s website during the trials. Ultimately Google and Kijiji are very different platforms; they display advertisements in different locations and formats, have different cost structures, and may attract different clientele looking for different things. Therefore, to evaluate which platform is more important for the MOL’s objectives would depend on much more exhaustive data than was available in this trial. Nevertheless, if we assume that both Google and Kijiji are important in the roofing market then the research conducted in these trials is valuable because it has identified optimal advertisements for each platform.

Conclusion

The Google and Kijiji advertising trials were a great success. By employing a randomized controlled trial methodology, the Ministry of Labour and the Behavioural Insights Unit were able to test multiple advertisements simultaneously.

The "worst case" columns in Table 5.1 illustrates the amount of web traffic that would have resulted if only the poor performing advertisements had run during the trial – they would have increased web traffic by only 82%. This is an important possibility to consider because all of the advertisements we developed seemed promising on paper, but only a minority performed optimally when we actually tested them in the field. As a result the Ministry of Labour can be confident going forward that they are deploying high performing advertisements in their campaign, and expect the amount of web traffic to their site to look more like the “best case” columns in Table 5.2, in which their advertisements are increasing web traffic by approximately 180%.

All together during the trial the advertisements that the team tested increased traffic to the Ministry of Labour’s website by 144%. If just 1.2% of these 3,299 visits resulted in roof replacements that were not conducted in the underground economy these advertisements would have paid for themselves in sales tax alone.

The benefits of better informed consumers who are less likely to engage in the underground economy extend beyond additional tax revenue. The more homeowners who avoid the underground economy when reroofing their home, the more are protected from scams, unnecessary legal liability, and expensive damage to their most valuable investment. Workers in the construction industry are also more likely to experience safe and fair working conditions when fewer employers are shirking their legal responsibilities, and the competitiveness of legitimate businesses is enhanced when consumers reduce their support for underground deals.

Questions and inquiries

To learn more about behaviour sciences research insights in Ontario email us with your questions and comments.