{"id":250,"date":"2019-12-09T09:00:22","date_gmt":"2019-12-09T09:00:22","guid":{"rendered":"https:\/\/keith.resar.org\/blog\/?p=250"},"modified":"2019-12-06T22:30:08","modified_gmt":"2019-12-06T22:30:08","slug":"earning-thousands-by-solving-the-wrong-email-problem-part-1","status":"publish","type":"post","link":"https:\/\/keith.resar.org\/blog\/earning-thousands-by-solving-the-wrong-email-problem-part-1\/","title":{"rendered":"Earning Thousands by Solving the Wrong Email Problem (Part 1)"},"content":{"rendered":"

Stumbling into a web site that actually pays (the email-format.com story) – Part 1<\/p>\n

<\/p>\n

I’m sharing my startup stories while building out my latest project – Calenzen<\/a> – fixes the meetings you’re invited to so you can always instantly click-to-dial. No special apps, calendar programs, or software is ever needed.\u00a0 Learn more about Calenzen<\/a>.<\/em><\/span><\/p>\n

My Problem, But Not <\/b>The<\/i><\/b> Problem<\/b><\/p>\n

The idea behind this project came after a job interview. \u00a0I wanted to send the hiring manager a follow-up email, but I never got his address. \u00a0I spent a few minutes with a search engine looking for \u201cexample.com email address formats\u201d and similar searches. \u00a0It didn\u2019t take long to find what I was looking for. Drilling into the search results it showed a clear pattern of \u201cfirst.last@example.com.\u201d<\/span><\/p>\n

\"\"
While a current search brings email-format.com<\/a> to the top of the results, historically finding the best format took sleuthing through dozens of search results pages.<\/figcaption><\/figure>\n

The next day I spent a couple hours pondering how to automate this process. \u00a0Weren\u2019t there people in the same situation, looking to make a good final impression after an interview?<\/span><\/p>\n

I envisioned a web site that would do this work for me, so visitors could instantly find the email address format in use at every company. \u00a0It seemed easy enough that after just one evening hacking on code I could already see real results.<\/span><\/p>\n

The Real Problem<\/b><\/p>\n

This project is a success story, to be sure. \u00a0I\u2019ll take you through all the gory details soon enough, but I\u2019d be remiss if I left my earlier vision of \u201cthe problem to be solved\u201d as anything near commonplace. \u00a0As the project grew, my understanding of actual customer drivers matured.<\/span><\/p>\n

The typical visitor to my email format website wasn\u2019t looking for a resource to use as a one-time lookup. \u00a0They were using the site as a resource for complete customer contact information as part of lead generation campaigns. \u00a0<\/span><\/p>\n

Most visitor traffic came from repeat visitors, not organic, and it came primarily from off-shore sources. \u00a0In talking to customers, they were looking for lower cost access to data than is already available from other resources. \u00a0The cost difference was easily one or two orders of magnitude.<\/span><\/p>\n

\"\"
List price for Clearbit’s service. Expect similar metered prices from Salesforce, Discover Org, and others.<\/figcaption><\/figure>\n

The Solution<\/b><\/p>\n

Whatever the problem visitors to the email-format.com<\/a> web site are looking to solve, the solution ultimately remains the same. \u00a0In the ten years this site has been running, it has gone through three major iterations.<\/span><\/p>\n

With each new version the UI, underlying technical model, and even the commercial model adapted to take advantage of changing traffic trends.<\/span><\/p>\n

Version 1, Where it all Began<\/b><\/p>\n

I started this project questioning how best to get the data I needed. \u00a0There was no way I\u2019d consider any solution that required ongoing manual effort, so manual data collection was out of the question. \u00a0Additionally, given email-format.com\u2019s long-tail nature, each piece of data itself wasn\u2019t too valuable. Therefore the revenue model couldn\u2019t support paying for data sourced by another party.<\/span><\/p>\n

The first thing I needed was a list of interesting domains. \u00a0Ideally, a zone transfer of the entire .com database (which of course isn\u2019t available). \u00a0Failing that, I found a friend in the Alexa top sites database. This offered a publicly accessible csv containing the top 1000 .com sites.<\/span><\/p>\n

\"\"
Access is more restricted now than 10 years ago, but as an original source the Alexa top sites list proved critical.<\/figcaption><\/figure>\n

A thousand domains is a start of course. \u00a0And with that, the second thing I needed was to get the email address formats associated with these domains. \u00a0I wasn\u2019t about to create my own Google-style web index. But maybe I could take advantage of someone else\u2019s?<\/span><\/p>\n

At the time all major search engines – Yahoo, Bing, and Google – allowed limited API access to their search indexes. \u00a0Yahoo\u2019s offering was particularly attractive, permitting something on the order of 10,000 monthly searches without charge. \u00a0Given that individual data points weren\u2019t particularly valuable, any paid API access could never have been profitable, especially since only a small subset of search result pages contained email addresses to analyze. \u00a0<\/span><\/p>\n

While staying within these small search limits, I slowly indexed all email addresses associated with those top Alexa sites. \u00a0That meant I needed a new source of domains. But how could I tell what new domains existed? And more importantly, of the millions of domains not yet in my index, how could I tell which ones would be the most valuable?<\/span><\/p>\n

I solved this with a neat plugin called the \u201cJob-a-Matic\u201d from simplyhired.com\u00a0 This was a snippet of JavaScript code that would run with each site visit that showed the visitor a list of available jobs. \u00a0If anyone clicked on a link and was hired, my website would get a small referral fee. <\/span><\/p>\n

\"\"
Long since abandoned, SimplyHired hosted the “Job-a-matic” widget. I accessed the API from the backend and given keywords and the lat\/long coordinates of my visitor and it returned a list of domains.<\/figcaption><\/figure>\n

While I was never paid out for referrals, I did get access to continual stream of \u201chot\u201d companies. \u00a0Remember – at this point I\u2019m still focussing on the niche use case of job hunters looking to connect with someone they know at a company, but whose email address they don\u2019t have. \u00a0In this context that means the companies (and therefore domains) to focus on should be actively recruiting.<\/span><\/p>\n

So using the \u201cJob-a-matic,\u201d I had a fresh stream of target domains that I could burn my no-charge Yahoo API search credits on.<\/span><\/p>\n

I built all of this over the course of a few evenings and didn\u2019t give it a second thought for a long time. \u00a0Traffic was low, and ad click through rate was even lower. Google AdSense owed me a few pennies, but they would not pay out until I reached $100 or more, which at the time was an insurmountable threshold.<\/span><\/p>\n

One day, after almost two years, traffic had increased enough that AdSense advertising was actually cutting a monthly check. \u00a0Out of nowhere I got an email from Google saying a check was in the mail.<\/span><\/p>\n

Traffic increased slowly, but was very steady. \u00a0The site was the top search result for a number of highly trafficked domains so when visitors searched for \u201cemail address format at example.com,\u201d my site was the top hit.<\/span><\/p>\n

Version 2, Scaling Revenue<\/b><\/p>\n

Over the next 18 months as traffic grew, the total revenue from advertising grew as well. \u00a0While advertising was bringing in $200 or $300 a month, I was looking to increase that by an order of magnitude.<\/span><\/p>\n

My strategy was more domains, leads to more traffic, which drives my banner ad click throughs. \u00a0You don\u2019t have to be a genius to see that for a long-tail site like email-format.com<\/a>, increasing your overall footprint will bring in more traffic.<\/span><\/p>\n

Getting Big Data for a Low Price<\/b><\/p>\n

After a bit of research, I stumbled upon the Common Crawl Database<\/a>. \u00a0It was a fully accessible and searchable index of all the web pages they\u2019ve crawled. \u00a0This data set was like having my own personal Google index (without any of those fees and limits that originally sent me towards Yahoo).<\/span><\/p>\n

\"\"<\/p>\n

This database contained tens of terabytes of data. \u00a0All just sitting there for me to scan. There was a catch though – there\u2019s always a catch! \u00a0The Common Crawl index was hosted on Amazon S3 and downloading the content would cost a nickel for every GB – a potential cost of thousands of dollars. \u00a0<\/span><\/p>\n

The clear options were to either pay the network transport fee and process the data on my low cost server or to process the data directly on Amazon AWS compute instances and pay for the relatively expensive compute time but get a free ride on the network transport charges. \u00a0Given my total monthly expenses to keep the site up were less than $50, either option would quickly run into the multiple thousands of dollars range.<\/span><\/p>\n

This seemed like a big commitment. \u00a0Wasn\u2019t there the risk that I wouldn\u2019t find new domains and email addresses? \u00a0What if the traffic didn\u2019t come?<\/span><\/p>\n

It was time to take a break from the idea. \u00a0While revisiting a few weeks later I had a stroke of genius. \u00a0My plan was to process the Common Crawl web index<\/a> on a special class of AWS hosted compute servers. \u00a0This meant zero network transport charges to access terabytes of data.<\/span><\/p>\n

Instead of paying the rack rate for compute, I would take advantage of a broken marketplace Amazon ran that allowed you to bid on access to excess capacity for a discounted hourly rate. \u00a0This allowed me to get compute resources for $0.10\/hour instead of $1.50 or more. The downside is that if anyone else needed access to the resources, and they were willing to pay even an extra couple cents, then my computer would instantly terminate losing all progress.<\/span><\/p>\n

I spent a few nights building my own cluster manager that operated outside of this ephemeral \u201cspot\u201d compute instances from Amazon. \u00a0Once complete the processing flow looked like the following:<\/span><\/p>\n