Is Data Scraping Legal?
The question of Data Scraping has been a controversial issue for a long time. The year 2017 witnessed several important developments focused on interpretations of the Computer Fraud and Abuse Act (CFAA) in separate cases involving Craigslist and Linkedin.
What is Data Scraping? Also known as Web Scraping, it is the automated method for extracting large amounts of data from a website, often through the use of Bots.
One of the weapons used against Data Scraping is the Computer Fraud and Abuse Act (CFAA), a federal cybersecurity law enacted in 1986 as an amendment to existing computer fraud law (18 U.S.C. Sec. 1030) which had been included in the Comprehensive Crime Control Act of 1984. The CFAA prohibits accessing a computer without authorization, or in excess of authorization if any. It is designed to protect private password mainframe computers.
In Craigslist, Inc v. Instamotor, Inc, Craigslist claimed that Instamotor scraped Craigslist content to create listings on its own service and sent unsolicited emails to Craigslist users for promotional purposes to sell used cars. Craigslist has become very aggressive in pursuing claims against hackers based on breach of contract (terms of service TOS), violation of the CFAA, and the CAN-SPAM Act (disguised emails).
The case settled in favor of Craigslist for $31 million in a Stipulated Judgment and Permanent Injunction Aug. 3, 2017.
The next case involved a start up company named hiQ, a member of Linkedin (and subject to Linkedin’s TOS). What hiQ did was to scrape information from Linkedin user profiles and use the scraped data to create workforce data products that it sells to employers. Actually, hiQ tracks user generated changes to profiles in areas like work history and skills, and then uses the data to offer two products, one that helps companies identify employees who are at risk of being recruited away, and another product that helps companies map the skillsets of their employees.
The California U.S. District Court held that hiQ can use web scapers to collect information from PUBLIC Linkedin data. The Key factor in the Linkedin case was that hiQ could access and scrape only public data that was not protected by any authorization technique (such as password protected). Interestingly, the Court granted a preliminary injunction to prohibit Linkedin from employing electronic blocking techniques designed to prevent hiQ from scraping information from public linkedin profiles.
Linkedin has appealed to the 9th Circuit Court of Appeals. So we need to wait and see whether data scraping of public data is legal. Still, there is a change in the legal landscape with respect to Data Scraping. Website owners will need to examine how they control or limit access to content they collect from users.
Source: Webinar from https://www.ftcguardian.com/ – I am a charter member
In 2018 the Information Technology and Innovation Foundation (ITIF) organized a panel discussion about the challenges and opportunities to increase competition in the real estate industry through technological innovation.
Participating in the panel were lawyers from NAR, the FTC, and the DOJ, as well as Brian Larson counsel for CMLS. The only non-lawyer in the group was Ben Clark, former broker for eRealty and noted data expert in the real estate industry. It is interesting to hear the broker’s perspective, so we will highlight some of those discussions.
Video of the 90 minute session is available here.
One of the key points from Ben Clark was the need for MLSs to remove any barriers to MLS participants and subscribers receiving access to relevant MLS listing data. This is based on the fact that real estate professionals have the following fiduciary duty under the NAR Code of Ethics Standard of Practice 11-1, “When REALTORS® prepare opinions of real property value or price they must
- be knowledgeable about the type of property being valued;
- have access to the information and resources necessary to formulate an accurate opinion, and
- be familiar with the area where the subject property is located UNLESS lack of any of these is disclosed to the party requesting the opinion in advance.”
You can find some of Ben Clark’s remarks at the following times of the video:
- 5:55 to 8:08;
- 35:40 to 39:50;
- 46:30 to 49:42;
- 1:03:40 to 1:06:50.
There is also a discussion on MLS Copyright at 1:11:25 to 1:14:05, with a supplemental blog post by Mitch Skinner Click Here
Finally, for an excellent summary of the history of MLS from Brian Larson, go to the video at 27:00 to 32:30
This November marks the expiry date of the Consent Decree in the NAR/DOJ lawsuit 10 years ago. Mark your calendar for June 5, 2018 when the FTC and DOJ will hold a relevant joint Workshop to discuss “What’s New in Residential Real Estate Brokerage Competition” – Click Here
Here is the entire panel discussion from the June 5, 2018 Workshop:
Several recent articles have focused on the increase use of the new Top-level Domains by brands, referred to as Dot Brands.
Tony Kirsch, head of professional services at Neustar, in an article discussing the progress of Dot Brands, “2016: The Year That Was in .brands TLDs”, wrote:
“While some astonishing progress has been made in 2016, the .brand movement is by no means complete. I personally am thrilled to be heading into 2017 with such momentum behind us and I’m optimistic about the continued growth of the .brands space in the New Year.
Already this year we’ve been surprised by the creative approaches many brands have taken and we’re looking forward to seeing which companies will use 2017 to take .brands to places we haven’t considered before.
Perhaps 2017 will be the year in which mainstream advertising strategies adopt .brands as a unique, memorable call-to-action and the marketers of the world find significant value in creating more direct, meaningful customer relationships through smarter digital marketing.
With almost all .brands now added to the Internet, the game is on. Which brands will emerge as frontrunners and truly revolutionize the way we look at digital?
I don’t know the answer, but I sure am excited to find out.”
Some major launches noted were:
Google – consolidating news and updates from over 19 of Google’s corporate and product blogs, ‘The Keyword’ was launched at www.blog.google in October, joining the internet giant’s retail domain name sales play at www.domains.google
Canon – one of a handful of ‘full transitions’ to .brands, the company’s global site is now at www.global.canon , with the existing canon.com domain redirecting to this awesome new site.
As well as: Dell, Deloitte, Chanel, Microsoft, Audi, Cisco, Fage, Orange, Komatsu, Shell, Mini, Lidl, Linde, Bradesco, Hotmail, NTT, UOL, Pictet, Windows, Bloomberg, Sky, Fox, Emerck, Weir and BNP Paribas are just a handful of the other organizations with live .brand websites. http://www.circleid.com/posts/20161219_2016_the_year_that_was_in_brands_tlds/
According to another article “Brand Top-Level Domain and Brand Value” by Guiillaume Pahud,
“Brands are an asset and have a value for organizations, as they generate revenue: Customers are happy to pay a premium for a brand they love. They will show a preference and be loyal to a brand they trust.
Some brands have decided, in 2012, that their name should actually also become the global name for all of the digital assets, and registered their brand name as a Top-Level Domain. Google, for instance, located their new blog platform, where they post news and information, on the domain blog.google. That is a typical example of a branded top-level domain.”
Realtors can acquire their own Dot Realtor TLD by going online to www.claim.realtor — the first year is free. If ordering multiple domains or years, use the word discount as the promo code for a 10% saving.
Is a Bot in Your Future?
For sure in 2017, real estate professionals will be hearing more about the use of Bots, Artificial Intelligence (AI), and machine learning as productivity tools in their practice. Already we are seeing more and more articles and online discussions about Bots.
To give you an idea of what’s to come in this exciting space, here are some highlights from several articles about a San Diego company led by real estate veteran Grant Gould that is making news with the anticipated January 2017 launch of its Automabots service – www.Automabots.com
Real estate professionals, REALTOR® Associations and MLSs will want to take note that initial use of Automabots will be directed at the real estate industry. As mentioned below, AI manifestations can be powerful tools for business efficiency, so now is the time to get informed and use them to your advantage.
- Inman’s Real Estate Technology Expert, Craig Rowe, recently reviewed Automabots, –
Craig noted that Grant Gould is developing several bots with different purposes in the agent realm one at a time, including BuyerAgentBot, SellerAgentBot (in beta) along with a CRMBot, DigitalMarketingBot and TransactionBot that will eventually form a real estate suite.
During the initial interaction, users indicate their preferred ZIP codes, price, number of beds and baths, and communities they’re interested in to populate a relevant list of homes, which are pulled directly from the MLS, current to the last hour.
After users finish looking at the homes provided, the Automabot asks how often they would like to receive updates on available listings. Gould notes that users’ response to this question serves as a lead qualifier — someone who wants an update every two hours (or daily) is probably more motivated than the casual monthly browser. Today, that entire lead cultivation process is being replaced by text apps and bots. But agents shouldn’t see bots as a threat — the element of data control has been slipping for some time from the industry’s hands, and Kubrick’s eerie vision (of HAL-2000) isn’t quite so dark in real life. Rather, these AI manifestations can be powerful tools for business efficiency, so get informed and use them to your advantage.
- Adrian Bishop, Editor for OPP.Today authored a Business Technology Editorial entitled “How Bots Can Turn Leads Into Clients And Boost Sales”
– http://www.opp.today/how-bots-can-turn-leads-into-clients-and-boost-sales/ – In his editorial, based on an interview with Automabots founder Grant Gould, Mr. Bishop wrote:
Essentially, bots are a small application that live in the cloud and utilize the machine learning technology recently made available by the tech giants. These little programs, now referred to as bots, can be taught to perform nearly any digital task that normally would be done by a human being. People can communicate with bots via chat platforms, like; Facebook Messenger, Alexa and the dozens of others.
“We teach real estate bots to do the work, so you can get more done in less time at a lower cost, “says Automabots.
Using natural language technology, your personal bot interacts with prospects to determine their exact real estate needs and delivers the information they desire without having to log in or download an app.
When Grant Gould was asked how does his service work, he responded: “Similar to Siri, Amazon Alexa, and IMB’s Watson, Automabots uses machine learning that is a combination of rule-based and natural language parsing.
“We are teaching the Bot real estate lingo, terminology and the nuances of buying and selling properties. The Lead Capture Bot can engage online users through chat applications, like Facebook Messenger, to assess the consumer needs and then search and display the information right in the chat session. MLS properties, market stats, and home values are just some of the info it can deliver.
“It then asks the user if they would like to get alerts and updates on this information, which results in capturing a lead. As it continues to learn what the user likes and dislikes, it gets smart in finding matching homes.”
Bots save both time and money and boost sales, says the company. “Get a highly skilled Lead Qualifier Bot for just 14 cents an hour. No benefits, sick days, or employer cost. You get a ready-to-work real estate Bot that is specially trained to do a specific job and do it great. And these workers get smarter and more efficient over time.
“Real estate bots can be taught to perform many tasks currently done by humans. We have trained our Bots to greet every person immediately, qualify their buyer and seller needs, and deliver the information they want without using any human resources.
“Having a dependable Lead Qualifier Bot on duty 24/7 to greet buyers and sellers on your website, social profiles and other lead sources will improve your lead conversion rate by as much as 300%. Add a great lead nurturing system and watch business soar.”
- In an article in VisionAR entitled “Productivity AI Bots for Real Estate Agents” –
http://visionar.today/index.php/2016/11/22/automabots-makes-productivity-bots-real-estate-agents/ – the author Lauren Twardy interviewed Grant Gould and stated:
“Though Gould expects the Bots’ utility will extend into other industries before long, Automabots is currently focused solely on real estate. Gould is a real estate veteran and understands the kind of help agents need to be productive. Gould likes to call his line of product “Productivity Bots,” which are designed to enhance rather than take over business. He explains, “consumers today want instant gratification,” and his Bots help Realtors provide that gratification by remaining on active duty 24/7. While Realtors can get thousands of visitors to their websites, social media profiles, and other lead sources, the Bots are trained to convert visitors into leads.
Gould’s “Productivity Bots” are productive. The Lead Qualifier Bot is designed to save real estate agents time and money; for just 14 cents an hour, “you get a ready-to-work real estate Bot that is trained to do a specific job and do it great.” The Automabots website explains the basic functions of the Bot:
“Greets Every Prospect.” It is designed to engage potential customers 24/7 on real estate websites, social media, and other listing sites. This provides instant gratification for consumers, which they don’t receive with website contact forms or email.
“Qualifies Buyer and Seller Needs.” The Bot can interact with customers to determine what they’re looking for and can dole out information from the massive amount of stored data.
“Captures Leads and Intel.” Not only can it record information about prospects, but the Bot can also transfer information directly to the Realtor or to an automatic lead nurturing program.
Of course, the functions of this AI are meant to grow beyond these basics. Automabots “build intelligent Bots that [they] teach to perform important tasks and that gets smarter as it does its job.”
2017 could very well be the year of the Bot in real estate. Stay tuned and explore ways the services of a Bot can help you be more efficient and productive.
Dot WEB (.WEB) sold at auction for $135 million on July 29
First, it was the new Top-level Domain (TLD) .APP that was acquired at auction for a $25 million bid by Google; recently it was .SHOP that was won with a $41.5 million bid from a Japanese company.
And now, the contentious battle among 7 companies who applied to operate the .WEB suffix has ended in a winning bid of $135 million, with speculation that the winning bid was backed by .COM operator Verisign. For that kind of money and contention, the domain market indicates that .WEB could be even bigger than .COM
So, who gets the $135 million? It goes to ICANN, who has now banked over $235 million as a result of ICANN’s last resort auction mechanism that is used when several parties applied for the same suffix but were unable to resolve this contention by themselves.
And what will ICANN do with all this auction money? Good question! See http://bit.ly/2apFOlM
In February of 2015, the Internet community was shocked to find out that someone paid $25,001,000 in an auction to acquire the new Top-level Domain (TLD) for .APP (Dot APP). That someone, of course, was none other than Google. And just last month, on January 27, 2016, the new TLD for .SHOP (Dot Shop), was sold at auction for a whopping $41,501,000. The winning bidder was GMO Domain International, Inc from Tokyo. They outlasted Amazon and Google in a bidding war that went 14 rounds, the final round going from $36,800,000 to $46,000,000.
We are talking some serious money here in the ecommerce space, a space in which there are now over 12 million new TLDs.
How do we even get to the auction stage? First, for anyone to apply to operate a new TLD, there is a $185,000 application fee. In the event more than one applicant applies for the same domain (called a ‘string’), those competing entities move to a string contention status under ICANN rules. ICANN is a not-for-profit public-benefit corporation with participants from all over the world dedicated to keeping the Internet secure, stable and interoperable.
According to ICANN, “Auctions are the mechanism of last resort to resolve string contention within the New gTLD Program. ICANN expects that most string contention will be resolved through other means before reaching an Auction conducted by ICANN’s authorized Auction service provider, Power Auctions LLC. However, there is a possibility that significant funding will accrue as a result of several Auctions. Auction proceeds will be reserved and earmarked until the Board determines a plan for the appropriate use of the funds through consultation with the community. Auction proceeds are net of any Auction costs. Auction costs may include initial set-up costs, auction management fees, and escrow fees.”
212 of the 233 contention sets are now resolved, the majority having self-resolved, but a few were resolved via ICANN’s auction process (“the method of last resort”). In some cases, contending applicants formed joint ventures in their efforts to resolve string contention. In other cases, some applicants withdrew after receiving some compensation from the surviving applicant — hopefully in an amount to at least compensate the withdrawing party for their $185,000 initial application fee.
In one case of a contention over the new TLD .coupon (Dot coupon), the battle was between Donuts, Inc and the Coupon.com company. Donuts prevailed. According to an SEC filing, Coupons.com received $4.8 million dollars, apparently based on a privately negotiated agreement. As one pundit noted, it only cost Coupons.com $185,000 to file its application for .Coupons TLD, making this the best coupon the company ever redeemed.
And what about the $41,501,000 winning bid for .SHOP – who got that money? Well, ICANN of course. In fact, in the 15 auctions that have concluded since June 2014, ICANN has amassed $101,357,812 in net proceeds. Some of the top auction winning bids besides Google’s .APP, were .TECH for $6,706,000, .REALTY for $5,588,888, and .MLS for $3,359,000 by CREA. https://newgtlds.icann.org/en/applicants/auctions/proceeds
According to ICANN, “Possible uses of auction funds include formation of a foundation with a clear mission and a transparent way to allocate funds to projects that are of interest to the greater Internet community, such as grants to support new gTLD applications or registry operators from communities in subsequent gTLD rounds, the creation of an ICANN-administered/community-based fund for specific projects for the benefit of the Internet community, the creation of a registry continuity fund for the protection of registrants (ensuring that funds would be in place to support the operation of a gTLD registry until a successor could be found), or establishment of a security fund to expand use of secure protocols, conduct research, and support standards development organizations in accordance with ICANN’s security and stability mission.”
With more than 1500 TLD applications submitted to ICANN in the first round, there are sure to be companies that do not turn their TLDs into successful businesses. But with such a large amount of investment money at stake, we can expect to see a lot of ecommerce activity built around many of the new TLD ventures.
New Airbnb disclosure law in California
In a matter of a few years, Airbnb has developed a global presence that has attracted many enthusiastic supporters as well as numerous vocal critics, especially those living in the neighborhood. Interestingly, during Airbnb initial attempts to gain venture capital funding, it seems that Brian Chesky, co-founder, took his idea to seven venture capital firms – five threw him out the door after 15 minutes and the other two didn’t even return his phone call. After all, why would anyone give the keys to their house to a stranger who might be a serial killer? And today, that goofball idea has a valuation of $10 Billion.
Airbnb has also attracted the attention of government regulators who have attempted to assert some controls over what is called “hosting platforms.” One law recently enacted in California is of special interest to landlords and property managers. The law — Business and Professions Code Sec. 22590, 22592, and 22594, effective January 1, 2016 — requires a “hosting platform” to warn a tenant that subletting the tenant’s residence may violate his/her lease and could result in eviction.
A “hosting platform” is a marketplace that is created to facilitate the rental of a residential unit offered for tourist or transient use for compensation to the offeror of that unit, and the operator of the hosting platform derives revenues, including booking fees or advertising revenues, from providing or maintaining that marketplace. Airbnb is an example of such a platform.
This law requires a “hosting platform” to provide notice to an occupant listing a residence for short-term rental that states:
“if you are a tenant who is listing a room, home, condominium, or apartment, please refer to your rental contract or lease, or contact your landlord, prior to listing the property to determine whether your lease or contract contains restrictions that would limit your ability to list your room, home, condominium, or apartment. Listing your room, home, condominium, or apartment may be a violation of your lease or contract, and could result in legal action against you by your landlord, including possible eviction.”
The notice must be in a particular font size and be provided immediately before the occupant lists each real property on the hosting platform’s Internet Web site in a manner that requires the occupant to interact with the hosting platform’s Internet Web site to affirmatively acknowledge he or she has read the notice.
Well, that’s the latest from California. It’s a disclosure law, one that probably doesn’t have compliance or enforcement teeth.
So, what’s the status of these Airbnb type short-term rentals in your neighborhood? Are they drawing the attention of local regulators?
Undoubtedly, the most popular Top-level domain (TLD) is the dot com domain (.com) that you find as the extension in most web addresses. Dot com has served the Internet well for the past 30 years, mostly because it was simply available, and could be used for most any purpose. Looking at a dot com address, though, you couldn’t be sure if you were going to arrive at a business, a blog, a school, a church, a whatever.
Until recently, many businesses looking for a web presence discovered that the inventory of meaningful domain names had been heavily exhausted and they had to settle for web addresses containing hyphens, multiple words, awkward abbreviations and numbers. Fast forward to the past two years and we find there are now hundreds of available Top-level domains. We have entered an age of complete words being used as web addresses.
So, you ask: Where are these new top-level domains? Is anyone actually using them? The answer is yes, and you will start seeing more of the TLDs being activated in 2016. It has taken several years for the momentum to build, but that was also true back in the early years for dot com.
As of the close of 2015, Internet users had registered more than 11 million new domain names (what some domain companies sarcastically refer to as not-com domains). Donut, Inc, one of the largest TLD registries, said that “The second year of not-com availability has been marked by promising growth in consumer awareness and adoption. As the new domain registrations surge past 10 million, a look at the growth rates reveals a remarkable trend. Since January 2015, total new domain registrations have grown by 263 percent and the average monthly registration rate has more than tripled. Wall Street analysts forecast that not-com registrations could top 20 million by 2016.”
Here’s a random selection of the kinds of complete word domains you’ll be seeing more of in 2016. Please note that many businesses will be building marketing campaigns focused on the use of keywords both to the left and to the right of the dot, and they will be doing this while keeping active any successful dot com web address they have maintained over the years. After careful consideration, one major brand, Barclays, did decide to transition out of their well-established dot com web address and move entirely to their new Barclays TLD address.
https://www.lily.camera/ – check out the video 1.5 min – it is so cool
https://manhattan.dental/ — notice the email marketing punch they get with Smile@Manhattan.Dental
http://bluecarp.tax – Bluecarp Tax Consulting assist U.S. and Canadian individuals and entities dealing with their U.S. tax issues.
You can add to the list the two TLDs that NAR was able to acquire last year – the .REALTOR and the .REALESTATE top –levels. While most of the above examples are open for anyone to obtain, NAR’s TLDs are only available to REALTORS®. They are what the registration authority (ICANN) calls Restricted TLDs. For more information, go to www.About.realtor
As you look at your marketing plans for 2016, be sure to spend time evaluating your Domain Strategy. Let us know if we can help –
John and Saul – www.BetterCallSaul.realtor mailto:Saul@BetterCallSaul.realtor