Why Google search sucks these days, through the lens of evolutionary biology
Foragin’ Friday
Foragin’ Friday
Hi there, and welcome to Engineering Our Social Vehicles— I’m your host, Paul Logan! Today is Foragin’ Friday. If you’re new to the blog, that means we spend Fridays talking about searching for things. Today, we’re going to talk about Google Search.
Innovation & Expiry
Google search is a filter. It is intended to separate the wheat of relevant information from the chaff of internet spam, hearsay, and advertising. If you’ve been using Google long enough, you’ve no doubt noticed that it does this job much worse than it did 10 years ago. This article is the first in a series about expiry of societal filters. In it, I’m going to discuss several theories from the field of evolutionary biology, and apply their lessons to the decline of Google search.
Things age. Things decline. Certain innovations in history represent saltatory restructurings of society, and spur massive improvements in quality of life over short periods of time.1 These revolutions exist in every category (science, technology, politics), and Google search was one of them: it shepherded the internet from a series of muddy backwater roads into the superhighway it is today.
However, the world eventually catches up to the rapid change, the benefit of the innovation effectively “expires.” A reduction in quality of life follows; necessitating further revolution. For this reason the pattern of human social progress mirrors the evolutionary theory of punctuated equilibrium.
The Red Queen: Search Engine Optimization
Are you familiar with the Red Queen Hypothesis? It’s a description of parasitism as a biological arms race. Host and parasite are in constant adaptation to outsmart one another. The result is global equilibrium, despite massive local energy expenditure.2
If you’re unfamiliar, Search Engine Optimization (SEO) is the industry that has grown up around manipulating search engines for financial gain. The “optimization” title is euphemistic. Since the majority of traffic on the internet is directed by search engines like Google, there’s a lot of money to be made by putting a thumb on the scale to appear higher in the rankings and grab a larger slice of traffic.
Google and other search providers OK some SEO strategies, making them “white-hat,” and punish/disallow others, making them “black-hat”. Despite the division, all SEO techniques boil down to reverse engineering the algorithm used for search, and then attempting to game it in some way. A few examples of these techniques:
White Hat:
Meta tagging— adding additional hidden information to sites that can be used to “hydrate” search results with images, categories, and description text.
Crawlability— formatting sites to make them consumable to the web-crawlers that feed information to search algorithms.
Content Marketing— producing in-depth content tied to your brand in order to draw traffic and buoy sales with non-advertising search entries.
Black Hat:
Content Farming— producing mass amounts of low-quality content geared towards capturing traffic from search engines by leveraging keywords.
Click Farming— employing large amounts of people to visit and sign up to sites, spoofing organic traffic and boosting search rankings
Link Farming— creating a network of interlinked sites to create the illusion of a greater depth of discussion or interest than really exists.
Google is essential web infrastructure— the interstate highway to the internet’s road system. This infrastructure is so vital that some have argued it shouldn’t be held privately, or that it should be held to a standard of neutrality. These argument reiterate the common argument against all societal filters: there exists bias, and not everyone can benefit.
As a for profit entity, Google is locked in an eternal struggle to preserve its own bias (and thus potential for profit) against the bias introduced by SEO. With an industry market cap some estimate at ~50 billion, there is a lot riding on the degree to which Google and other search engines can be biased. Multiple suits against Google by SEO firms have already seen court.
Over the years, Google has repeatedly reworked its search algorithms to stay ahead of SEO. It’s a game of cat and mouse, and the two will never be rid of one another. The situation calls to mind the Alice In Wonderland scene for which the Red Queen is named:
"Now, here, you see, it takes all the running you can do, to keep in the same place." — Lewis Carroll
Of late, AI generated sites have begun to overwhelm the search giant in its battle with SEO. However, SEO isn’t the solely to blame for ruining Google search— self interest and the profit motive take their toll as well.
The Red King: Monopoly & Corporate Bloat
The Red Queen has a lesser-known twin, born of the Mutualism-Parasitism Continuum: The Red King Hypothesis. The Red King is brother and complement to the Red Queen, and suggests that slow coevolution between mutualistic symbionts allows them to eke out fitness and superiority over time.
The Red King holds court with an estranged cousin, the Symbiogenesis Speciation Hypothesis (SSH). Both suggest mutualism and symbiosis as primary drivers of evolution. SSH, however, suggests explosive evolutionary events via breakthrough-symbiotic events like mitochondrial capture, which stands in contrast to the Red King’s slow change.
As Google has grown into a global monopoly, it has consumed and created hundreds of smaller services, from YouTube to Maps to Shopping. There’s money to be made in synergy, and redirecting users back to Google properties keeps valuable traffic and data “in the family.” Internally developed products like Maps represent the gradual mutualistic incorporation of the Red King. Externally developed and quickly subsumed products like YouTube are closer to SSH. Both represent mutualism driving change.
As the company has progressed from the singular focus of search, it has refined the way it makes money. The focus on profit is reflected in search results themselves over time. Here’s a comparison of results over the years:
It’s not only bloat from internal services, but ads as well. 20 years ago, Google highlighted inline ads, nowadays, they are made to blend in:
As incentive to keep traffic in-house has grown, the quality of Google search has waned. Google now often functions as a directory of services that make the search provider money, rather than an integral piece of infrastructure correctly routing traffic to relevant sites.
This is mutualism through and through: Google is tripping over itself to favor its own internal business units. When combined with the SEO issues we discussed before, we are trudging uphill against the bias injected into search both by external actors and the search provider itself.
The Court Jester: Mobile Phones
The Court Jester Hypothesis is a foil to the Red Queen that posits abiotic environmental changes are the major driving force behind evolutionary change, rather than interspecies competition.
The move from majority desktop to majority mobile user agents changed the game vis-a-vis what an optimal return is from a search service. If someone is sitting at a desk, they’re likely more willing to do in depth reading or research. If someone is looking at a mobile phone, they likely want a quick answer. Smaller screens also contribute to the sense that one has to scroll further to see relevant results.
The Google we have today may be different, but that doesn’t mean it’s necessarily worse— just optimized to a different environment than before. The many services like Google Plus that have been introduced and then dismissed over time have further altered that environment and caused the service to slip.
The Black Queen: Loss of functionality
Some communities of micro-organisms actually simplify their genome over time. The Black Queen Hypothesis attempts to explain this gene loss via symbiosis. The basic gist is that in communities, individual species of micro-organisms become specialized in production of vital resources, allowing other species to drop the coding for production of those resources from their genome. This increases replication/reproduction speed and drives loss of genes over time. Both the Red King and SSH support the Black Queen.
As Google has grown, it has also stripped many of the features that once made it a powerful search platform. It has killed or stopped supporting upwards of 10 search operators over time:
+— Force an exact-match search on a single word or phrase.3 (Deprecated 2011)
~— Include synonyms. Doesn’t work, because Google now includes synonyms by default. (Deprecated 2013)
inpostauthor:— Find blog posts written by a specific author. This only worked in Google Blog search, not regular Google search. (Deprecated 2011)
allinpostauthor:— Similar to “inpostauthor,” but removes the need for quotes (if you want to search for a specific author, including surname.) (Deprecated 2011)
inposttitle:— Find blog posts with specific words in the title. No longer works, as this operator was unique to the discontinued Google blog search. (Deprecated 2011)
link:— Find pages linking to a specific domain or URL. Google killed this operator in 2017, but it does still show some results—they likely aren’t particularly accurate though. (Deprecated in 2017)
info:— Find information about a specific page, including the most recent cache, similar pages, etc. (Deprecated in 2017).4
daterange:— Find results from a certain date range. Uses the Julian date format, for some reason. (Not officially deprecated, but doesn’t seem to work.)
phonebook:— Find someone’s phone number. (Deprecated in 2010)
#— Searches #hashtags. Introduced for Google+; (Deprecated 2019)
After monopoly is achieved, there’s incentive to remove features and bring them back as paid, or simply leave them dead if they stand in the way of profit. 2023 should usher in changes to certain Google APIs that make full adblock possible only for paid enterprise users.
If you have trouble buying the idea that Google might intentionally deprecate functionality for profit, I have good news! There exists an obscure fourth monarch who agrees with you: the Grey Queen.
The Grey Queen: Subfunctionalization
The Grey Queen is the figurehead of Constructive Neutral Evolution (CNE); an approach to evolutionary biology that attempts to explain change without the need for it to be “driven” by anything more than neutral random error.
An example of this is “subfunctionalization,” which is essentially the process I’ve described above for delegation of vital genetic processes across the genome. Instead of taking the adaptationist angle that the division of labor speeds reproduction, the Grey Queen instead proposes the division of labor as a natural consequence of genetic drift. To illustrate:
You start with some vital function A, coded for by gene A1. Function A produces materials B and C, then combines them to make vital nutrient D.
Then, a gene duplication event happens, and A1 now exists in two places in the genome; A1 and A2.
Now, mutation occurs and causes errors in A1, such that it is no longer able to produce material C. Now A1 can’t produce vital nutrient D. This doesn’t cause any disruption to the overall system, since gene A2 still produces D. A1 has now become B1.
This process repeats with A2, which loses the coding to produce B, and becomes C1.
The host organism can’t survive without vital nutrient D. Any mutation that loses the ability to produce either subcomponent B or C is weeded out via negative selection.
At the end of the process of subfunctionalization, vital function A is maintained, and the genome is still able to produce vital nutrient D. The difference is that now instead of the single gene A1, we have two genes B1 and C1 that perform the same function through a division of labor.
No step in this process was adaptive or advantageous to the host organism. It is a series of random events that establish path dependence in such a way that there is low likelihood of undoing random drift. CNE’s wikipedia article says it well:
While each of these steps are individually reversible (for example, A may regain the capacity to function independently or the A:B interaction may be lost), a random sequence of mutations tends to further reduce the capacity of A to function independently and a random walk through the dependency space may very well result in a configuration in which a return to functional independence of A is far too unlikely to occur, making CNE a one-directional or "ratchet-like" process
You might think of the Grey Queen as a “runaway bureaucracy,” where increasing complexity of systems is driven by random drift, not adaptation.
Nowadays, many append “reddit” onto their Google search to get actual information and sidestep the issues discussed above. This functions in much the same way as subfunctionalization: Google has sufficient search and deficient information, Reddit has deficient search and sufficient information. Separately they aren’t game-changing, but as parts of a complex whole they yield the same functionality Google once provided on its own.
This arrangement did not arise through consultation between the two services. It is the end result of a series of random events that have aligned through time and circumstance to make odd bedfellows. It’s very difficult for either company independently to reverse the situation without shooting themselves in the foot. As such, the arrangement persists due to the complexity of systems, not due to incentive or advantage derived by either party.
Conclusion
Over the years Google search has gotten worse due to a variety source from without and within. The point I want to make here, and in the forthcoming article, is that all things expire. Google was a great product when it first came out. After almost 30 years of entropy, it’s time to let go, and search for the next innovation that will improve quality of life.
Be sure to subscribe so you can read Pt:II, which will deal with the idea that any societal filter you can pay to bypass is inherently broken, and bring back the idea of forced expiry we explored in Surrogation Nation. Until then, leave a comment or tweet at me and let me know what you thought of this piece!
Oftentimes these are technological innovations that reduce significance of space(radio, airplanes, locomotives), or labor (the cotton gin, the printing press, the wheel); OR political innovations which redistribute control of resources and productive capacity among classes.
If you’re familiar with Scott Alexander’s Moloch as a personification of the race to the bottom, the Red Queen is Moloch incarnate, but for evolution.
You can do the same thing by using double quotes around your search.
The id: operator can also be used—the results are identical. Although the original functionality of this operator is deprecated, it is still useful for finding the canonical, indexed version of a URL.
I have a theory like this for Amazon: Because Amazon also sells advertising via "sponsored" or "featured" or "Amazon's Choice" listings, there is a disincentive to actually help you quickly find exactly what you want. The incentive is you feed you exactly what you want IF it's got a sponsored product, IF not, then show you similar sponsored products, and finally show you what you want after you've waded through sponsored products. Another may be the longer it takes you to find what you actually want, the more sponsored products you see and perhaps click on out of curiosity.
It's like going to a car dealership and having the salesman show you a bunch of cars you didn't ask about (but have higher margins or have been sitting on the lot longer) before finally getting to the car you actually asked for.
what a fantastic article. thank you.