Phase 01 · Day 0
Take back control of robots.txt
We switched off the plugin’s robots.txt override and put a real, site-level file back in place. It lets search engines crawl the site and only keeps them out of admin and system files.
EACase Study•Technical SEO Recovery
How we helped a business recover its Google rankings.
Exit Advisor spent two years building real search traffic. Then an SEO plugin quietly told Google to stay out, and almost every ranking disappeared. We found the cause, restored the robots.txt, rebuilt the sitemaps, and got Google crawling the site again.
- Organic traffic lost
- 274 → 3 est. monthly visits (Ahrefs)
- Keywords wiped out
- 1,595 → 1 ranking keywords (Ahrefs)
- Search clicks after the fix
- TODO: 0→000 Google Search Console
The short version
Problem
Traffic and rankings collapsed by more than 99% in four months, with no penalty and no content changes.
Cause
An SEO plugin replaced the site’s real robots.txt with its own, blocking every crawler. There was no working sitemap.
Fix
Restored a site-level robots.txt, built and submitted XML sitemaps, and requested re-indexing of key pages.
Result
Google could crawl the site again and rankings started coming back. See the numbers below.
01The Client
A business broker site Google already trusted.
Exit Advisor helps business owners sell their companies for maximum value. The team are entrepreneurs and M&A professionals who have bought, scaled, and sold their own businesses, with $500M in acquisition experience behind them.
Their website had been doing its job. Between January 2024 and August 2025, estimated organic traffic grew from about 7 visits a month to 274, and the site ranked for nearly 1,600 keywords. It held the top spots for its own category, the searches people make when they are thinking about selling a business.
Rankings Exit Advisor held at its peak
| Search | Google position |
|---|---|
| exit advisor | #1 |
| how to sell a percentage of your business | #1 |
| business exit advisor | #2 |
| exit advisors | #2 |
| business brokers software | #2 |
| business brokerage software | #3 |
| businesses to start in mexico | #4 |
| business broker crm software | #5 |
Source: Ahrefs, August 2025.
02The Problem
Why did the website’s traffic suddenly drop?
In October 2025 the traffic fell off a cliff. Estimated visits went from 260 to 48 in a single month, then to 11, then to 3. The site went from ranking for 1,402 keywords to just one.
Nothing was wrong with the content. There was no Google penalty. The pages that had been ranking for two years were still there. Google simply wasn’t allowed in anymore.
An SEO plugin took over the robots.txt
A plugin was installed to help with SEO. Instead, it served its own generated robots.txt in place of the real, site-level file. Google could no longer read the site’s actual instructions, and the file it did get effectively told every crawler to stay off the entire website.
No sitemap, no robots.txt, no way in
When we found the site, it had no working XML sitemap and no real robots.txt. Google had a locked front door and no map of the house. With nothing to crawl, it dropped pages from its index one by one until the site all but vanished from search.
How fast the rankings disappeared
| Month | Ranking keywords | Top 10 | Est. visits |
|---|---|---|---|
| Aug 2025 | 1,595 | 80 | 274 |
| Sep 2025 | 1,402 | 84 | 260 |
| Oct 2025 | 509 | 51 | 48 |
| Nov 2025 | 59 | 9 | 11 |
| Jan 2026 | 5 | 2 | 3 |
| Feb 2026 | 1 | 1 | 3 |
Source: Ahrefs Site Explorer.
03The Diagnosis
How we found out robots.txt was blocking Google.
A site-wide drop this sharp almost always has a technical cause, so we started with the question that matters most: can Google actually get in?
Checked the live robots.txt
We loaded exitadvisor.io/robots.txt directly. What came back wasn’t the site’s file. It was one generated by the plugin, and it blocked all crawlers.
Confirmed it in Google Search Console
The robots.txt report and URL Inspection showed Google couldn’t reach the pages. Key URLs came back as blocked instead of indexed.
Looked for a sitemap
There wasn’t one Google could use. Nothing was submitted in Search Console and no sitemap was referenced anywhere.
Traced it back to the plugin
The timing lined up with the plugin install, and its robots.txt setting was overriding the real file at the website level.
04The Fix
How we fixed the robots.txt and got Google crawling again.
Phase 02 · Day 0
Build proper XML sitemaps
We created a sitemap index with separate sitemaps for pages, posts, categories, tags, and archives, and listed every one of them in the robots.txt so any crawler can find them.
Phase 03 · Week 1
Tell Google the door is open
We submitted the sitemaps in Google Search Console and requested indexing for the most important pages first: the homepage, the sell-your-business pages, and the posts that had ranked best.
Phase 04 · Ongoing
Watch the recovery
We tracked crawl stats, indexed pages, and impressions, making sure Google was reading the new files and pages were coming back into the index.
# Served by the plugin, not the site
User-agent: *
Disallow: /
# No sitemapSitemap: https://www.exitadvisor.io/sitemap.xml
Sitemap: https://www.exitadvisor.io/sitemap-pages.xml
Sitemap: https://www.exitadvisor.io/sitemap-posts.xml
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-*.php
Disallow: /cgi-bin/
Allow: */wp-content/uploads/05The Results
How long did it take to recover the rankings?
We fixed the robots.txt and sitemaps in TODO: Month YYYY. Once Google could crawl again, the site started coming back into search.
- Clicks
- TODO: 0→000 Google Search, TODO: e.g. three months to Sep 28, 2026
- Impressions
- TODO: 0→00K Google Search, TODO: e.g. three months to Sep 28, 2026
- Avg. position
- TODO: 00.0 Google Search Console
- Pages indexed
- TODO: 000 back in Google’s index
06The Lesson
How to make sure an SEO plugin isn’t blocking your site.
Exit Advisor’s owner did the right thing by investing in SEO. The plugin just had a setting that quietly did the opposite. If you run a WordPress site, these five checks take ten minutes.
- Open yourdomain.com/robots.txt. If you see
Disallow: /underUser-agent: *, every search engine is blocked. - Check WordPress reading settings. Make sure “Discourage search engines from indexing this site” is unticked.
- Look for a sitemap. Try yourdomain.com/sitemap.xml and make sure it’s submitted in Google Search Console.
- Inspect a key page in Search Console. URL Inspection will tell you if it’s “Blocked by robots.txt” or excluded by a noindex tag.
- Re-check after every plugin install or update. SEO, security, and caching plugins can all rewrite robots.txt or add noindex tags.
07Common Questions
Robots.txt and ranking recovery questions.
Can an SEO plugin block Google from crawling my website?
Yes. Many WordPress SEO plugins can generate their own virtual robots.txt file. If that setting is misconfigured, the plugin can replace your real robots.txt with one that tells every crawler, including Googlebot, to stay off the whole site. That is exactly what happened to Exit Advisor.
How do I check if my robots.txt is blocking Google?
Open yourdomain.com/robots.txt in a browser. If you see “User-agent: *” followed by “Disallow: /”, every crawler is blocked. Then check the robots.txt report in Google Search Console and run URL Inspection on a key page. If it says “Blocked by robots.txt”, Google can’t read that page.
Why did my website suddenly disappear from Google?
A sudden, site-wide drop is usually technical rather than a content problem. Common causes are a robots.txt block, a stray noindex tag, a broken or missing sitemap, a failed migration, or server errors. Exit Advisor lost over 99% of its ranking keywords in about four months because Google could no longer crawl the site.
How long does it take to recover Google rankings after fixing robots.txt?
Once Google can crawl again, recovery usually starts within days to a few weeks, as Googlebot re-crawls and re-indexes pages. Submitting a sitemap and requesting indexing for your most important pages speeds that up. Full recovery depends on how long the site was blocked and how many pages dropped out of the index.
Do I need an XML sitemap for SEO?
A sitemap is not strictly required, but it gives Google a complete list of the pages you want indexed and helps it find new and updated content faster. For a site recovering from a crawl block, a clean sitemap submitted in Search Console is one of the fastest ways to get pages back in the index.
What does a technical SEO audit look for?
A technical SEO audit checks whether search engines can crawl, render, and index your site. That includes robots.txt, sitemaps, noindex and canonical tags, redirects, broken links, page speed, mobile usability, and structured data. It’s the first thing we run when a site’s traffic drops without an obvious reason.
Lost your rankings? Let’s find out why.
A sudden traffic drop is usually fixable once you know the cause. We’ll check whether Google can crawl and index your site, find what’s holding it back, and fix it.