GEO You Don't Know: Principles, Practices, and Trade-offs of AI Visibility
Spend an Hour Making AI Find Your Content
In the past few days, several friends @mentioned me, saying my open-source tools were actively recommended when they asked AI. Without doing anything, they got indexed. I thought maybe I should spend an hour structuring the content properly, which would be even better. So after finishing, I quickly posted a quick note on Twitter, but the structure was unclear. Since everyone seemed interested, I decided to write a well-structured article for easy reference and searching.
I hate chasing rankings or producing junk content. I'd rather make existing content more visible to AI. So this article won't teach you shortcuts, but how to help AI better understand your existing content itself.
I looked into it and found that AI search logic is completely different from traditional search. Traditional SEO is about getting into Google's top 10, but 83% of AI Overview citations come from pages outside the top 10. AI cares about clear structure and reliable sources, not much about PageRank. The project itself may be small, but the README and documentation are fairly clear. When large sites have thin content, AI can find me—that's probably why friends could find Pake and MiaoYan.
AI search is growing fast, up 527% year-over-year in the first half of 2025. ChatGPT reached 900 million weekly active users by February 2026, and referral conversion rates are about 5 times higher than traditional search. However, it still accounts for less than 1% of total referral traffic. It's more of a brand visibility strategy than a traffic strategy. Worth spending an hour on, but not a week—because your product itself is your core competency, not this.
Use robots.txt to Distinguish Crawler Types
Many people treat robots.txt as a switch—either block all AI crawlers or allow everything. But AI crawlers actually fall into several categories, doing different things.
My approach is to allow search/retrieval crawlers and user-triggered crawlers, block training crawlers and undeclared crawlers:
markdown# Search & retrieval: allow User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # User-triggered: allow User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / # Training: block User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / # Opt-out tokens User-agent: Google-Extended Disallow: / # Undeclared: block User-agent: Bytespider Disallow: /
Write a Good llms.txt and Cross-Reference Sites
llms.txt is a new standard, similar to robots.txt but specifically for AI. Place a Markdown file in your site's root directory, clearly stating what your site does, key pages, and authors. When AI retrieves content, it reads this file first to understand your site.
BuiltWith tracks over 840,000 websites that have deployed llms.txt, including Anthropic, Cloudflare, Stripe, and Vercel. However, adoption is only 10% among the 300,000 domains surveyed by SE Ranking. It's still early, so early adopters have a first-mover advantage.
The format is simple:
markdown# Your Project Name > One-line description of what this is. ## Links - [Documentation](https://yoursite.com/docs) - [GitHub](https://github.com/you/project) - [Blog](https://yoursite.com/blog) ## About Short paragraph explaining the project, its purpose, key features, and what makes it different.
After finishing, you can submit to directory.llmstxt.cloud, llmstxt.site, and the llms-txt-hub repository on GitHub via PR.
I also did something interesting: cross-reference llms.txt files across my sites to form a mesh structure. I maintain several sites: tw93.fun, weekly.tw93.fun, yobi.tw93.fun. Each site's llms.txt references the others, so no matter which entry point AI uses, it can follow links to find other content.
These changes take effect only after crawlers re-fetch, usually a few days. After configuration, periodically search for your project name on ChatGPT; the citation sources and description accuracy should improve.
To tell AI you have a Markdown version, the simplest way is to add this line in your page's
:text<link rel="alternate" type="text/markdown" href="/page.md" />
Claude Code and Cursor already send an Accept: text/markdown header when fetching documentation—this is standard HTTP/1.1 behavior from 1997.
Submit Your Site to Search Platforms
The robots.txt and llms.txt mentioned earlier help AI understand your content, but only if AI can find you. ChatGPT Search uses Bing, Google AI Overview uses Google's own index, and Perplexity relies on search APIs. If your pages aren't indexed by search engines, the structured work you do is invisible to AI. So the first step is to ensure Google and Bing have indexed your site.
Simple steps: Go to Google Search Console and verify your domain using DNS or an HTML file. After verification, submit your sitemap URL (typically yoursite.com/sitemap.xml). In the "Pages" report, you can see which pages are indexed and which have issues. If an important page isn't indexed, use the "URL Inspection" tool to manually request indexing.
You might think Bing is rarely used, but Copilot, DuckDuckGo, and Yahoo's AI search are all powered by Bing on the backend. Register on Bing Webmaster Tools, submit your Sitemap—it has an AI Performance panel showing how many times your content has been cited by AI. Also set up IndexNow to proactively notify Bing when new content is published, without waiting for crawlers.
IndexNow integration: place an API key file in your site's root directory, then send a POST request to api.indexnow.org/indexnow with the list of changed URLs. Bing will crawl within minutes. Many static site generators and CMSes have IndexNow plugins ready to use.
Google Search Console currently has no AI-specific panel, but submitting your Sitemap and monitoring index status is still worthwhile. Google AI Overview pulls content from a broader range than traditional results, so even if your page isn't in the top 10, it may appear in AI answers.
Perplexity has more overseas users than you might think. They have a publisher program—go to pplx.ai/publisher-program and submit a form. If approved, you get an 80/20 revenue share and can view citation analytics.
I Built a Knowledge Page Specifically for AI
Instead of waiting for AI to gather fragmented information from various sites, give it a centralized entry point with everything you want it to remember.
A knowledge page should provide three layers: overview (llms.txt), full version (llms-full.txt, 30-60KB), and individual knowledge pages for each core project. Plus, structured JSON APIs for AI tools to programmatically fetch data. Don't hardcode data; pull it in real-time from upstream sources like GitHub API, with caching and periodic refresh for minimal maintenance.
One easily overlooked point: give AI a narrative structure, not just a scattered list of projects. If you have multiple projects, write a paragraph connecting them—their relationships, your technical direction, overall positioning. When AI answers "who is this person" or "what does this team do," a narrative is far more effective than a list.
My implementation is called Yobi (from Japanese 呼び / よび, meaning "call" or "summon"), providing an llms.txt overview, a 50KB llms-full.txt, individual project pages, and four JSON endpoints: /api/profile, /api/projects, /api/blog, /api/weekly. Data is fetched in real-time from GitHub API, with ISR cache refreshing every hour. Tech stack: Next.js + TypeScript, deployed on Vercel.
Structured data returned by JSON API, including project info and real-time GitHub stats:
Give Each Project Its Own Page
Each project needs its own dedicated page—not just a line in a list, but a self-contained Markdown document with a citable summary, core features, competitor comparison, use cases, and installation commands. Ahrefs research found that cited pages have higher semantic similarity between page titles and user queries. Natural language URL slugs (e.g., /projects/pake) also have higher citation rates than opaque IDs (e.g., /page?id=47).
URL structure matters. /projects/pake tells the model what the page is about before it reads a single line, while /page?id=47 says nothing.
Sync Structured Data to Your Main Domain
Subdomains carry less weight than root domains. If an AI crawler finds example.com, it won't automatically go looking for docs.example.com or api.example.com. If your llms.txt, project pages, and API data are scattered across multiple subdomains, AI may only see part of it.
Solution: mirror key structured data to your main domain. Have example.com/llms.txt, example.com/projects/xxx.md, and example.com/api/projects.json all under the same domain. AI crawlers discover your main site via search index, then fetch all data within the same domain. Implementation can be CI sync, build-time fetching, or reverse proxy—choose what fits your deployment. I use a GitHub Action that syncs sub-site data to the blog repository every night.
When launching a new site, configure items on a checklist to avoid omissions. Core items: robots.txt (classify crawlers), llms.txt (write site summary and cross-reference), sitemap (submit to search engines), Bing Webmaster Tools (enable IndexNow), Google Search Console (monitor index status). Each site's llms.txt references others, forming a mesh discovery structure.
The biggest pitfall is getting carried away with various GEO tricks, adding everything you can, leading to chaos and losing sight of priorities.
Things That Didn't Work
<meta name="ai-content-url"> and <meta name="llms">: no standard, no mainstream AI support./.well-known/ai.txt: multiple competing proposals, no real adoption. Wait for a winner.JSON-LD Isn't as Useful as You Think
I initially thought it was a silver bullet, but deeper research revealed it's more complex. SearchVIU ran an experiment: putting data only in JSON-LD without displaying it on the page—none of the five AI systems read it. Mark Williams-Cook's follow-up experiment found that LLMs treat <script type="application/ld+json"> as plain text, not understanding structured semantics.
The only confirmed use is Bing/Copilot, which follows an index enrichment path. Keep existing JSON-LD, but don't expect adding it to make ChatGPT or Claude cite you more.
What Research Says
Princeton and IIT Delhi's GEO paper, published at KDD 2024, found that adding authoritative citations increased AI visibility by 115%, relevant statistics by 33%, and directly citing trustworthy sources by 43%.
My friend @yaojingang does professional GEO research. His geo-citation-lab ran 602 prompts across three platforms, fetching tens of thousands of pages for feature analysis. Check his full report for details. Here are the most useful patterns for content creators from his data:
Being Retrieved ≠ Being Cited
Only 15% of pages retrieved by ChatGPT end up appearing in its answers; 85% are never cited. Getting into the retrieval pool is just the first gate; the model then decides which citations are worth including.
Ahrefs found that cited pages have significantly higher semantic similarity between page titles and user queries. Pages with descriptive natural language URL slugs also have higher citation rates than opaque IDs. llms.txt and Markdown routing work because they give the model a clean, clear signal about what the page is about.
Brand mentions from third-party sources are 6.5 times more likely to be cited than mentions from your own domain. Others saying good things about you on Reddit or Hacker News is far more effective than you saying it yourself. So having a structured llms.txt is important—it gives the model a stable anchor to cite, even if the triggering conversation happens on Reddit.
There are various AI SEO audit tools that score your site, telling you you're missing FAQ, trust pages, or that your body text is too short. Don't get led by scores. The judgment criterion is simple: does each piece of content you add provide information not already on the page? If not, don't add it. I once added a FAQ section to Yobi that said exactly the same thing as the About paragraph, purely to boost the score. Later I realized it was fluff and deleted it.
Everything you do should help AI understand what you have more accurately, giving it a clean workspace. This direction goes further than short-term tricks.
Basic configuration takes about an hour. Knowledge endpoints and project knowledge pages take longer, but once the data structure is set, maintenance is easy—daily syncs run automatically.
After finishing, periodically search your name or project name on ChatGPT, Perplexity, and Claude. Citation sources should become more accurate.
AI citation attribution is still unreliable. CJR and Tow Center tested 200 AI-generated citations and found 153 were partially or completely wrong. Doing structured work makes your content easier to retrieve accurately, but don't treat AI citations as proof that users saw your original words—the mechanism is still improving.
If you have your own product, blog, or website, give it a try. Have fun with the process. You can even give this article to your Claude Code and let it do most of the work.