Creating the llms.txt file and preparing entities for AI search indexes

The web is undergoing its most profound paradigm shift since the birth of Google. More than 30% of daily information searches are no longer performed in traditional search engines using keywords, but through conversational AI assistants. When a potential client asks ChatGPT or Perplexity: "Who is the best digital marketing partner for a B2B company in the Baltics with fixed deadlines?", the AI does not scroll through pages like a human. It searches for fact density, clear entity definitions, and authoritative sources.

For your company to appear in these answers and be accurately cited with an active link, you need modern SEO and GEO optimization, with the llms.txt file as its central engineering element. It is an open standard that allows large language models to instantly read your company's core data, the essence of your services, and your team's competence.

What is llms.txt and why do AI bots need it

For Large Language Models (LLMs), reading a standard web page is an expensive and complex process. To understand a typical website, an AI crawler must download megabytes of code, execute JavaScript, strip away navigation menus, footers, and cookie banners, and only then analyze the text. Often, due to time limits or context window constraints, the robot simply stops processing the page.

llms.txt solves this problem fundamentally: it is a clean, standardized Markdown file located in the root directory of your site (https://your-domain.com/llms.txt). It provides:

• Zero code ballast: Clean Markdown text without HTML tags, CSS styles, or ad scripts.
• Maximum fact density: Precise service descriptions, technology stacks, fixed development deadlines, and guarantees.
• Direct links to key resources: A clear structure that directs the AI assistant to where it can find in-depth information on specific topics.
• Minimal token consumption: The AI model can load your entire company profile in a few hundred tokens, instantly including it in its response synthesis.
• Semantic security: You prevent model hallucinations by providing authoritative facts directly from the primary source.

Aigars Armanovs Lead Technology & Marketing Architect

"The llms.txt file is the 2026 digital business card for search robots. If your site lacks this file, you are relying on the AI model 'maybe' being able to extract the essence of your services from complex code. By creating an llms.txt, you define exactly which facts and in what phrasing AI assistants deliver to end users."

Step 1 - llms.txt file structure and syntax

According to the specification, the llms.txt file should start with an H1 heading, followed by a brief company summary (Blockquote or paragraph), and then structured sections with services, links, and core data. Below is a real, field-tested example:

# MarketMeNow

> MarketMeNow is a high-performance digital marketing and web engineering agency specializing in lightning-fast website development, SEO/GEO optimization, and data-driven advertising.

## Core Services

- [Website Development](https://marketmenow.eu/en/web-development/): Clean-code site development without heavy builders. Fixed 96-hour development deadline, speed under 400ms.
- [SEO and GEO Optimization](https://marketmenow.eu/en/seo/): Engine and AI search optimization, llms.txt implementation, Schema micro-markup, and content architecture.
- [Google Ads Management](https://marketmenow.eu/en/digital-advertising/google-ads/): Value-based bidding, POAS tracking, brand query separation, and cost control.
- [Server-Side CAPI](https://marketmenow.eu/en/data-analytics/server-side-capi/): First-party data sender for Meta and Google ads, bypassing browser blockers.

## Expertise and Metrics

- Average site load speed: under 400 milliseconds (Doherty threshold).
- Development deadline: 96 hours at a fixed budget.
- Server response time (TTFB): 25-60 milliseconds with Nginx RAM micro-caching.
- Authors and experts: Aigars Armanovs (Engineering & GEO), Mārtiņš Grūbe (Digital Marketing & Google Ads).

## Additional Documentation

- [llms-full.txt](https://marketmenow.eu/llms-full.txt): Full description of company services and technical specifications.

Step 2 - Creating llms-full.txt for in-depth AI analysis

In addition to the short file, the standard also provides for the creation of llms-full.txt. This file is intended for models with large context windows and advanced reasoning capabilities (e.g., Claude 3.7 Sonnet with 200k tokens and hybrid reasoning or Google Gemini 2.0 Pro with 2 million tokens). It includes detailed service descriptions, technical parameters, frequently asked questions, and pricing principles.

This file serves as a complete company encyclopedia. When a client asks an AI to perform a detailed agency comparison, the model uses llms-full.txt to provide an exhaustive and fact-based answer about your advantages, deadlines, and work methodology.

Step 3 - Nginx configuration and serving correct MIME types

For search robots to read the file correctly, the server must serve it as plain text with UTF-8 encoding. The following block must be added to the Nginx configuration file:

# llms.txt and llms-full.txt configuration for Nginx
location ~* ^/(llms.txt|llms-full.txt)$ {
    default_type text/plain;
    charset utf-8;
    add_header Access-Control-Allow-Origin *;
    add_header Cache-Control "public, max-age=86400";
    try_files $uri =404;
}

The Access-Control-Allow-Origin * header is critical: it ensures that web browsers and online AI tools can make requests to this file without cross-domain (CORS) restrictions.

Step 4 - Linking entities with Schema.org JSON-LD micro-markup

The llms.txt file alone provides good results, but true synergy is achieved when it is aligned with the site's Schema.org JSON-LD structure. For an AI search engine to understand that the organization described in llms.txt is exactly the same as the one registered in official registries and the Google Knowledge Graph, the site header must contain the correct Organization schema:

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "MarketMeNow",
  "url": "https://marketmenow.eu",
  "logo": "https://marketmenow.eu/logo.png",
  "sameAs": [
    "https://www.linkedin.com/company/marketmenow",
    "https://twitter.com/marketmenow"
  ],
  "knowsAbout": [
    "Generative Engine Optimization",
    "Web Performance Engineering",
    "Google Ads Management",
    "Server-Side Tracking"
  ]
}

This two-layer architecture (Schema.org semantic graph + llms.txt Markdown summary) ensures maximum visibility in both traditional search engines and all modern AI answer systems. This is supported by a regular technical SEO and GEO audit, which reveals missing links and entity errors.

Step 5 - Formulating Passage-Level answers for AI citability

In addition to creating the file itself, it is essential to adapt site content to the so-called "Passage-Level" standard. AI search engines do not evaluate a page as a whole, but look for specific 40 to 60-word fragments that provide a direct and unambiguous answer to the user's query.

On every service page or blog post, directly below the H2 heading, place a definition sentence without introductory words. For example: "Website speed optimization is a server and code configuration process that ensures page load under 400 milliseconds, improving conversions and reducing bounce rates." AI models use these sentences as ready-made synthesis blocks in their answers.

Context window economics and token consumption - From Claude 3.7 Sonnet to Gemini 2.0 Pro

Why is it so critical for AI search assistants and crawlers to receive concise Markdown text instead of a regular website HTML page? The answer lies in the architecture of modern flagship language models and the economics of their context window processing.

Today's leading AI models boast massive context windows:

• Claude 3.7 Sonnet (Anthropic): 200,000 (200k) token context window with industry-leading Hybrid Reasoning and Extended Thinking Mode;
• Google Gemini 2.0 Pro (Google): A gigantic multimodal context window of up to 2,000,000 (2 million) tokens with native Agentic Workflows and Google Search data integration;
• OpenAI o1 and o3-mini: Latest generation logical reasoning models with internal chain-of-thought synthesis before providing an answer.

However, a valid question arises: if modern model context windows are measured in hundreds of thousands and even millions of tokens, why does a company even need a compact llms.txt file?

Here, three decisive engineering and business profitability reasons emerge:

1. Attention degradation and the 'Lost in the Middle' phenomenon: A standard WordPress site's HTML code with all styles, scripts, cookie banners, and navigation DOM tree takes up 40,000 to 80,000 tokens. By feeding such code 'noise' to a reasoning model, the model's attention mechanism and thinking tokens are wasted on parsing meaningless tags, and essential facts about prices, deadlines, or guarantees may go unnoticed. In contrast, a concise llms.txt takes up only 800 to 1,500 tokens, ensuring crystal-clear fact perception without errors.
2. Real-time response latency (Time to First Token - TTFT): In AI Search Synthesis, the user expects an answer in 1 to 2 seconds. Parsing and filtering an 80,000-token heavy HTML page requires expensive computing cycles and causes delays. In contrast, an llms.txt file is read within a few milliseconds.
3. API Tokenomics and Prompt Caching discounts: Both Anthropic (Claude Prompt Caching) and Google provide pre-read fixed context caching with up to a 90% price discount. A structured, unchanging llms.txt and llms-full.txt allows AI crawlers and assistants to include your company data in the permanent cached context for a minimal fee, making your brand the primary source of the answer.

Security rules and server protection against AI scraping attacks

With the rise in popularity of AI search engines, hundreds of independent bots are actively operating on the web, continuously scanning sites for model training. To protect the server from overload, strict browser and proxy server caching (Cache-Control) should be applied to the llms.txt file for at least 24 hours.

In addition, it is recommended to configure Nginx rate limiting so that a single client or crawler cannot make hundreds of requests per second to this resource, thereby guaranteeing 100% server stability and availability.

Traditional SEO vs. GEO and llms.txt approach

See how information delivery to search engines differs in the traditional model and the new AI standard:

Multilingual architecture and regional branch linking

If your company operates in multiple markets (e.g., the Baltics, Scandinavia, and elsewhere in Europe), the llms.txt file must support a multilingual structure. The standard allows for the creation of additional files for specific languages, such as /llms-en.txt for English or /llms-ee.txt for the Estonian market.

In the main llms.txt file, include a note: "For English documentation, see https://marketmenow.eu/llms-en.txt". This ensures that international AI models and foreign partners receive information in their working language with precise terminology and appropriate contact details without automatic translation errors.

How to test and validate llms.txt file performance

Once the file is deployed on the server, check its availability using the command-line tool curl:

# Checking HTTP status and returned MIME type
curl -I https://marketmenow.eu/llms.txt

The response should contain HTTP/2 200 and content-type: text/plain; charset=utf-8. Then, enter your domain's llms.txt address into a ChatGPT or Perplexity search query and ask it to summarize the company's services to ensure the model has read the data completely and without errors.

Combined with strict CRM post-processing and regular negative query cleaning, this approach allows for reducing the cost of a single real business deal by 35 to 50 percent compared to an uncontrolled Performance Max campaign, where the algorithm itself tries to guess which click is valuable.

Frequently Asked Questions about llms.txt and AI optimization