What Sources Does AI Use? Training Data, Web Search and RAG Explained
Last updated: 2025 · Reading time: about 9 minutes
Ask an AI assistant which CRM suits a mid-sized manufacturer, or which agency is best for B2B ads in your city, and you get a confident answer in seconds, often with links. For sales and marketing teams, that raises a practical question: what sources does AI use, and why do some brands appear in those answers while others never do? Most articles on this topic actually answer a different question (“where is AI used?”) and list industries. This guide answers the real one. It explains where AI's knowledge comes from, how assistants choose the websites they read, how reliable those sources are, and how your brand can become one of them.
AI uses two kinds of sources: training data and live sources. Models are trained on web archives such as Common Crawl, Wikipedia, books, academic papers, code repositories and licensed content. Assistants like ChatGPT, Gemini and Perplexity also run web searches while answering, read current pages and cite them. Enterprise systems add the company's own documents through RAG.
What sources does AI use? (Short answer)
Think of AI's sources as three layers stacked on top of each other:
- Training data: Large volumes of text, code and other data the model learns from before release. This layer is frozen at a specific date.
- Live web search (retrieval): When search is enabled, the assistant queries a search index, reads a handful of pages and summarizes them with citations.
- Your own data: Files a user uploads, or a company knowledge base connected through Retrieval-Augmented Generation (RAG).
One distinction matters more than any other: a model does not store facts like a database. During training it learns statistical patterns in language. That is why it can phrase an outdated or wrong fact fluently, and why it has to search the web for anything recent or specific.
What is AI, briefly?
Artificial intelligence is the field of building systems that perform tasks normally requiring human intelligence: understanding language, recognizing images, making predictions and taking decisions. Its main sub-fields are:
- Machine learning: Algorithms that learn patterns from data instead of following hand-written rules.
- Deep learning: Machine learning with multi-layer neural networks; the basis of modern language models.
- Natural language processing (NLP): Understanding and generating human language.
- Computer vision: Interpreting images and video.
- Robotics: Applying AI to physical machines that sense and act.
Across these fields, AI learns from several data types: written text, audio, video, photos and sensor data. Chat assistants are large language models (LLMs), a combination of NLP and deep learning trained mostly on text and code.
What data are AI models trained on?
The backbone of almost every large language model is Common Crawl, a nonprofit that has been crawling the public web since 2008 and publishes its archive openly. Each crawl contains billions of pages. Because raw web data is noisy, model builders filter it heavily; Google's C4 (Colossal Clean Crawled Corpus) is a well-known cleaned version.
On top of web data, typical training sets include:
- Wikipedia for structured, encyclopedic knowledge
- Digital book collections for long-form, high-quality prose
- Academic papers, for example from arXiv
- Code repositories such as GitHub
- Q&A sites such as Stack Exchange
Real data mixes: GPT-3 and LLaMA
Two research papers published exact training mixtures. The GPT-3 paper (Brown et al., 2020) reports:
| Dataset | Tokens | Weight in training mix |
|---|---|---|
| Common Crawl (filtered) | 410 billion | 60% |
| WebText2 | 19 billion | 22% |
| Books1 | 12 billion | 8% |
| Books2 | 55 billion | 8% |
| Wikipedia | 3 billion | 3% |
Note that weight is not the same as size. Higher-quality sets were sampled more often: according to the paper, Wikipedia was seen about 3.4 times during training, while filtered Common Crawl was seen less than once.
Meta's LLaMA paper (Touvron et al., 2023) reports this pre-training mix:
| Source | Sampling share | Disk size |
|---|---|---|
| CommonCrawl | 67.0% | 3.3 TB |
| C4 | 15.0% | 783 GB |
| GitHub | 4.5% | 328 GB |
| Wikipedia | 4.5% | 83 GB |
| Books | 4.5% | 85 GB |
| ArXiv | 2.5% | 92 GB |
| StackExchange | 2.0% | 78 GB |
Newer frontier models rarely publish this level of detail. The GPT-4 technical report, for example, explicitly declined to describe dataset construction, citing the competitive landscape and safety. So treat these tables as representative of the method, not as the exact recipe of today's assistants.
Licensed content deals
As legal questions around web scraping grew, AI companies started paying for access. In 2024 Reddit announced data agreements with both Google and OpenAI. OpenAI also signed content partnerships with publishers including The Associated Press (2023), Axel Springer (2023), the Financial Times (2024) and News Corp (2024). Depending on the deal, licensed content can be used for training, for displaying and citing articles in answers, or both.
Human feedback and synthetic data
Raw text is not enough to make a helpful assistant. Models are refined with reinforcement learning from human feedback (RLHF), where people rank model outputs, an approach described in OpenAI's InstructGPT paper (Ouyang et al., 2022). Developers also increasingly use synthetic data, meaning examples generated by other models, to teach specific skills.
Why AI doesn't know recent events: the knowledge cutoff
Training data is collected up to a knowledge cutoff date. Anything that happened afterwards, such as a product launch, a price change or a new competitor, is simply not in the model's weights. Unless the assistant searches the web, it will answer from older patterns, sometimes without signalling that its information may be outdated.
Which websites do ChatGPT, Gemini, Perplexity and Claude read?
When search is active, each assistant relies on a different retrieval infrastructure:
| Assistant | Search infrastructure | Cites sources? | Note |
|---|---|---|---|
| ChatGPT | OpenAI web search, using third-party search providers (including Bing) and publisher partner content | Yes, when search is used | Without search, it answers from training data |
| Gemini | Grounding with Google Search | Yes, links to sources | Benefits directly from Google's index |
| Perplexity | Its own index (built by PerplexityBot) and Sonar models | Yes, numbered citations on answers | Search-first by design |
| Claude | Built-in web search tool | Yes, when search is used | Availability depends on plan and settings |
| Google AI Overviews | Google Search index | Yes, link cards | Appears directly on results pages |
How retrieval (grounding) works
- You ask a question.
- The assistant rewrites it into one or more search queries.
- It retrieves results and reads a few pages.
- It summarizes what it read into an answer.
- It attaches links to the pages it used.
Which kinds of sites get read most
The pages that tend to appear in AI answers include Wikipedia, news sites, forums (Reddit globally, local forums such as Ekşi Sözlük in Turkey), comparison and “best of” list articles, official brand websites, review platforms and YouTube.
Why different assistants cite different sources
The same question produces different citations because each assistant uses a different index, rewrites queries differently, ranks pages with its own logic, weighs freshness differently and may factor in language and location. That is why your brand can be recommended by one assistant and ignored by another.
How does AI access websites? Crawlers and robots.txt
AI companies reach your site through named bots:
| Bot | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collecting training data |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search |
| ChatGPT-User | OpenAI | Fetching a page during a user's request |
| Google-Extended | Control token for Gemini training and grounding (does not affect Google Search or AI Overviews) | |
| ClaudeBot | Anthropic | Collecting training data |
| PerplexityBot | Perplexity | Indexing for Perplexity search |
| CCBot | Common Crawl | Building the open archive many models train on |
The key difference: training bots decide whether your content shapes future models, while search and user bots decide whether your pages can be read and cited in answers today. You can control them separately in robots.txt:
# Block training crawlers, allow AI search crawlers
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Be careful: blocking search bots removes your pages from cited answers, which usually hurts visibility. Providers also treat user-initiated fetchers differently, so check each provider's documentation before deciding.
How does enterprise AI use its own sources? (RAG)
Retrieval-Augmented Generation (RAG) connects a language model to a company's own knowledge: documents, FAQs, product catalogs, price lists and CRM data. When a question comes in, the system first retrieves the most relevant internal passages, then asks the model to answer using only that material.
A typical example is a sales or support assistant that answers prospects exclusively from approved product information instead of the open web. The same principle applies when you set up AI agents trained on your brand voice and qualification criteria: the narrower and better curated the source set, the more consistent the answers.
Restricting sources does not eliminate hallucination, but it reduces it significantly in practice, because the model has relevant, verified text in front of it and can point to the exact document it used.
Are the sources AI uses reliable?
Only partially. Hallucination, meaning fluent but false output, can happen even when an answer shows citations. The model may misread a page, merge two sources, or attach a link that does not actually support the claim. Verifiable sources reduce the risk, but only if someone verifies them.
Language is another factor. English dominates web archives such as Common Crawl, and Turkish makes up a comparatively small share. The LLaMA paper, for instance, states that its Wikipedia data covered 20 languages using Latin or Cyrillic scripts, and Turkish was not among them. Less Turkish training data typically means weaker coverage of local brands, regulations and terminology, which makes live search and well-structured Turkish content even more important.
A quick checklist for users:
- Open the cited links and confirm they say what the answer claims.
- Check the publication date of each source.
- Ask the same question in at least two assistants.
- For numbers, prices and legal points, go to the primary source.
How can your brand become one of AI's sources?
Generative Engine Optimization (GEO) is the practice of making your brand easy for AI assistants to find, understand, cite and recommend. Content that AI tends to quote shares common traits:
- Verifiable claims with specific numbers and named sources
- Question-and-answer formatting that mirrors how buyers ask
- Structured data (schema markup) and clean, crawlable pages
- Visible, recent update dates
- Third-party mentions in news, directories, comparisons and forums
Measurement comes next. You need to know which assistants mention your brand, which sites they read to form their answers, and your share of voice against competitors. A citation source analysis that shows which domains AI assistants read for your category turns GEO from guesswork into a concrete list of pages to win.
Jumpix AI Visibility asks the questions your customers ask ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews, then reports whether your brand is mentioned, recommended, cited and described correctly, along with what to fix. You can run a free instant AI visibility check by entering your industry and city: it tests 5 buyer questions across 3 engines with no sign-up. Plans start at 9.99 USD per month for one brand, and Turkish companies that prefer a done-for-you approach can use the managed GEO service.
Where is AI used most? (Quick overview)
Each application relies on its own data source:
- Healthcare: medical images and patient records for diagnostic support.
- Finance: transaction histories for fraud detection and credit scoring.
- Education: learner performance data for adaptive lessons.
- E-commerce: browsing and purchase history behind Amazon and eBay recommendations.
- Navigation: location and traffic data behind Google Maps routes.
- Music: listening history behind Spotify recommendations.
- Film and series: viewing history behind Netflix suggestions.
- Automotive: camera, radar and sensor data for driver assistance.
- Marketing: campaign and engagement data for targeting and content.
- Sales automation and chatbots: CRM records, form submissions, conversation history and approved product documents to qualify leads and answer customers.
Conclusion
So, what sources does AI use? Three layers: training data built mostly from Common Crawl, Wikipedia, books, code and licensed content; live web search that differs by assistant; and private data connected through RAG. For B2B sales and marketing teams, the middle layer is the one you can influence now. Make your pages crawlable for AI search bots, publish verifiable and well-structured content, earn third-party mentions, and measure which sources assistants actually cite in your category.
Sources
- Brown et al. (2020), Language Models are Few-Shot Learners (GPT-3), arXiv:2005.14165
- Touvron et al. (2023), LLaMA: Open and Efficient Foundation Language Models, arXiv:2302.13971
- Ouyang et al. (2022), Training language models to follow instructions with human feedback, arXiv:2203.02155
- OpenAI (2023), GPT-4 Technical Report
- Common Crawl, commoncrawl.org
- OpenAI, Google, Anthropic and Perplexity crawler documentation
- Public announcements by Reddit, OpenAI and partner publishers (2023–2024)
Frequently Asked Questions
Where does AI get its information, and is it connected to the internet?
A model's base knowledge comes from training data collected up to its knowledge cutoff, so the model itself is not “online.” Assistants become connected when they use a web search tool, which lets them read current pages and cite them. Without search, answers come only from patterns learned during training.
What sources does ChatGPT use, and does it cite them?
ChatGPT was trained on a mix of public web data, licensed content and human feedback data. When its search feature is active, it uses OpenAI's web search, which draws on third-party search providers including Bing and publisher partners, and shows links to the pages it used. Without search, it does not cite sources.
What are the sub-fields of AI?
The main sub-fields are machine learning, deep learning, natural language processing, computer vision and robotics. Chat assistants mainly combine deep learning and natural language processing. Each sub-field learns from different data types such as text, audio, images, video and sensor data.
Where is AI used most? 10 examples
Healthcare, finance, education, e-commerce recommendations, navigation, music recommendations, film and series recommendations, automotive, marketing, and sales automation with chatbots. Each relies on a specific data source, such as purchase history for Amazon or listening history for Spotify.
What data does AI-powered sales automation use?
AI sales automation typically uses CRM records, form submissions, website and campaign source data, email and WhatsApp conversation history, firmographic details and approved product documents. With a RAG setup, the assistant answers prospects only from this verified company information, which keeps replies accurate and on-brand.
Related reading
<!-- rank-agent-related -->- What Is ChatGPT SEO? A Guide to Getting Cited by ChatGPT <!-- rank-agent-link:02b602fe-8ecd-48ad-b111-2cee16bedb3d -->
- What Is llms.txt? What It Does, How to Add It and Does It Work <!-- rank-agent-link:cdefb7dc-953f-4df0-918c-8f4a13e23c49 -->