Tag: Markov process

national crossword puzzle day december 21

^{Type
post

Eric Holloway
Date
March 3, 2025

Categorized
Large Language Models (LLMs)

Tagged
Andrey Markov (1856–1922), Claude Shannon (1916–2001), Markov process, Model collapse, Redundancy in language}

How the Father of Information Theory Invented Modern AI

_{In 1948 Claude Shannon used Andrey Markov’s 1906 process to formulate an appproach that enabled the development of chatbots (large language models)} _{Eric Holloway
March 3, 2025

5

Large Language Models (LLMs)}

The Markov process also explains why an LLM degrades when it is trained on its own output — its drift toward the most probable state ends in rubbish.