History
In 1989 Christopher Watkins published his PhD thesis “Learning from Delayed Rewards” where he introduced the concept of:
Q-learning.
Card 1 of 259. Answer: Q-learning.
Q-learning.
Card 1 of 259. Answer: Q-learning.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Q-learning. Christopher Watkins introduced it in his 1989 University of Cambridge PhD thesis "Learning from Delayed Rewards". Q-learning is a model-free reinforcement learning algorithm that learns Q(s, a), the expected discounted future reward of taking action a in state s and acting optimally afterwards.
Each step updates Q(s, a) ← Q(s, a) + α [r + γ maxₐ′ Q(s′, a′) − Q(s, a)]. With Q(s, a) = 2, r = 1, γ = 0.9, maxₐ′ Q(s′, a′) = 5 and α = 0.1, the target is 1 + 0.9 × 5 = 5.5, so Q(s, a) moves to 2 + 0.1 × 3.5 = 2.35. Watkins and Peter Dayan published a full convergence proof in 1992.
Interviewers ask why Q-learning is off-policy: the max learns the greedy policy's values while the agent explores, unlike SARSA, which learns the values of the policy it follows. Lookup tables fail for large state spaces, which led to DeepMind's Deep Q-Network (Nature, 2015), and the max overestimates values, which Double Q-learning reduces.