Chapter 8: Reinforcement Learning

8.1 Chapter Overview

Reinforcement Learning, or RL, is a Machine Learning approach where a model learns by interacting with an environment. The learner, called an agent, takes actions and receives rewards or penalties. Over time, the agent learns which actions lead to better long-term results.

A common example is a robot learning how to move correctly. If the robot moves closer to its target, it receives a positive reward. If it hits a wall or moves into danger, it receives a penalty. Through repeated practice, the robot improves its movement strategy.

8.2 What is Reinforcement Learning?

Reinforcement Learning is inspired by trial-and-error learning. Instead of learning from labelled answers, the agent learns from feedback. The feedback may be positive, negative or neutral.

8.3 Supervised vs Unsupervised vs Reinforcement Learning

8.4 Robot Learning Example

Imagine a robot inside a 5 × 5 grid. The robot starts at one corner and must reach the goal. Some cells may contain danger. The robot can move up, down, left or right.

8.5 Reward Function

The reward function defines how the environment gives feedback. Good reward design is very important. If rewards are poorly designed, the agent may learn the wrong behavior.

8.6 Exploration vs Exploitation

Reinforcement Learning must balance two behaviors:

8.7 Policy, Value and Q-Value

The agent learns Q-values to decide which action is best in each state.

8.8 Q-Learning

Q-Learning is a popular reinforcement learning algorithm. It learns the value of actions in different states. These values are stored in a Q-table.

8.9 Interactive Robot Reward Simulator

Use the buttons to move the robot. The robot receives rewards or penalties depending on the action. This simple simulator helps learners understand state, action, reward and penalty.

8.10 Python Example: Simple Grid Reward Logic

# Simple reinforcement learning style reward system

robot_position = 0
goal_position = 4
danger_position = 2
total_reward = 0

actions = ["right", "right", "right", "right"]

for action in actions:
    if action == "right":
        robot_position += 1

    if robot_position == goal_position:
        reward = 10
        print("Robot reached the goal.")
    elif robot_position == danger_position:
        reward = -10
        print("Robot entered danger.")
    else:
        reward = -1
        print("Robot moved to position:", robot_position)

    total_reward += reward
    print("Reward:", reward)
    print("Total Reward:", total_reward)
    print("-----")

8.11 Python Example: Q-Table Structure

A Q-table stores values for each state-action pair. Initially, values can start at zero.

import numpy as np

states = 5
actions = 2   # 0 = left, 1 = right

q_table = np.zeros((states, actions))

print(q_table)

Line-by-Line Explanation

8.12 Python Example: Basic Q-Learning Update

old_q_value = 0
learning_rate = 0.1
reward = 10
discount_factor = 0.9
best_future_q = 5

new_q_value = old_q_value + learning_rate * (
    reward + discount_factor * best_future_q - old_q_value
)

print("Updated Q-Value:", new_q_value)

8.13 Mini Q-Learning Example: One-Dimensional Robot

This example trains a simple robot to move toward a goal in a one-dimensional line.

import numpy as np
import random

states = 5
actions = 2  # 0 = left, 1 = right

q_table = np.zeros((states, actions))

learning_rate = 0.1
discount_factor = 0.9
episodes = 100

goal_state = 4

for episode in range(episodes):
    state = 0

    while state != goal_state:
        action = random.choice([0, 1])

        if action == 0:
            next_state = max(0, state - 1)
        else:
            next_state = min(goal_state, state + 1)

        if next_state == goal_state:
            reward = 10
        else:
            reward = -1

        best_future_q = np.max(q_table[next_state])

        q_table[state, action] = q_table[state, action] + learning_rate * (
            reward + discount_factor * best_future_q - q_table[state, action]
        )

        state = next_state

print("Trained Q-Table:")
print(q_table)

8.14 Applications of Reinforcement Learning

8.15 Common Beginner Mistakes

8.16 Hands-On Activities

8.17 Interactive Final Assessment Quiz

Each correct answer gives +1 mark. Each wrong answer gives -0.5 mark.

8.18 Chapter Summary

In this chapter, learners studied Reinforcement Learning, agents, environments, states, actions, rewards, penalties, policies, Q-values and Q-learning. Learners also explored robot learning through an interactive grid simulator and Python examples.