What are the Models of Reinforcement Learning

 

Reinforcement learning (RL) is concerned with how the machine learning models can train agents to make decisions in an environment by performing actions and monitoring the results. For every good action, the agent gets rewards, and for every bad action, the agent gets penalties. In short, RL models learn by trial and error to make better decisions in the future.

There are 5 key components in Reinforcement learning:--

a) Agent: An entity which performs action
b) Environment: The world or system in which the agent operates.
c) State: The situation or condition the agent is currently in.
d) Action: The different choices that the agent can make
e) Reward: The result from the environment based on the agent’s action.

This Machine Learning category can be implemented by following the below approaches--

• Value-Based Models
• Policy-Based Models
• Model-Based Reinforcement Learning
• Deep Reinforcement Learning Models

In this article, we will talk about the functionality of the Models used in each of the above categories.

Value-Based Models

This RL category focuses on learning the value of states or state-action pairs to derive an optimal policy. The goal is to maximize the value function, which represents the long-term cumulative reward.

Q-Learning

Q-Learning is a model free RL algorithm that helps an agent learn how to make the best decisions by interacting with the environment. “Q” stands for Quality which represents how valuable the action is in maximizing future rewards. 

The working of this model is through a set of key components as illustrated in the diagram below:

The functionality of these components can be explained as follows—

a) Agent: A learner or decision maker who observes the environment and takes actions.

b) Action: The move that the agent can make based on a specific strategy.

c) Environment: Defines the problem's rules and determines the outcome of each action.

d) State: Represents the agent's current condition or situation.

SARSA

SARSA stands for State-Action-Reward-State-Action, which helps an agent to learn an optimal policy by interacting with the environment. Being the modified version of the Q-learning, it learns from the actual action taken by agent.

The main idea behind SARSA is trial and error. The agent takes an action in a situation, observes the result, and adjusts its strategy based on the outcome. This process is repeated many times, leading to improvements in the agent’s decisions over time.

The Update Equation rule used in SARSA is—

Where:

• Q(s,a) = Current estimate of action value
• α = Learning rate
• γ = Discount factor
• r = Immediate reward
• Q(s',a') = Value of the next state-action pair actually chosen

Policy-Based Models

The models in this segment directly learn and optimize the policy mapping states to actions, often using gradient-based methods for continuous or stochastic action spaces.

REINFORCE

REINFORCE is a policy gradient algorithm based on Monte Carlo methods, which directly learns a policy by adjusting its parameters to maximize the expected cumulative reward. Introduced by Ronald Williams in 1992, it is considered as the foundation of modern policy-based RL methods.

The key working steps involved in this algorithm are as follows:--

a) Collect Episodes: The agent interacts with the environment, generating trajectories of states, actions, and rewards.
b) Calculate Returns: For each time step, compute the discounted cumulative reward (return) from that point onward.
c) Policy Gradient Update: Update the policy parameters using the gradient of the log-probability of actions, weighted by the returns.
d) Repeat: Iterate over multiple episodes to refine the policy.

Vanilla Policy Gradient (VPG)

VPG is one of the most fundamentally pure algorithms in policy-based Reinforcement Learning. This model operates by explicitly modeling the policy and updating its parameters in the direction that improves long-term performance.

The working process of the VPG algorithm is as follows—

a) Initialize Policy: Start with a random policy network.

b) Generate Episodes: Interact with the environment and collect states, actions and rewards.

c) Compute Returns: Calculate cumulative future rewards.

d) Compute Policy Gradient: Estimate the gradient using collected experiences.

e) Update Policy: Adjust policy parameters to increase expected rewards.

f) Repeat: Continue until the policy converges.

Model-Based Reinforcement Learning

In this type of Reinforcement Learning, the agent uses the model of environment to predict what will happen after taking actions, and then uses those predictions to choose better actions.

Dyna-Q

Developed by Richard Sutton in 1990, the core idea behind Dyna-Q is to combine real experience with simulated experience generated by a learned model. When the agent interacts with the environment, it stores:

• Estimated value of each state-action 
• What it has learned about the environment

The key steps involved in the working process of Dyna-Q algorithm are as follows—

a) Initialize the Q-table and an empty model (which will store transitions).

b) Take an action in the environment → observe reward and next state.

c) Real Experience Update: Update the Q-value using the Q-learning rule.

d) Update the Model: Record the observed transition

e) Planning Step:

• Randomly sample a few previously observed state-action pairs.
• Use the model to simulate what would happen next.
• Apply the same Q-learning update using the simulated data.

f) Repeat steps b)–e) until convergence.

World Models

This approach involves creating a model that simulates the environment's dynamics using past experiences. It often employs recurrent neural networks (RNNs) to learn the transition function f(s,a) from state s and action a.

World Models has following categories—

a) Explicit models: The model directly predicts environmental quantities. This is useful when the state is well defined.
b) Latent models: The model predicts future latent representations, rather than reconstructing the complete environment.
c) Probabilistic models: Instead of predicting one future, the model represents uncertainty. This is useful when the environment is stochastic.

Deep Reinforcement Learning Models

Deep Reinforcement Learning (Deep RL) combines reinforcement learning (RL) with deep neural networks. It enables an agent to learn how to make decisions directly from complex data such as images, sensor readings, or large state spaces.

Deep Q Network (DQN)

DQN in reinforcement learning is a combination of deep neural networks and Q-learning that enables agents to learn optimal policies in complex environments. Unlike tabular Q-learning, which stores Q-values in a table, DQN learns a function approximation.

The core components used in DQN architecture are as follows—

a) Input Layer: receives state information from the environment in the form of numerical values.
b) Hidden Layers: consist of multiple fully connected neuron that transform the input data into more complex features that are more suitable for predictions.
c) Output Layer: Each possible action in the current state is represented by a single neuron in the DQN's output layer. The output values of these neurons represent the estimated value of each action within that state.

d) Memory: DQN utilizes a memory replay to store the training events of the agent.

e) Loss Function: the DQN computes the difference between the actual Q-values form replay memory and predicted Q-values to determine loss.

f) Optimization: It involves adjusting the network's weights in order to minimize the loss function.

Proximal Policy Optimization (PPO)

PPO is a policy optimization algorithm that trains an agent to make better decisions by directly improving its policy (the mapping from states to actions). The key idea is to update the policy gradually and prevent it from changing too much in a single training step.

The training cycle behind PPO is as follows—

a) Agent interacts with environment.
b) Collect trajectories (state, action, reward).
c) Compute advantages (how good an action was).
d) Update policy network using PPO objective.
e) Repeat until convergence.

Conclusion

Overall, Reinforcement Learning is a powerful Machine Learning technique that allows agents to learn from rewards and penalties. The models discussed in this article demonstrate different ways of achieving effective decision-making.