Raghad Alshawish

Artificial Intelligence Specialization · Cohort 2026

Reliability-Weighted Advantage Estimation for Proximal Policy Optimization

Proximal Policy Optimization (PPO) relies on advantage estimates to guide policy updates, while Generalized Advantage Estimation (GAE) controls the bias–variance trade-off through its 𝜆 parameter. However, conventional PPO does not explicitly account for the reliability of individual advantage estimates. In this work, we propose a TDerror-based reliability weighting mechanism that adaptively adjusts the contribution of GAEadvantages according to their associated temporal-difference errors. The proposed weighting is applied to the policy learning signal while preserving the standard PPO critic update. Weevaluate the proposed method on the HalfCheetah continuous-control environment using two independent random seeds. Standard PPO is evaluated across four GAE-𝜆 configurations, and the best-performing configuration is selected independently for each seed. RW-PPO is evaluated using a fixed reliability coefficient and GAE-𝜆 configuration. Compared with the selected PPO baselines, RW-PPO improves the mean evaluation rewardby2.66and15.31pointsandthebestevaluationrewardby12.13and8.12pointsfor Seeds 42 and 123, respectively. It also achieves higher area under the evaluation-reward curve (AUC) for both seeds, indicating improved performance throughout training. These results provide initial evidence that incorporating TD-error-based reliability information into the advantage signal can improve PPO beyond conventional GAE-𝜆 selection in the evaluated HalfCheetah setting.

Read the final report (PDF)