The request here is practical, and the advice is sound: start with Sutton and Barto, but treat it as a foundation, not the final word. The selected chapters, Intro, Finite MDP, TD Learning, and the three on on-policy methods plus policy gradients, form a coherent spine. They give you the core mechanics of how agents learn from interaction, how value functions are approximated, and how policies are refined. That is the right scaffolding for understanding what modern RL-for-LLM papers are building on. If you grasp those chapters, you will not be lost when a paper mentions policy optimization or reward models. You will recognize the lineage.
But here is the part that matters for someone with your background: the gap between Sutton and Barto and the current frontier is not a gap in math, it is a gap in engineering and scaling. PPO and GRPO are not theoretical departures; they are practical adaptations designed to stabilize training when the policy is a massive neural network and the reward signal is sparse or noisy. You do not need a new textbook to understand them. You need to see how the old principles are stretched and patched in the context of transformer architectures and token-level decisions. The Alberta course is excellent, and Joseph Suarez's guide is worth a read, but both are oriented toward robotics and game playing. Your interest is AI for math, where the reward is verifiable and the action space is symbolic. That changes the texture of the problem, not the underlying framework.
So do not overthink the reading list. Work through the chapters the LLM suggested, but do it actively. Implement a small TD learning algorithm on a toy environment. Then move to a simple policy gradient method. Once you have that muscle memory, read a few recent papers on RLHF or reasoning models with the math already in hand. The papers will reference concepts like advantage estimation or KL penalties, and you will recognize them as variations on what you studied. That recognition is the goal. You are not trying to become a deep RL researcher overnight. You are building a map so that when you encounter a new technique, you can place it on the terrain.
The real takeaway is that your math background is an asset, but it does not do the work for you. The theory gives you precision, but only experimentation gives you intuition. So pick a small project, run it, break it, fix it, and let the experience teach you what the equations mean in practice. That is how you move from understanding RL as a topic to using it as a tool. The chapters are a good start. The next step is to close the laptop and write some code.