The state value, or V-value, is the expected return (cumulative future reward) starting from a particular state and following a specific policy thereafter. It is represented as \( V(s) \), where \( s \) is the current state. Unlike Q-values, state values do not consider individual actions — they provide a single number that summarizes how good a state is under the current policy.
State values are particularly useful in policy evaluation, which is the process of assessing how good a given policy is. In dynamic programming and Monte Carlo methods, state values are updated based on returns observed during episodes. They are also used in actor-critic algorithms, where the critic estimates V(s) to help the actor improve its policy.
While Q-values provide more detailed action-level information, V-values are more abstract and are often used when actions are chosen by a separate policy mechanism. Estimating accurate state values is critical for learning efficient behaviors in large or continuous environments. They also serve as a foundation for computing advantages, which quantify how much better a specific action is compared to the average expected value of the state.