I have to wonder if RL would really be able to move beyond the look-up table interpretation though. Since RL still has the problem of exploration (or coverage). From what I've heard there was a fork in RL for robotics in which there was a question of if pure RL policies should be task-based or if they should be tracking-based. Task-based in the end lost out because it's much harder to solve the problem of exploring enough to learn a good behavior without getting stuck in a local optima. Now it seems like you either need human demonstrations to "warm-start" the policy or you go with RL tracking policies. Perhaps I'm misinterpreting your idea at the end though! I'd love to hear more
I agree that talking about "generalization" of RL is its own can of worms. To be clear, you are correct that after you train an RL policy (whether warm starting from imitation or not), its generalization to new environments is questionable -- I think it would be a fair characterization to say that it interpolates between its gathered training experience. (I might be writing more about RL vs. MPC generalization in the future, which would dig deeper into this).
However, (sim2real) RL is still in a whole different category from collecting real-world data. I'm thinking specifically of how parallel simulations can roll out orders of magnitude more data than you could collect of a person actually performing the same task. (Not to mention that it also generates data of recovery)
I have to wonder if RL would really be able to move beyond the look-up table interpretation though. Since RL still has the problem of exploration (or coverage). From what I've heard there was a fork in RL for robotics in which there was a question of if pure RL policies should be task-based or if they should be tracking-based. Task-based in the end lost out because it's much harder to solve the problem of exploring enough to learn a good behavior without getting stuck in a local optima. Now it seems like you either need human demonstrations to "warm-start" the policy or you go with RL tracking policies. Perhaps I'm misinterpreting your idea at the end though! I'd love to hear more
I agree that talking about "generalization" of RL is its own can of worms. To be clear, you are correct that after you train an RL policy (whether warm starting from imitation or not), its generalization to new environments is questionable -- I think it would be a fair characterization to say that it interpolates between its gathered training experience. (I might be writing more about RL vs. MPC generalization in the future, which would dig deeper into this).
However, (sim2real) RL is still in a whole different category from collecting real-world data. I'm thinking specifically of how parallel simulations can roll out orders of magnitude more data than you could collect of a person actually performing the same task. (Not to mention that it also generates data of recovery)