I’d say it’s a grey area… or in this case, a green area.
Actually, the Rules do state that you can remove detached leaves (and other “loose impediments” – natural objects, like twigs or stones or squirrels), unless the impediments and the ball are both in the same hazard (such as a sand trap). It makes no limitation on how many leaves you are allowed to move. And if your ball is on the putting green, you don’t even incur a penalty if moving the leaves causes the ball to move (on any other part of the course, you incur a 1 stroke penalty). You simply have to replace the ball where it was.