Headcanon 100% accepted! I kind of have him doing that in a story, actually, though not about plums. (Shadow Boy over on Ao3). He's very shy about approaching a vendor (an old woman who crochets doilies and sells them), speaks very softly, hurries away before she can ask questions... but keeps coming back to watch from a distance, because watching her crochet pings some good emotions connected to a warm childhood memory that he hasn't quite remembered fully. I can see him doing just as you describe with the plum vendor, too. Those plums just look too good to keep passing up!