It is possible that the trees were all eaten and it is why we don’t see them today.
Legend has it that the tree grows "in remote mountain valleys in China." The jinmenju's fruit appear to be "...human heads. The faces are always smiling or laughing, even as they fall from their branch."
It is said that the Jinmenju seed also has a human face, eyes, ears, nose, and mouth.
Significant accounts across Asia
According to the Edo period Hyakka Jiten encyclopedia Wakan Sansai Zue (和漢三才図会; A Collection of Pictures of Heaven, Earth, and Man from China and Japan), the Jinmenju trees are found in the south, and the fruit of the tree is called the jinmenshi, or human-faced child. They ripen in the fall, and if eaten the fruit will have a sweet/sour taste. In the past it was said that people planted great orchards of the laughing Jinmenju.
The legend of the Jinmenju comes from China, and was passed onto Japan where it was considered to be a kami due to its peculiar nature. However, there are also stories of trees bearing human-faced fruit from India and Persia, usually with the faces of beautiful girls.