When I teach my workshop on 3D vision, the students also play with a 3D graphics framework called Scene Kit. It is very powerful and easy to use. With just a few lines of code they can create a vivid 3D scene. At some point in the tutorial, the student has created a scene with a red sphere on a grey shiny floor against a deep blue sky. Then they start playing with the controls, changing the distance, rotating the scene, or panning to the side. Invariably someone pushes the sphere to one of the corners of the image and then checks with me whether there is a mistake in the code.
What happens is that the sphere does not look round anymore but strangely stretched, or elliptical in mathematical parlance. I remember my own confusion when I first learned about this. In the beautiful book “Optics, painting, and photography” by M.H. Pirenne (1970), the author discusses pinhole cameras and writes that he built one himself. He took a picture in Rome of a rooftop scene with a sphere. It clearly does not look round.

Pirenne explains that the effect is due to the fundamental difference between “natural perspective” and “linear perspective”.
The explanation is quite simple and is as old as the first mathematical treatise on vision written by Euclid. In his Optics (300 BC) the second definition reads: “the figure contained by a set of visual rays is a cone of which the vertex is at the eye and the base at the surface of the object seen.” This is called “natural perspective” and for a sphere the visual cone is simply a circular cone.
As a side note. In the first definition Euclid states that light rays are “rectilinear”, or straight lines. But foreign to our modern understanding, he held the view that light rays go from the eyes to the objects in the environment. It took more than a thousand years for the Arab scholar Alhazen to set the record straight in his Book of Optics (1000 AD).
Back to the explanation. If the sphere is photographed, the circular cone is intersected by the image plane. That is called “artificial perspective”, or “linear perspective”. That section of the circular cone is literally a conic section. If the sphere is photographed off-axis, the cone will be cut by the plane at an angle and the resulting intersection is an ellipse. It is really not more complicated than this.
So why do we rarely notice this in photographs or paintings? For photographs, the visual angle of the field of view is usually relatively small for this effect to become apparent.
Painting is a different story. Let us look at one of the iconic Renaissance paintings that is often taken as the textbook example of a perspective painting: the endlessly fascinating The School of Athens by Raphael (1509-1511).

Raphael shows us a large scene with the vanishing point somewhere between the shoulders hips of Plato and Aristotle, the central characters of the painting. All the way to the right, we see two characters carrying a celestial and a terrestrial globe. Those are spheres and they look perfectly round. However, in “correct” linear perspective they should have been rendered elliptical.

Why did Raphael do this? He may not have been aware of what he did. While he was painting the figures at the periphery, he was standing right in front of them and painting them as he normally would. So the painting is not entirely in linear perspective with one overall vanishing point, but more like a “perspective collage” with multiple local vanishing points.
That the spheres are not elliptical has been discussed many times before in the history of art literature. What I have not seen before is a more interesting effect that you can observe yourself. You need a large reproduction of the painting, or show it on a large HD display. It must span at least 90 degrees of your visual field. Close one eye and position your open eye at the optical axis of the painting. Now focus on Plato and Aristotle and their direct environment and the scene gradually gains more depth. At some point you may experience that the people and the spheres on the right may actually look a bit flat. Especially Ptolemy in his yellow robe who holds the terrestrial globe appears flattened to me. This effect is often observed in stereo photographs and is called “cardboarding” (View-Master!).
Because Raphael did not correct for linear perspective, our natural perspective leads to cardboarding.
Our vision is as common as it is strange.