Virtual Reality Engineering
Prof. Steve Lavalle
Department of Multidisciplinary
Indian Institute of Technology, Madras
Lecture – 11-2
Human Vision (frame rates and displays)
(Refer Slide Time: 00:17)
Which brings me into the next topic; which again this is still under perception of motion and it is stroboscopic, or scopic, apparent motion; So, again let us think about frames per second, or FPS. (Refer Slide Time: 00:22)
Now, it would be very nice to just talk about frames per second, as if I am just showing you one image, and then a second image, and then another image, and it just quickly flips through like it that is how you could do it if you draw images perhaps on paper and go flipping through them, although of course, it is not exactly right because the pages are flipping down.
And there is some partial obstructions in between and the same kind of thing is true in digital systems the pixels do not quite switch at 0 time, you know, they do not switch instantaneously all the pixels are not necessarily changing synchronously. There are a lot of strange artifacts that are going on and I want to talk about how that interfaces with your vision system and brain in some way. One kind of general thing to think about with regard to showing you a bunch of frames is there is pulsing versus being always on.
So, draw a kind of picture of what I mean here to the pulsing case versus. So, it may be the case that in the pulsing and if I were just flipping pictures to you, then here is what here is what this would mean, inside of a very small interval (another color chalk) inside of a very small interval; so, maybe this is 1 millisecond, I just flash an image at you ok. I just flashing at me for 1 millisecond, and the rest of the time it goes blank.
So, this entire time in here, which could be as long as maybe 99 milliseconds up to 100 milliseconds or so the photoreceptors may remember let us say the signal um. So, this is blank completely blank time. So, if I just give you quick pulses like, about a millisecond that is enough to charge up let us say your photoreceptors. So, that an image will be perceived if I turned off again as long as this image is bright enough over that short period of time, it will be a complete picture will be stored, let us say, if I do this setup on a pixel by pixel basis or I draw an entire continuous image to you and flash it to you very fast for a millisecond that is enough for you to receive the stimulus.
So, one possibility is to do that the other possibility is each one of these time periods, here I am trying to show you some kind of frame rate, let us say, right maybe it is 10 frames a second in which case the time from pulse to pulse would be a 100 milliseconds. So, every 100 milliseconds here I might just change the picture, all right.
So, one possibility is to show you a picture for a 100 milliseconds, and instantaneously switch to another that is always on the other one, of this it is I can just pulse in and blink. I am showing you just pulse in and blank case, because our brains our vision system is capable of handling this.
And we have been doing this for a very long time. And one of the motivations was for cathode ray tube displaced, old television displays they ended up pulsing the images on for a very short amount of time, based on the amount of time that the phosphor remained on the elements across the screen remain on, and they were dark most of the rest of time. But your brain does not perceive the dark times, because it is capturing enough from the pulse.
So, that is one thing to keep in mind, and now I want to talk about various frame rates; give a kind of historical perspective as well.
(Refer Slide Time: 04:56)
So, let me write over here how many frames per second, frames per second, do we need? And it depends on what you want. So, in terms of hertz if we go as low as 2 hertz; we end up with the beginning of perceived motion. In other words, if I just show you 2 different images I have maybe a ball it is in one location and one image in another location in another and I just oscillate back and forth at 2 hertz, that is already fast enough so that it will appear to you like the ball is moving back and forth. It might not be a very smooth or continuous motion, but you at least perceive some motion in that.
Um if you get up to about 10 hertz, then you no longer perceive frames individually. One of the most famous videos of this this is considered by many to be the oldest movie. This is by eadweard muybridges, 2 horse video from 1878. So, there are I believe this was captured at 25 frames a second, but they could play it back at various speeds. And they they found that people cannot perceive individual images anymore once it is up to about 10 frames a second.
(Refer Slide Time: 06:28)
They actually had a horse running across a field with a bunch of cameras a sequence of cameras set up in the field and the horse triggered the cameras by tripwire so that they could all take pictures and they could assemble this rumor has it there was a bet about whether or not all 4 of the horses feet would be off the ground at any time, which is a very hard thing to see without a high-speed camera. And I think there is a frame and there were all the feet are off the ground. So, this looks like motion right.
I am not sure how many frames a second this particular one, and I believe this is about 10 frames a second right now that we are looking at. So, it is hard to perceive individual frames, but it is right on the threshold. So, you may be able to. So, if you go up a little higher, 16 you have early silent films, right? Old silent movies or old home movies, I would say in the 1960's and 70’s people are buying video cameras and making home movies with, and showing them on projectors, and in that case about 16 frames a second video.
Let us see here, what is interesting about these old-time films is that this is musti suffer from 1916, it is about 16 frames a second, but what is interesting is they hand tuned the rate of playing the frames. So, it seems is it uncomfortable to us does it make us nauseated. I would say no, but you can tell of course, that the motions are not realistic are captured exactly right in terms of what they would be in the real world, but it works somehow. And in fact, this is a form of comedy at that time, and it is perfectly acceptable to us.
So, I guess one of the reasons why I want to show you this is that, there is perception of motion the frame rate is actually variable here and was hand tuned, this is before they matched to the audio to the various. So, they could do anything they wanted to it then they didn’t have to worry about the audio getting out of sync, and it gave enormous opportunity to make entertainment.
So, I think virtual realities in a very similar place, and that you do not have to get all of the aspects perfect. You should just make sure people are not getting sick. And if there is something you do to distort the video in some way or whatever it is you are presenting in some way, if it is more entertaining or more appropriate for the application that you are trying to solve go ahead and do it.
So, there is a lot of room for expert for exploration, here you do not have to make everything completely realistic. So, I thought it was very interesting based on the technology of the time, and the fact that there was no audio track. They had the liberty to vary the frame rate, and nobody seemed to complain it was all just fine thank god let that was a great.
Eventually the motion picture industry standard got up to 24 frames a second that is where it mainly is today. And 24 frames a second if you show these frames on a standard projector, at 24 frames a second you will see flicker, right? If I just have these frames pulsing on and then going black for the rest of the time, or even in this case, if the frame is on and then quickly changes to another, you will perceive some kind of flickering. So, people started making multi blade projectors that will show the same frame multiple times.
So, you can then turn 24 into 48 with a 2-blade projector to avoid or let us say reduce flicker. Really it is reducing perceived flicker I guess technically there is twice as much flickering happening, but it is reducing perceived flicker, also 48 is a new standard that is been proposed for movies. James Cameron and other movie producers are are are pushing for higher frame rates there is a backlash between some between some movie fans and technologists as to whether it is whether they really want to have that level of temporal resolution at the movies. Or is it just you know do they prefer the way it is always been for very fast-moving objects that come very close to your field of view, they look quite different at 48 frames a second if you are actually drawing 48 different frames rather than just using a 2-blade projector. So, people are debating that right now whether we really need 48 frames a second.
If you get up to 60 hertz, that is the most common rate for PC's and smart phones. Guess I skipped over classical television here. What was pal at pal television 25 frames a second, I think, right was it and an NTSC, and in the u s we used NTSC which is 30 hertz. So, I think it is 25 was palin I see first armanis, nodding right some of the older folks know alright so, that is very close.
So, just more than it needed to be for the motion picture industry standard; At some point when they started making big computer monitors, and people started sitting close to them. That became a greater perception of flicker again, and they had to increase the frame rates even more. This is very interesting because it is related to what is happening now.
(Refer Slide Time: 12:24)
In virtual reality so, at some point when we had big cathode ray tube monitors, people are sitting close to them, the flicker they perceive at the periphery ended up causing a problem. So, it became the case that at 60 hertz, we had a noticeable flicker at 72 hertz. It was the minimum ergonomic recommendation. Personally, I still would get some kind of headaches from 72 hertz.
So, there may be thresholds one at which you can see the flicker definitely see it there may be, another threshold where you do not directly perceive it, but it may still have some long-term effects of you sitting in front of it for 20 minutes you may get a headache or fatigued more. So, that is something very careful and important to pay attention to here this happens over and over again in virtual reality and over and over in human perception, there are different levels.
One of them is where you are consciously aware of the problem and another level may be where you think you have fixed it, but it causes that some subconscious level some trouble later on it may fatigue you um. And and I talked about the we organs changing for example. And that may be something that is at a subconscious level is fixed, and you are not aware of that, but it may cause you fatigue later on some other kind of discomfort.
So, eventually if you get up to 85 hertz, this was deemed to be comfortable for everyone. And so, very often monitor timings were up in the 80's for the refresh rate on CRT displays. Once we went to LCD monitors, the display rates went back to the frame rates went back to 60 hertz; because a flicker was not a problem because the pixels are always on. So, then the flicker is not as much of an issue anymore, there is not really a perceived flicker, but when we had flicker on CRT monitors the rates got pushed up like this.
So, this continues onward on our quest we go down for higher and higher frame rates. So, I had 72 85; which I write there currently you have seen a lot of people targeting 90 for consumer virtual reality headsets. I would like to explain a little bit of why people are saying that and there is another threshold which I guess I will put a thousand here just to really complete it all. And this is where I can still perceive flicker, there is a certain experiment you can do which is very interesting, which is you know grab your household function generator waveform generator.
So, you do not have one in your house probably, but go to an electrical engineering lab, and generate a pulse wave at a thousand hertz and then connect it up to an led, right put a resistor in there or something and then move it back and forth like this.
So, fast that your I can not track it, because otherwise it will try to keep the same image on the retina as you move around. So, move it very fast like this and what you will see is a kind of zipper pattern appearing through space turn. Off the lights and just move a red LED back and forth for example, and if you do that even at a thousand hertz you can perceive the flickering of the LED. And the reason why is because it is going to generate multiple images across your retina.
Even moving at that fast speed because your eye can not keep up and track it; So, this implies I think that even if you had a very fast display going at 100s of hertz, if you move a virtual object to it very fast, you will still perceive it as being a flickering object right seems unavoidable, unless you go perhaps well beyond a thousand hertz.
I think people can see it even at 2 or 3000 hertz before this effect stops. It all depends on how fast you are moving the object right it has to generate enough images across the retina. If anyone has anyone seen that phenomenon before? The very simple experiment to do if you have an electrical engineering lab around you a very simple way for a generator and an LED circuit. So, just move it back and forth.
So, that is something to think about it is even in spite of this whole progression, I can still design a scenario, where if I have a very sharp bright spot let us say very small bright spot, and I want to move it very quickly through VR even at a very high frame rate, I may perceive some kind of flickering. Because my eyes cannot track that using smooth pursuit.
But what is going on here why is it that 60 hertz is not enough people are making a big deal out of 90 hertz right now. The headsets that you have in the lab are at 75 hertz and that is interesting, because that is very close to the minimum ergonomic recommendations that people had for CRT monitors in the 1990’s. And 90 hertz is very close to the comfort for all let us say thresholds for CRT monitors from the 90’s.
So, these things are very closely related and I will explain why in just a bit, questions?
Virtual Reality Engineering
Prof. Steve Laval
Department of Multidisciplinary
Indian Institute of Technology, Madras
Human Vision (frame rates and displays)
Welcome back. Continuing onward in the previous lecture, we were talking about perception of motion.
(Refer Slide Time: 00:25)
And we looked at a various frame rates, all the way down to as low as 2 hertz where you have the beginning of perceived motion this is stroboscopic apparent motion. We get up to 16, 24 frames per second motion picture industry standard also old CRT based television signals were on 25 or 30 hertz frames per second.
(Refer Slide Time: 00:50)
And Eventually I talked about a CRT monitors, and how as monitors got larger there became noticeable flicker as people sit close to them because of the flicker perceived in the peripheral vision. And they became minimum levels beyond which people tend not to directly perceive flicker, but then it may still cause some discomfort headaches and fatigue, until you get up beyond a certain level. So, very often people quote 85 hertz as being sufficient for just about everyone.
So, I want to talk a little bit more about problems with displays, and our perceptions of motion and stationarity, which is the absence of motion. We talked about how the brain perceives motion earlier, before this topic in the last lecture.
(Refer Slide Time: 01:41)
So, one of the problems we have with displays is display scan out, which is that the image that is produced on a monitor this goes all the way back to CRT monitors, but it still happens on most nearly all of the screens that we use, today is that the image is being scanned out in some kind of order line by line, it does not have to be in the same order that it was back in the days when CRT monitors were prevalent. But it is some kind of progression, that goes line by line goes pixel by pixel and then as it is scanning out the pixels start switching.
So, one of the effects of that, is that, if I am looking at some object that is supposed to be stationary in VR, and I turn then because of the time it takes for the scan to go down. So, if you are at 60 hertz for example, then it may take a 15 milliseconds to complete the scan; 16.67 millisecond the time from frame to frame at 60 hertz right. So, it is one divided by 60 um, one second divided by 60. So, there is a blank time as well so, but it is about 15 milliseconds. And so, if you do that and you turn your head back and forth then this causes a waggle effect.
(Refer Slide Time: 03:12)
In other words, some object that perhaps it is supposed to look vertical, but when you turn your head back and forth when you have the scan out problem may look like this, alright. So, as you turn your head back and forth, this object may seem to waggle side to side in some way, and undergo some kind of distortion.
So, that is one problem that we face and another problem that we face is even if you were to update all these pixels going in this progressive pattern, the pixels themselves could take up to 20 milliseconds to switch, 20 milliseconds is a long time, right? By that time, you are ready for the next frame. How long does it take to switch? It depends on what particular kind of display you have. If you have an LCD display it could take up to 20 milliseconds if you have an OLED display, it may take only about 80 microseconds, it could be much faster.
So, one of the ways to help alleviate these kinds of problems, is to use old LEDs instead of LCDs because you get faster pixel switching. But that may then lead to another problem, which I will explain in just a bit. The waggle effect has to do it to scan out time, if I have if it takes a long time to update the pixels another thing that will happen is blurring, it will look like this object is smearing in some way because the pixels cannot change faster. So, I am turning my head back and forth, trying to use the vestibular ocular reflex, in addition to the waggle effect there will be a blurring or smearing.
This was very obvious in the oculus rift DK-1 for example, if any of you had a chance to see which is using LCD display and you see very obvious blurring. You see other virtual reality headsets with LCD displays; you may notice a kind of blurring like that. So, you can switch to another kind of display to try to overcome the blurring problem.
We still have the display scan out problem. I am sure people in industry are working on this as we speak to try to make complete synchronous scan out and updating of pixels. So, that you get one frame altogether under the assumption that it indeed is necessary. Well, otherwise you have to show that it is not through enough human subject studies.
(Refer Slide Time: 05:49)
So, perception of stationarity and smooth motion in VR in a VR headset VR a head mounted display in particular I want to talk about. So, if I fixate on a point in the world and that point is not moving, and I turn my head back and forth, and I use my vestibule ocular reflex. That point should be image in the same place on my retina, right. If I just pick one I should be able to the same place on the retina. So, when I do that experiment of let us say I turned my head in one sweep, then the position on retina of that point over time should look constant. That is what should happen in the real world.
So, position on retina some fixed point should look constant. Now, what happens if I have a display in front of my eye, and I do the same thing and I want to have that object that feature appear to be in the same place in the world. Well, I hope that my head tracking is working correctly, and let us assume that that is working perfectly, and let us think about other kinds of problems. If it is not working perfectly, if there is some latency being introduced or some other kinds of errors there is can be additional artefacts on top of what I am telling you today.
So, let us think about what happens when looking at a head mount to display. Before we get to that, in the process of getting to that let me just think about some different cases here. Let us think about displays and I want to look at four different cases, at four different cases here. (Refer Slide Time: 08:05)
Here I want to have a world fixed screen, in this row, I am going to look at two cases of a world fixed screen and down here I want to have a head mounted display in other words a head fixed screen.
(Refer Slide Time: 08:26)
And then on this side, I want to consider the case of pursuit, in other words, that is the kind of eye movement that is going to be happening.
(Refer Slide Time: 08:36)
And in this case I want to consider vestibule ocular reflex. So, let us think about pixels on a display. So, I am up here, I am looking at a screen. So, I have some pixels, I have an object that starts moving. Of course, I have not drawn that right, because I should be drawing a blocky version of this, correct? So, I am just trying to illustrate things here. So, I did not draw a blocky version try to make it simple, but it should be blocky and in the case of pursuits. So, suppose I am sitting at the movie theatre, that is a fixed screen and an object starts moving through there, right. So, the pixels will have to be changing, and this is the place where people are arguing that maybe modern movies should be 48 frames a second instead of 24.
So, that these motions become smoother, and more natural like they are in the real world. Alright, So, we have been accustomed to watching movies at 24 frames a second. So, after some very fast motion that comes across the screen it might not look right. And with cinematographers videographer’s people who make produce movies know this, and they be sure they take care to avoid these kinds of artefacts and problems whenever they are constructing movies.
Now, if I am over here, and you are sitting at the movies, and again I guess maybe sitting at the movies is not the right comparison, because in that case I might imagine there is some kind of analogue process going on through the project. And it may you know it is hard to see pixels it depends, I mean, if I project a digital screen, I certainly see pixels on the screen if everything is clearly in focus. But because of the focus part maybe it might look like it does some kind of a conversion to analog if you like so.
So, I can also just say looking at a fixed screen like an LCD monitor like we have in these rooms right. So, if that is the case imagine I am going to have my fixed monitor let us say, suppose the monitor has some object on it. And it stays in one place, I decide to look at the monitor, and I go back and forth looking at the monitor move my head back and forth using my vestibule ocular reflex.
Do the pixels need to change while I am doing that are the pixels are not changing. So, this is very similar to the way a stationary object would appear in the real world, except for the fact that I have quantization due to pixels, but it does not have a problem with re-guard to changes over time.
So, anything has to do with how the pixel gets scanned out how the pixels get scanned out from frame to frame, the frames per second does not seem to matter too much here. So, if I am just sitting looking at a picture on the monitor, I fixate on some feature on the monitor, I move my head back and forth looking at say a particular icon on the desktop it looks stationary in a very natural way. Now, I have a head mounted display, and I hold my head fixed let us say. So, I am holding my head fixed, I am looking through my head mounted display. I see pixels, and the object starts to move, pixels are changing, right?
So, again the same kinds of issues of frames per second are going to matter as they would at the movie theatre. And so, so this is a very similar kind of situation if I decide not to move my head, if I move my head, then I have additional problems of there is going to be even more motion of this object well or less depending on how they combine, but if I am moving my head and the objects moving, they are certainly going to be pixels to change, right? But just the object moving by itself is enough, if my heads moving in addition then it is some kind of combination of those two motions; that are going to cause pixels to be changing as this objects represented.
Now, the 4th case is the most unusual and interesting, and a is very important to pay attention to, which is that we said if we sit and look at a screen, and we are looking at some stationary object it does not matter, how the pixels are switching or the frame rate. But if I want to use my VOR, I have some object that is supposed to be fixed in the real world, sorry, it is supposed to be fixed in the virtual world, right? In the in this world frame, if I turn my head back and forth while wearing a head mount to display, does this object is this is object supposed to stay on the same pixels or not should not right.
So, the image I sort of have or the feeling I have is like this it is almost as if when I put on a head mounted display imagine there were grid lines drawn on it, but it is like a piece of glass, and I see through the glass to the outside world. That is hard you cannot keep them both in focus the grid lines on this piece of glass and the outside world, but suppose you can forget the focus problem.
If I move my head back and forth, then I would see a grid pattern sliding back and forth with respect to all of you right. And so, that is exactly what is going on here, these are the pixels that have to be switching you know as I see you sliding back and forth across my grid or the grid sliding back and forth across you it is it does not matter which way you look at it, this corresponds to the part that has to change.
So, if these pixels are not changing fast enough, then how does that affect my perception of stationarity. So, let us look at a case where we have a very fast switching display I have the same picture.
(Refer Slide Time: 14:16)
Here I want to look at a particular object just call it a point if you like do not worry about maybe it is so small that it is like one pixel wide. But it can be a few pixels wide we fixate on one particular point and let us suppose that maybe we have a display that updates at 60 hertz. I am going to forget about the waggle problem, let us just suppose you know I update all the pixels at the same time, they switch very, very fast, but they only update it at 60 frames a second right. So, I go back and forth like this, what is going to happen to the image on the retina?
So, what you should instead get is a picture like this. Where if you are at 60 hertz just as an example let us say 60 hertz, then this amount of time here, this would be 16.67 milliseconds, does that make sense why that is happening? Because if the image is frozen in my heads turning it is not supposed to stay the same image for 16.67 milliseconds so, your brain will perceive that object as moving a little bit.
So, that will cause an object that is supposed to be stationary, seem like it is going back and forth like this, it will gradually move across and jump in, and gradually move across, and jump gradually move across, and jump that is what is happening on your retina, when it is supposed to be stationary.
Terrible problem, right? So, one name for this kind of oscillation is judder, judder kind of a fairly generic term used for all sorts of video flaws and things. But this is sometimes considered one example of judder. Now remember I said that you could flash the screen on for as little as one millisecond, and that will be enough to trigger your photoreceptors. So, that is what I could do here, and that should help with this judder problem. So, I could, all right, this is the always on case and this is going to be the other case, which I called what will see what do they call the other case pulse? So, pulse case.
So, in the pulse case it will look something like this. I will get a picture like this, where during all of these intermediate times between these small diagonal line segments, the screen is black let us say completely off. So, I pulse it on for one millisecond. So, this I will make this little thickness here be one millisecond, right? Not rendered too well here, but.
So, it is on for one millisecond, and in the remaining us 15 and something milliseconds the screen is completely off. So, I could do that, and then when I move my head back and forth the amount of judder the amount that this object appears to be sliding is significantly less.
If I try to make it too much less than a millisecond, it might not fire long enough to fire my rota to trigger my photoreceptors. Now I have another problem, if it is off most of the time, and it is just pulsing these bright lights, I see I perceive flicker. So, you have solved one problem, and I have another problem.
So, because of flicker, we have to go back to the same scenario that that we had back with giant CRT monitors in the 1990’s and we have to get the frame rate up higher. Now we could try to use that trick of multi blade projectors, and just show more frames here, but if we do that the objects still not going to be in the right place.