Virtual Reality Engineering
Prof. M. Manivannan
Department of Multidisciplinary
Indian Institute of Technology, Madras
Welcome to this course on virtual reality engineering. This is a 12 week course; online course. This is an extended version of our summer course on Virtual Reality Engineering that we offered in 2015 summer. This was taught by Professor Steve Lavalle of UOI UC, University of Illinois Urbana Champaign. He was visiting here that summer and he offered this course. We will be using most of the lectures from this summer course taught by Professor Steve Lavelle himself, I will be adding more additional topics such as an advanced topics to make it as a 12 week course. There are 12 assignments in this course; roughly about 1 assignment every week. The assignments are either Computer Programming assignments or Numerical Problem solving assignments; Computer Programming assignments you will using unity or unreal engine, you do not need a extensive programming to learn unity or unreal engine the (Refer Time: 01:28) will help you.
Even if you donot have extensive programming background, by the way this is not a
programming course. This is a course in which Engineering fundamentals needed to design a better Virtual Reality System is emphasized. We believe that if you learn the engineering fundamental that goes into the virtual reality systems design. Then you will be a much better programmer than compared to those without these fundamentals. Additional to those assignments, there is a optional project that you have to submit at the end of the project. The project is the optional, but you are encouraged to take this project seriously as possible; doing this project will help you to learn the engineering fundamentals better. So, you all encouraged to take this seriously. At the end of this course we will select few excellent projects and we will invite those students who have done these excellent projects to the campus and offer them internship either in the lab or the company. There are very exciting options available coming up in India; if neither we will help the students to make a career out of Virtual Reality. There are 2 reference books suggested for this course. The 1st one is a book written by Steve Lavelle himself; this is an online book. The link to the online book is given in this course websites. The 2nd book is a common Computer Science book and Computer Graphics by Foley. The description of the book again, is given in the websites. Apart from these 2 reference books, I will be using additional references or reading materials from several sources. I will provide you well then and there whenever the topic is covered, you are encouraged to use your own references if you find it convenient for you to learn. There are 3 TA’s associated with this course; 1 TA has a background in Computer Science, the other in Electronics and another in Instrumentation. You are encouraged to interact with them as often as possible, specifically for solving assignment questions and doing projects.
In the next set of videos Steve, will be talking about what are the goals of this course; what to expect from this course. He will introduce couple of terminologies which you will be using throughout the course. While watching those videos, I strongly encourage you to keep one thing in mind; the entire course on Virtual Reality, imagine that it is standing on 2 pillars; 1 pillar is their Immersion and another pillar is Interaction. Virtual reality is made up of these 2 pillars. We will be talking about in the course, what is immersion and what is interaction? What are the different types of the interactions? The whole course is about how do we improve immersion, how do we improve interaction so that we can make a better virtual reality system. We will be covering the topic of immersion roughly to the half of the course; the other half will be dedicated to talking about the interactions. I hope you will enjoy this course; you are welcome to register this course.
Thank you, bye.
Prof. Steve Lavalle
Department of Multidisciplinary
Indian Institute of Technology, Madras
Lecture - 1-1
Goals and VR definitions
Let me now start to give a high level overview of what we are going to be covering and what the goals are for the course? So, I will start with the goals, there was going to be some kind of cloth for erasing, but there is not now; so, I am not sure what it was. So, I will use this; so, let me just write out the goals.
(Refer Slide Time: 00:40)
I will spend some time talking sometimes writing somewhat slower to write, but at least for some of the more important things I would like to write them down. So, some goals for this class; learn to build a good maybe great virtual reality experience. Now of course, this hinges on the definition of good; so, I will say a little bit about that in a bit, but let me state some more goals understand or at a university here our goal should be to understand.
So, understand how virtual reality works; learn to criticize VR; you should be very critical. I want to do my best to prepare you to criticize; don’t just put on a virtual reality headset and go wow this is the coolest thing I have ever seen, it may very well be perhaps, but learn to be critical. So, that we can all improve the experiences; so, that is one of the things I want to help train you on and finally, learn fundamentals to shape.
Or build the future of VR; so, in virtual reality the technology is evolving very quickly. So, if you want to keep on top of that, you have to read where it posts almost daily, look around throughout the internet, keep track of what is going on in the media, companies are raising to make consumer products and they are going to keep on improving very very rapidly. I want to focus on the fundamentals, I want to try to give you the concepts that should be more invariant with respect to time.
So, give you some basic truths and give you some basic concepts so that you can help; build the future of virtual reality and have some knowledge that hopefully does not become stale in 1 year. So, that is something I am focusing on now the lab experience you are going to have will be exactly using the latest technology. And some of those experiences may become absolute in a year, you may use different types of software tools, different types of headsets and things like that. So, the lab component in some sense is going to be time stamped to what we are able to do in the summer of 2015.
But I want these fundamentals to go into your heads as well and give you some perspective and understanding that will survive the test of time. And as many of these kinds of concepts in virtual reality have been around for decades, people have understood them for decades or understanding continues to improve. And I want to give you the snapshot of at least my understanding; as I consider both the intersection of the technology or the combination of the technology with these fundamentals in today’s world.
Refer Slide Time: 03:39)
So, as far as good; what do I mean by good, well I will say that it is comfortable plus. (Refer Slide Time: 03:46)
Adequate; I know adequate does not sound very demanding adequate for the task.
So, by comfortable I mean it should not make you nauseated, it should not give you headaches, fatigues things like that. So, I want to try to understand what makes an experience comfortable and it should be adequate or sufficient for whatever the task might be. So, what might the task be? Maybe the task is to develop an immersive first person shooter video game plan.
So, maybe that is the task; maybe the task is to give you the feeling of virtual travel; you may feel like you are in another part of the world, maybe live exploring into the part of the world. Maybe you want to maintain close relationships with your friends or your family while being far away; we try to do that with a phones or a video conferencing; Skype things like that. Maybe we can take that to the next level, maybe you simply want to relax, watch a film, a movie of some kind maybe the movies interactive maybe it is not.
So, these are the kinds of tasks we want to pick a task and then try to understand what kind of level of quality let us say or what types of things do we need to worry about for that tech for it to be adequate so that you succeed in the task. You may feel like you are virtually traveling and it works wonderfully and your that is a minimum baseline it’s comfortable.
I am going to now give a definition of virtual reality; I feel like it is important to give definitions from time to time. So, let me do that and then we will talk about this; I have seen quite a few definitions around very often they seem technology dependent, I am going to try to give one that looks more fundamental let us say and perhaps a bit surprising.
(Refer Slide Time: 05:41)
So, I will call as a definition of VR inducing; I have a lot of writing here to do apologize for that inducing targeted behavior in an organism.
Refer Slide Time: 06:00)
(Refer Slide Time: 06:22)
Using the word organism may or may not be a human. So, pay attention to that by using artificial here is where the technology comes in sensory stimulation.
Refer Slide Time: 06:30)
Might be visual, might be audio could involve touch, smell taste all kinds of things sensory stimulation.
(Refer Slide Time: 06:56)
While the organism has little or no awareness of the interference.
Refer Slide Time: 07:09)
(Refer Slide Time: 07:22)
How is that? So, in some sense; the organism is not consciously aware that its being fooled if consciousness is appropriate for that level of organism. So, you are not quite aware of being fooled at the moment; if you are a human you might remember putting on a virtual reality headset at some point, but while you are in the experience, it is very easy to forget that and believe that you are somewhere else.
So, that is the idea artificial; the artificial part is where the technology comes in; you are somehow taking over the senses. You are hijacking the ordinary sensory input that you get and replacing it with something that is been engineered and its fooling the organism. So, that is something that is very important to take into account and when I think of a VR system.
(Refer Slide Time: 08:14)
If I draw kind of a blah; then I call this a VR system, say VR system I like to think that its splits into two halves, there is the human part or I will say organism or generally to fit my definition. And then this engineered part; engineered hardware and software; so this is coming together, it is very easy for those of you who are trained as engineering and computer science students, to just think about this engineered.
However, it is very important to remember that this part is here as well the organism part is here as well. So, that when you try virtual reality you become part of the system; it is very easy to forget that. Something really fascinating that is happened lately is that; virtual realities become popular in neuroscience, biology or we could say among neurobiologists and perceptual psychologists.
Did you know that they use VR systems extensively in their research? The reason why they do that is; so, they can study organisms, they can present a controlled stimulus to an organism and see what the effects are. So, I have seen virtual reality applied to rodents; say laboratory rats, mice, gerbils I have seen it down to bugs, cockroaches; all the way down to even drosophila or fruit flies. They get them to run on little tiny spherical treadmills about 3 millimeters in diameter.
And they have using some chemical processes, they can essentially turn off some of their neurons and then present visual stimulation to them and get them to run in different ways through the visual stimulation and see the effect of changing their own neurons and study the effects of that which I really find fascinating. So, for them virtual reality is about as powerful as say an MRI machine to a doctor, it is some way to measure the human body.
And we work most of us here probably work in forward engineering; let us say. If you work in neuroscience and biology and perceptual psychology, its essentially working in reverse engineering. The system has already been built and you have to poke it and prod it and manipulate it and try to figure out; how it is working with a goal to understand.
So, I found that fascinating and that fits very well with this definition of virtual reality; they exactly see it like this and that is in fact, why they love this technology. And that field actually has quite a bit to contribute; I want to show you a few pictures related to that this; almost looks like my definition of virtual reality. Imagine a gerbil or a hamster; I will call it a gerbil; so, imagine a gerbil running on a wheel this is at least in terms of your motor skills.
(Refer Slide Time: 11:08)
And motor activity giving some kind of feedback so that the gerbil feels like it is running.
(Refer Slide Time: 11:17)
And humans can be fooled in the same way; I guess this ones actuated with a motor. I suppose if you had a low friction wheel you could get a human to run on that just fine as well to match the gerbil.
(Refer Slide Time: 11:27)
This is an example of a virtual reality applied to a cockroach. So, it runs on a foam ball that is suspended using air to make it nearly frictionless. And there is a wire there to study to do single unit recordings of its neurons and then it is presented with scenes visually on a screen and so, it will start running and exploring and you can control and manipulate it in some way. This was done at the University of Oulu in Finland; hope I had one more picture, but it seemed to have disappeared.
(Refer Slide Time: 12:04)
This is a picture of humans more or less doing the same thing as the cockroach. So, here they put on a virtual reality headset, but you could also have the surround screen; if you like and they are running around in some kind of omnidirectional treadmill dish; which was produced by a startup company recently.
So, it is very interesting to me to see humans and let us say cockroaches and other lower life forms fooled in the same way; so, that is just something to pay attention to. So, there is one very interesting distinction which is that; for a scientist they have a very clear distinction between them and the subject; there is the scientist and let us say the laboratory rat.
Now, when you are the developer of virtual reality systems; you become both the scientist and the laboratory rat, if you are not careful. So, you are doing the experiment on yourself while continuing to develop it so; that means, you are giving yourself inappropriate feedback because you are fooling your brain while developing it; eventually you fool yourself into thinking you have made a wonderful virtual reality experience. But you are strongly biased because you might like what you have created.
So, it is very very important to get feedback from other people especially from subjects who have maybe never been exposed to virtual reality. Especially ones who have not been playing a first person shooter games since they were 6 years old or something, so it is very important to get feedback from other people and and also to train yourself to try to look for flaws in the system.
And I will get into some of the details of that, but that is a something very different about let us say neuroscience applications of VR, very clear distinction between the scientists and the subject a VR developer is both at the same time; they flip back and forth without even being aware of it; so, I think that is an interesting thing that happens. (Refer Slide Time: 14:16)
So, other terms that I want to talk about; we often use words like immersion the feeling that you are surrounded by the sensory stimulation.
So, you may feel immersed by wearing earphones and you have an immersive audio experience; you may go to an Imax theater or some kind of panoramic movie theater and you feel immersed in that sense. So, immersion a stronger word is presence where people like to say informally that your brain has been tricked and you feel like you were somewhere else completely and tend to forget about the fact that you are not.
And like when I first heard that I thought that was; I do not know kind of a fuzzy concept
I was not so, sure what I could do with that it does not sound like an engineering kind of concept to me; all your brain is fooled. Well, I recently learned that this can be established scientifically; there are something called play cells in your brains and in the brains of rodents. where this particular neurons that fire whenever you are in a particular place.
(Refer Slide Time: 15:17)
So, they have put for example, mice in a maze this is a simulated picture of a maze here. And they have noticed that by doing what are called single unit recordings just by studying the neural activity in particular places they have they have observed that when the mouse is in a certain location, a single neuron will start firing that corresponds to a code for that place.
And they have done virtual reality experiments where they have shown that these play cells are learned even for virtual environments. So, I think that is a pretty concrete definition of fooling your brain, your brain will start to learn three dimensional environments; places inside that you have never really been to. They may be real places that you have not actually been to or they may be virtual places that are completely fictitious, but your brain is fooled by that.
So, I think that is pretty reasonable concept presence; there is another interesting concept which is and; this one is used by the neuroscientist. There is open loop versus closed loop; those of you have control theory background may recognize this distinction. So, in the open loop sense, the stimulation is just timed though the sensory stimulation provided this artificial sensory stimulation provided; is simply times, it does not matter what you do.
So, if you are sitting in a panoramic theater watching; I do not know maybe your favorite Rajinikanth movie; let us say, everything is just coming right along. You are not really participating in the action, you might kind of feel like you are there it might be immersive. You may feel even a sense of presence, but its open loop; if you actually get to interact maybe you get to fight with the bad guys and actually change the outcome; then this would be closed loop.
And scientists make a big deal out of this because the organism needs to interact with the environment in most of the simulations that they do so. I attended a workshop at Ludwig Maximilian University in Munich of about two or three weeks ago and I gave a keynote there about virtual reality, I learned a lot about the science of neurobiology and virtual reality applied to that.
There were about a half a dozen labs working on these kinds of things and part of the title of the workshop included the phrase closed loop. So, it was close loop virtual reality which I thought was very nice; some more words and then we will move on to historical perspective.
(Refer Slide Time: 17:43)
So, telepresence or teleoperation; I have worked for a couple of decades as a robotics researcher. So, I cannot help imparting some robotics perspective on this as well; so, imagine I am having cameras connected to a robot could be a flying robot, a swimming robot, a rolling robot, a humanoid it does not matter really the kind of robot, but you can use virtual reality interfaces to feel like you are somewhere else.
So, the reality itself; it may be real reality, but you are simply not there, you have been transported. So, that is just something to pay attention to the experience you have may correspond to something quite physical and quite real and it is been captured by cameras in some way maybe; 3D cameras, may be 2D cameras and its fooled your brain and that maybe just quite fine and it does not have to be a perfect cartoon like world that is been created like in a three dimensional video game.
So, it can be either kind and; so, it is important to consider these kinds of applications; tele-operation being an important case of that as well where you interact with that world and issues of latency or delays become very important.
There are lot of interesting applications from aerospace to medicine and telesurgery and things of that nature; a lot of interesting research going on here and that subject for example. So, that is something we keep in mind; I think one thing to think about as well in terms of this definition that I gave.
(Refer Slide Time: 19:15)
Is think about; what does VR include, I skipped over one word that I meant to put here; let us say augmented reality augmented reality.
(Refer Slide Time: 19:30)
I will consider that as a kind of special case; a very important kind of special case of virtual reality; so it fits well within the big picture. In this case, very similar to telepresence and teleoperation; the actual physical world is being strongly emphasized in fact in augmented reality, your brain mostly believes that you in the same place where you have always been standing; everything is fine.
However, some additional information is being superimposed into the world. And if you take it in the limiting case maybe it is superimpose so, much information that you cannot remember the world where you are at anymore. And then you get you feel like you have been completely teleported, so these are very into a virtual world. So, these are very closely related; so I wanted to put those things together. So, I would include that in virtual reality what else does not include?
Let me put some may be absurd sounding things; let us see. Well let us just try watching a movie; I do not know would that be virtual reality? No; no answers here, no; yes, I do not know; there is no clear answers here. I do not know just something to think of about. So, I want to give you something to think about; video conferencing; what about listening to music? You put on your earphones, talking on the phone.
(Refer Slide Time: 21:11)
Talking on the phone really? Have you ever been immersed in a conversation with someone? And then maybe fall into a pit or something as you are not paying attention to what is going on around you.
So, we could say just playing a video game; this leads me to the next comment which is very important; which is think about the difference between being first person versus third person. Do you feel like everything is happening to you all around you? Or do you feel like you are watching it happen; yes.
Student: Sir, in many of your examples is more about us ignoring sensory information to excluding sensory information to just focus on one of the senses.
Student: As suppose to be fooled by false sensory information that is coming to us; I mean I am listening to music and then as well shut out for rest of the world.
Student: So, it is not that I am getting a false image that is coming to me; I mean so.
I am not sure, well that is very interesting; so in terms of the audio sense. So, one thing I should point out is that our visual sense is the most powerful the most complex that we have; the largest fraction of our brain is devoted to that vastly compared to the other senses. So, I really think it is quite equivalent when you just put on earphones and you listen to music, you have this sense that there is a band either inside of your head or all around you; depending on how well it is been recorded and reproduced and that gives you a feeling of presence in the music somehow, but I do not know.
Student: (Refer Time: 23:01)
I do not think these are; I am trying to intentionally pick some things that are on the boundary to get people to think of it. But I am not going to go to the sort of a crazy obsessive level of just saying; by the time this course is done you believe that everything is virtual reality, including your everyday life all the time; so, I am not going to make the;
Student: That is Hindu philosophy.
Perfect; here is a good one, so this one I will finish with because it is probably maybe the craziest reading a book. If you are reading a really good story, do you feel present somewhere else? You completely lose yourself; you forget your surroundings. Wow! How can it be happening? So, that kind of virtual reality has been around for quite a long time; any other questions or comments?
Student: So, you solving a problem also become as a virtual reality. So, I just tried to solve a problem. So, I close everything and I am just focused on that problem; so does that constitute a virtual reality or (Refer Time: 24:05)
Very interesting; so, maybe a really difficult math problem, I am sitting down writing and I have blocked out everything else, I am doing some kind of interaction with a pencil and paper; very interesting, I do not know that seems like another level yet. So, if you are sitting and drawing nice diagrams and you are imagining you are in some other place where the diagrams are that may work.
If it is simply mathematical equations; I do not know, it is definitely a fuzzy place where I am afraid to answer; I am not sure. I feel like in the case of reading the book, I start to imagine a scene; I imagine some other people’s lives, I feel like I am there is somehow. I may be in Victorian England or something all of a sudden you know reading Charles
Dickens and I feel like I am there; it is a little different than what I get from solving math puzzles. But there is that that sense of blocking out all the other sensory stimulation, which is attention and you know focusing on one thing.
Student: That comes under that immersion that;
I think that is right. So, I think definitely this; let us say selective attention is an important part of it here, but I feel like there is also something else like the kind of belief that you are immersed in some other culture place people’s lives something that is another part of it.
Prof. Steve Lavalle
Department of Multidisciplinary
Indian Institute of Technology, Madras
Lecture - 1-2
All right, I want to give some historical perspective. So, I guess by going by going on these kinds of questions, I am not sure where to put the boundary of the beginnings of virtual reality.
(Refer Slide Time: 00:20)
Here is the oldest known cave painting it is about 40,000 years old in Spain el Castillo, they did hand stencils, I really have a hard time believing that was done 40,000 years ago, it looks like maybe some kids were playing, but they have done I guess the appropriate carbon dating and such. So, is that virtual reality, I am not really sure, but it is trying to communicate something.
(Refer Slide Time: 00:55)
This one is from ancient India, all right. So, this is about 30,000 years old, Bhim Betka all right at least in this case you have some kind of display it looks like I see people on horses looks like some kind of battle those always seem important to record, we do the same thing recording, I do not know world war 2 in a textbook all right or on Wikipedia.
So, 30,000 years ago people were making experiences like this, I view this I imagine what the scene is like it is kind of like virtual reality, isn’t it? people for a very long time got obsessed with rectangles right. So, make a rectangular drawing and you feel like you are there you somehow it captures your imagination. So, one thing I like to say is that people have been staring at rectangles for a long time, now all right maybe a thousand years or more of staring at rectangles it goes back to paintings, and then evolves into evolves or devolves into movie theatres, television sets, computer screens, Smartphone screens, we have staring at rectangles for a long time as a way to present visual information, and give you the feeling of being somewhere else perhaps.
This particular picture is from the European Middle Ages.
(Refer Slide Time: 02:08)
It is around or maybe early renaissance it is from the year 1470, and I like this one because, I was particularly looking for pictures where the perspective is wrong right. So, this is a very funny kind of picture because, if you look this, this wall does not shrink appropriately as it goes backwards, these soldiers back here they just look kind of small rather than in the background. So, somehow it is not been drawn well, but I suppose that looked fine in it is time, and nobody had any idea how to make it look better right, and then not too much later around 1600, picked paintings like this started showing up. (Refer Slide Time: 02:48)
And this one’s correctly taking into account perspective, and the lines are drawn you know very, very nicely to where if I stand up very close to this picture, I start to feel immersed all right this really feels amazing to me. So, that is wow I could draw a threshold here and say this looks like virtual reality that is pretty amazing, I really get the 3-dimensional information. So, that is so, that is quite impressive and an important step oh yeah, I want to then go to film a bit and show you a couple of things, which were considered in their time to be very immersive. So, let us see.
(Refer Slide Time: 03:26)
You might have seen this before the train arriving in the station.
(Refer Slide Time: 03:31)
So, 1895; so, they showed this in a movie theatre in the 1890’s.
(Refer Slide Time: 03:42)
"Nós enviaremos as instruções para resetar a sua senha para o endereço associado. Por favor, digite o seu e-mail atual."