Loading
Note di Apprendimento
Study Reminders
Support
Text Version

Recommendation Engine

Set your study reminders

We will email you at these times to remind you to study.
  • Monday

    -

    7am

    +

    Tuesday

    -

    7am

    +

    Wednesday

    -

    7am

    +

    Thursday

    -

    7am

    +

    Friday

    -

    7am

    +

    Saturday

    -

    7am

    +

    Sunday

    -

    7am

    +

Hello everybody, welcome to Marketing Analytics course.This is Dr. Swagato Chatterjee from VGSOM, IIT Kharagpur who is taking this course foryou.We are studying week 7, session 1 and in week 7, we will discuss about Recommendation Engineand Retail Analytics.So this is our topic.So majorly, we have, in the next couple of weeks we will talk about what kind of problemsthat we face in retail and we, I will at least for the first two weeks means week 7 and week8, I will majorly focus on e-commerce.Now why e-commerce?E-commerce is something that is coming up in the whole world actually in a quite a bigway.So Internet is available with pretty much many people and let us say this Jio is givingthis much offer and the other like Vodafone, Airtel all are giving various offers in Internet.Internet is becoming cheap.At our time when let us say around 10 years back when we were youngsters our the Internetwas not very much available and the cost of having Internet in your mobile phone was veryhigh but now Internet has become cheap and people who are in rural areas also can accessInternet because availability of internet is also there means the connection is alsogood.Giving that as the background there are lots of e-commerce firms that has come up, so Flipkart,Amazon are there in India.Probably other in the B2B sector also there are Alibaba and let us say there is Blackbuckand there are various kinds of aggregator firms that is coming up and then also thereare aggregator firms who works as a platform, for example, they are actually connectingthe customer requirements and the, I would say the service provider firms, they are creatinga platform where these two people can come together.So those kind of platforms are also coming up and they are giving services in urban areas,semi urban areas, rural areas.So in short, e-commerce is becoming an important aspect of retail business.Still in India, I would say that majorly Indian retail is unorganized, probably majorly thewhole world's retail is unorganized.A very small part of that is organized and again very smaller part of that is e-commercebut the growth rate of e-commerce is pretty high.So though it is still in terms of size probably not significant if we compare with the totalretail industry, the growth rate is something which is very much higher for e-commerce firms.Given that as a background we have to focus on how Marketing Analytics can help retailfirms as well and to be specific e-commerce firms because that is where the applicationis pretty high.Now why a recommendation engine, is something that I will discuss now.So here you will see in this particular slide, you will see that there is something calledyou have, you can sell deep.Sell deep means getting the first initial sells is pretty easy but monetize the longtail with surprising recommendations is something that is this particular picture is saying.It is saying that up to that, up to this line this is easy to get, no issues.But after that whatever sells happen in this area which is the minute sells or which isthe incremental sells but that is a long tail is there, means you can do incremental sells,incremental sells slowly slowly as much as possible and as that incrementation can goup to infinite you can actually sell, upsell or cross-sell little little products.When you do it in a large scale, that contributes to a revenue in a very huge matter.So if I let us say all these recommendation engine and etcetera is incremental sells.For example, if I go to an e-commerce firm in a website, I have I will have somethingin my mind to buy.Let us say I want to buy a mobile phone.I have, I will probably do all these, I have already talked with my friends, I have talkedwith other people who has idea about mobile phones, I have searched in websites and inonline media and etcetera and then I have created a consideration set.Now with that consideration set I have let us say probably I have done a little bit moreanalysis and created a choice set.Now I want to make a choice.So when I go to the e-commerce firm's website, I want to buy something with very high probability,whatever I have planned for I will buy that.But if some recommendations come, I might probably switch from whatever I wanted tobuy to whatever these guys are recommending.So little bit or probably I will buy an extra item.For example, let us say somebody you are buying a mobile phone.They give a recommendation for cover, supportive cover for the mobile phone or protective coverfor that mobile phone.Now you might not have an idea that okay, you will buy that.You want to buy a mobile phone but because that particular thing was recommended andthat was a very small price in comparison to the mobile phone's price and you thoughtthat okay, I will buy that and when you thought that okay, I will buy that, you bought thatand that is a cross-selling that happened.Now this cross-selling is incremental sells that does not contribute towards the majorityof your sale volume.But that incremental sale, let us say there are 10 million mobile phones are being soldin a particular year and out of those 10 million 10 percent guys actually do a cross.So 90 % case it is a failure, only 10 % case that recommendation leads to some successfulconversion.10 % is a very big number.It probably does not happen that way.But even if it is 10 % then 1 million cases are sold and each case is let us say 500 rupees.So 1 million into 500 rupees you are getting 500 million extra revenue from probably ifyou have not done that, probably most of the people will not buy.You will say that how many people if you go and check your friends and how many peopleactually buy a supportive case.Not even 10 % probably, probably less than that.But this recommendation might boost a little bit up.So that is something that we are trying to see that how I can capture the long tail.So recommendations for you can be the coupons, recommendations for you can be the phone caseas it is told.It can be, recommendations for you can be whether you want to connect with other peopleor not.So there are various types of recommendations that comes and these recommendations creates‘aaha’ feeling.It is a surprise that comes into your mind.That aaha feeling is something that is important and we are leaving the age of informationand entering the age of recommendations.So please understand the "The Long Tail" he told that recommendation works lot.So often times we actually want recommendation from a legitimate source of information butsometimes even if the information is not legitimate in terms of you do not know who is recommendingit, it is the platform that is recommending you.You still probably would like to have a little bit of positive association towards this recommendationbecause without recommendation without any suggestion to you your chances of buying ismuch lower than with recommendation irrespective of from whom this recommendation is coming.For example, let us say you are walking in the road.You are hungry, you want to have some food.One thing is that let us you ask somebody, so why will I get a good food?Anything, good fresh food.Whatever he says even if you do not know that person, you might probably whatever he saysyou will have a positive association towards that particular thing.Let us say he says okay, go to that restaurant, it is good I know.So the moment he says that, even if you do not know the person, you might say that okay,why will he give a bad recommendation to me?He does not know, me I does not know him so I do not know him.So that will not create a problem.So I do not think that he is recommending to create a harm to me.So in general, we trust recommenders.So based on that I am saying that the moment is a simple nudge, a small recommendationcomes up, your chances of buying goes up and that is why recommendation engine is important.So there are search and so there are two types of products.So customer, so this is something that we see.So customers who have bought this item has also bought these items and it is a specialcase of in a brick and mortar retail store.But in a brick and mortar retail store, actually, it is not written like this.In brick and mortar store also recommendation engine works but they just put the product.For example, you will see that just beside utensils or just beside crockeries, therewill be a Scotch bite or the powder, washing powder, dish washing powder will be there,just beside the crockeries because it is often be the case that when you are purchasing this,you might have in your mind that okay, I might have to wash these dishes as well and thenthey create a special kind of dish washer, dish washing soap and etcetera, which is requiredfor specialized some kind of crockery.So you might have a feeling that okay, for my crockery which is of high value, I shouldbuy a liquid dish washing soap rather than a solid cake kind of soap because we havea perception that cake kind of soap is probably not for high quality good crockery.It might create a damage.It is there is too much of pH content high or low or whatever and then that will probablycreate a corrosion on my utensils and etcetera.So that kind of feelings might come and that is why you might prefer a liquid thing.That will be good to your hand also because you or whoever is dish washing and it willbe good for the crockeries also.So those kind of feelings are there and because those kinds of feelings are there people actuallyput those items beside the crockery items.So in natural cases, users search items, so you find out the user and you see that whatkind of items are good for him and in case of recommendations you find out the itemsand you suggest that the items are suggested to the users rather than users are searchingthe items.So this is something that is also coming up very commonly in both in e-commerce, mainlyin e-commerce but also in smaller in brick and mortar retail firms as well.Now going ahead this is some of the examples.For example, frequently bought together.So these are the two books, for example, book one and book two.These two books are frequently bought together.So these kinds of suggestions comes up in Amazon or customer who bought this item alsobought these items.So customer who bought this first book, so this is the page of the first book and thatis why the other book is tagged along and they will also buy these books as well.So these books are also there, so because they know that okay, this guy is searchingfor data mining books and there are lots of other data mining books, so those books willalso come in the recommendation of Amazon.So this is one type of recommendations.Another kind of recommendation is more personalized.So Amazon is a personalized opening online store where the introduction to data miningis what you are seeing.What other items do customers buy after viewing this item?So, see, they are different.These two are bought together.These two are when some of the item one is bought, the second item is also bought.And here when one item is seen, second item is bought.So the recommendations also when you create these recommendation, you can have differentkinds of recommendation.One is that both items should be in the same basket.Second is both items in past history might be in different baskets but both items hasto be bought.So you buy apple and you buy orange.You buy apple and orange together that is the first case, this case.You buy apple and orange differently but in a time period, in 7 days time period you buyin different trips you buy but you buy them both apple and orange, this is your case andyou have seen apple but bought orange.You have contemplated that or you have delicated with yourself that okay, this apple and theorange I have seen it but then I bought orange, that is this case.So different kind of purchase situation, in a customer journey there can be differentkind of situations where this will work.So this is probably applicable when you are let us say in information search stage inyour customer journey.On the other hand, this is applicable when you are actually purchasing, you are actuallymaking the purchase and this is post purchase.Customers who bought this item also bought is a post purchase situation.So you have bought it, you got to the probably it will be more applicable when you went tothe purchase you got the purchase order number and etcetera, payment successful, that particularpage and then it comes in the below that okay, you have purchased this and now there arelots of customers who already purchased this, also buys this.Probably also it can come when you open up your Amazon.You say that okay, so that is the last purchase that you have made then people who have boughtthese products also have bought this kind of products.So that is also one kind of recommendation engine.So different kinds of recommendation engines can be created in terms of when the recommendationis coming.So based on that the backend calculations might be a little bit different, differentin terms of what source of data you are taking.If you are taking this thing then you have to also consider what customers are seeing.Customer viewing data should also be collected because customers who have seen this thenthey bought this one.So customer viewing data and there customer buying data has to be matched.On the other hand, in this case only customer basket data in the same basket will be matched.Two different baskets will not be matched.So you bought product A and product B in the same basket then only it is 1, otherwise 0but here product A and product B can be bought in different baskets but both has to be boughtnot viewed both has to be bought then only 1.So the first hand calculations the initial I would say contributing data set, the primaryraw data set will be different in these three different cases but more or less the algorithmwill remain same.Going ahead, what is the recommender problem?A good recommender is which shows show programming titles to a software engineer and baby toysto a new mother.In other words, there has to be a fit.Whatever you are recommending, the customer’s requirements and your recommendation shouldhave good fit.You will not show a programming title, means that means a book for programming to a motherand baby toys to a software engineer.So you might show baby toys to a software engineer as long as he is a father or motherbut you will not show a baby toy to a software engineer.So mother or father irrespective of his or her profession.So programming language you will show it to a programming language's books you will showto a software engineer.So in other words, you have to have a good fit of whatever you are recommending and whoeveris the recipient of the recommendation and do not recommend items users already knowsor would find anyway.So see, we are doing incremental sales.This is something that is very very important when you are doing.So it this anyway recommendation engine will require a little bit of calculation.The more you can reduce the calculation the better because it in often case in e-commercefirms there are lots of products probably how many?Millions or if not millions lakhs of products that are there in the e-commerce firms thesedays.Now if I have to create some kind of matrix similarity and etcetera for 1 lakh products,that becomes a very difficult calculation.Lots of, not difficult I would say very time taking calculation, heavy job.Now I have to find out ways to reduce that heavy job, as simple as that.If I, and because that is costly, so any calculation see when it is 1 million by 1 million matrixyou are calculating that will take a huge space in your in whatever server or whateveryou are doing the calculations.That will take more time also.So 1 million, so 10 to the power 6 into 10 to the power 6 that is 10 to the power 12cells you have to calculate, so that will also take lots of time.So time and resources cost money as simple as that.So if it costs money it is better to find out ways to reduce that cost.So I will only incur cost when that incremental benefit, the marginal benefit of this recommendationis pretty high.If you anyway buy it, so I will not recommend anything which you will anyway buy and I willnot recommend anything which will you never buy.So these are the two things.Feed is important because you will never, otherwise you will never buy.For example, a mother I will recommend him, I will not recommend him let us say somethingwhich is related to let us say yeah, programming titles or books or probably items which shecannot use right now.I will not recommend them those kind of stuff.For example, a pregnant lady.You find out a pregnant lady.Will you recommend probably anti-pregnancy pills or let us say some pregnancy test tothat pregnant lady when you know that this person is already pregnant?For long period of time probably that will not be needed but you will probably recommendbaby toy not baby toys but probably books which is related to baby's upbringing.You can recommend like folic acid or different kinds of medications which helps her in keepingthings okay and etcetera.So those kind of stuffs you will recommend.So you have to find out products which are fit to this particular person who is gettingthe recommendation, that is number one.Number two is you will not recommend such kind of products which he will anyway buy.You do not have to, so because if you have to take those kind of products also in therecommendation platform then you have to also calculate, do some calculation for them.Now that some calculation will give you no result because this person will anyway buy,irrespective of the nudge he will buy or she will buy.Now the nudge will create some cost in your resource, in your time allocation and etcetera,it will create some cost.So it will be a basically a loss for you.So do not recommend items user already knows or find out anyway and expand user's tastewithout offending or annoying him her.So this is also important decision.So there is some time you have to find out that this is also actually associated withthe fit case that I will not recommend something which is very far from the fit which has tobe reasonably far.Now how far you will be allowing is something that is a very crucial decision.What are the challenges?So huge amount of data I told, tens and millions of customers and millions of distinct catalogueitems.So if there is millions of products and probably 10 million 20 million customers then thisbecomes a huge calculation and those such a size of calculation becomes problematic.What are the challenges?Results are required to be written in a real time.So many result you have to, so I will, let us say I am seeing a website in Amazon andyou give me recommendation probably half an hour later that does not work.You have to give me recommendation then and there.So you have to do all the high end-calculation beforehand so that at the time of the recommendationit works real time.New customers have limited information that is number thing, so it requires a little bitof customer purchased data previously.You will be giving me a good fit only when you know what is fitting for me, what kindof person I am, what I like, what I dislike.If you do not know that information you cannot recommend.Now if to get that information you have to have be old customer.I cannot be a new customer because for new customer those data is not available.Old customers can have a glut of information.So old customer have sometimes have huge information, you do not know which one to use, which onenot to use.A customer data is volatile.Customer change is overtime.The same customer might be different so the same person me let us say before my childwas born and after my child was born are two different people.The same person before marriage after marriage we say for female, there are lots of in acustomer segmentation video I told that women life changes and that is why woman's lifestylenot lifestyle say woman's life period, life span is a very interesting case for marketersbecause that changes overtime like anything.So the same woman who is in adolescent versus who is in let us say in her youth age andthen after a mom and probably after the second child and etcetera, the same person changesa lot.So customer data is volatile, how you can handle that?So all of these things are challenging which we will try to solve.So Amazon has created a recommendation engine where Amazon's model that implements the recommendationalgorithm and algorithm feature is most recommendation algorithms start by finding a set of similarcustomers who purchased and rated items and overlap the user's purchased and rated items.So they find out that whether customer A and customer B are similar.How they are similar?Based on whether they are purchased the similar kind of items or they have given ratings tosimilar kind of items.So ratings, why?Because sometimes you might not have purchased in this platform but you might have purchasedin some other platform but you have rated means probably you have used that product.So based on that I can give a similarity and the Amazon's item to item collaborative filteringis focusing on finding similar instead of similar items instead of similar customers.So I will talk about that what is collaborative filtering and recommendation engine work flowlook like this.The major focus here is personalization.So what is personalization?Recommendations are instances of personalization software so basically this is in a whole worldof retail, retail personalization is a topic under which this recommendation comes.Personalization concerns adapting the individual needs, interests and preferences of each user.So you actually try to adapt with customer's needs and some of the areas are recommending,filtering or predicting.So recommending is something that we are focusing on.In the broad domain of marketing, it comes under CRM and when we focus a little bit more,machine learning can allow learning a user model a profile of a particular user based.On the other hand, the model or profile can be used to recommend items, filter information’sor predict customer behavior.One of the major thing that we use, I told that Amazon use collaborative filtering.So what is collaborative filtering?Collaborative filtering requires maintaining a dataset of many users' ratings of varietyitems, sometimes ratings sometimes purchased decisions.From a given user find out similar users through those ratings.So you have to find out which two users are similar and recommend items rated highly bythese similar users but not rated by the current user.So I will recommend such item which I have not used till now means I will be recommendedsuch items which I have not used till now or not rated till now but similar customerswho are similar to me have rated it pretty high.So for example, let us say I and one of my friend both like Shahrukh Khan and both haveseen Kuch Kuch Hota Hai, Kal ho na ho, DDLJ and etcetera.These guy has seen let us say Main hoon na and I have not seen it.So when you see that okay, so these three four movies I have this guy have seen andthis particular person have seen Main hoon na and I did not see it.He will recommend it to me because both me and that person is similar depending our taste.How that comes up?Based on our past purchase history of Shahrukh Khan movies.Now I have not seen a Shahrukh Khan movie, he has seen a Shahrukh Khan movie, let usrecommend it to him.So that is the basic crux.Almost all existing commercial recommender using this approach.So this is the most common approach and that is why we will majorly focus on collaborativefiltering.Now weight all users with respect to similarity with the active user.So you have to find out that, so I will if there are 4 guys who are similar to me whoserecommendation whose behavior will be most matched to me?So after the calculation of similarity, we should also find out that who is active andwho is not active.We will majorly focus on the active user's purchase behavior and based on that you willrecommend somebody else than by inactive users and select a subset of users because thatwe will discuss later.Normalize ratings and compute a prediction from a weighted combination, for example,let us say do you think that all when you calculate distance between me and some otheruser, that distance is measured based on as I told the purchase ratings or purchase behaviors.Now do you think that all purchase behavior should be similarly treated?For example, let us say you are thinking about movie recommendations.In Netflix, you are talking about movie recommendations.Now he also watches one series, I also watched one series one-one but he has watched prettyhigh and I watched pretty low.So in a series he watched let us say Grey's Anatomy is a series.He watched all the seasons, I watched 5 seasons and did not watch the rest of the seasons.On the other hand, in case of movies, he watched the full movie, I watched also the full movie.So in first case it is in movies case it is both is 1, in Grey's Anatomy case also bothis 1 because I have not considered the how much I have watched.Should we give same weightage to these two cases when we are discussing similarity?So I do not know.So that is something that is a difficult question.So when you measure similarity, forget about purchase behavior’s data.Let us say two persons you have demographic data about them.Let us say I and one of my friend I am a male that person is also a male and I am a Bengaliand that person is let us say an Odia, let us assume versus another of my friend I ammale she is female but I am Bengali, she is also Bengali.In what kind of situation which attribute the gender versus your native which attributebecomes more important?When it is food items probably, let us say Bengali foods, probably Odia foods are equallyprobably very close Bengali foods but still I will say and if it is food item, probablyour food habits match based on the native.So native will get more importance.But let us say if it is apparel then probably the gender matching is more important ratherthan the so the male of Bengal and male of Odisha will probably wear similar kind ofproducts in comparison to male of Bengal and female of Bengal.So all I am trying to say that the which attribute you give more weightage depends on the situation,the product purchase situation that you are talking about.So present items, so normalize ratings and compute a prediction from a weighted combinationof selected neighbor's ratings and present items with highest predicted ratings as recommendations.So this is something that we are trying to do.Now how to find out the similarity?There are different ways.One is correlation.This is a very simple formula of correlation.So Pearson correlation you can find out.We also use cosine covariance, standard deviations.These are also measures.So these measures will be used to calculate the correlation.I will not go into details.You know what is covariance, what is standard deviation.And then there are other measures like cosine matrix we create.Cosine is also a measure.We will talk about that in a different video and then we created a weighted sum.So the significance weight is S a,u and C a,u is the correlation value and W a,u isS a,u into C a,u.Now what is S a,u?S a,u is 1 when m is greater than 50 and m by 50 if m is smaller than50.So something like that is what we are trying to create.And for a given active user a, select correlated users to serve as source predictions, thatis number one.Standard approach is to use the most similar n users, most similar users based on similaritymatrix W a,u.So whoever is the most similar you find out those people.Alternate approach is to include all users whose similarity weight is above a given threshold.So if that similarity W a,u is above certain cut-off, you consider them or you considerthe top 50, top 100 whatever.Now in case of rating prediction, how do you predict the rating?Your prediction rating is basically ra plus the weighted sum, this is the distance, youwill see this is the distance of your rating with the all the average ratings and theninto the weight divided by the summation of weights.This is the formula that we use for rating prediction and there are other collaborativefiltering methods for other kind of problems.We will talk about that.The basic problems are same for all cases.There is a cold start that means there needs to be enough other users already in the systemto find a match.Sparsity is also a problem.If there are many items to be recommended, even if there are many users, the user ratingmatrix is sparse and it is hard to find users that are rated same items.So that is so lots of zeroes will come in the matrix.First rater cannot recommend an item that has not been previously rated.If one item is very new, it is not only the customer is new will be create a problem eventhe item is new will create a problem.So if the item is new, nobody has rated nobody has seen it, I do not know whom to recommend.So that creates sometimes a problem and we will try to solve that and popularity biasis cannot recommend items to someone with unique tastes.So certain niche products you cannot recommend.So that is also a problem.To solve the first problem so to solve this problem where the first rater, sometimes wedo not believe on customer's purchased data or rating data.We believe on the content of the product.So let us say a new movie of Shahrukh Khan has come up.I do not know what is the quality of movie and etcetera, who has seen, who has not seen,whether those who have seen are similar to me or not?I do not know those details because this particular movie is new in the Netflix platform or whateverplatform and then I do not have past purchased data for that particular item.But what I will do is I will find out which of the products I have similar content.