Loading
Note di Apprendimento
Study Reminders
Support
Text Version

Collaborative Filtering

Set your study reminders

We will email you at these times to remind you to study.
  • Monday

    -

    7am

    +

    Tuesday

    -

    7am

    +

    Wednesday

    -

    7am

    +

    Thursday

    -

    7am

    +

    Friday

    -

    7am

    +

    Saturday

    -

    7am

    +

    Sunday

    -

    7am

    +

Hello everybody, welcome to Marketing Analytics course.This is Dr. Swagato Chatterjee from VGSOM, IIT Kharagpur who is taking this course.We are in week 7, session 2 and we are discussing about recommendation engine and retail analytics.We were discussing specifically about collaborative filtering and I have discussed quite a lotin the last video but I will continue from there.So, here we will talk about user-based and then item-based collaborative filtering.These are the two things we will discuss in details.So the approach is that it is similar.You have to find out a similarity matrix between the customers.So previously I talked about the rating based.In the rating based, we create the similarity based on the correlation of the ratings.Because ratings are often given in 1 to 5 point scale.We can find out correlations of that.On the other hand, if it is purchased decision based then represents, so approach is representa customer as a N-dimensional vector of items.So there are items and there are, so what is the dimension of vector?There are n items.Vector is positive for purchased or positively rated items and negative for negatively purchasedor negatively rated items.So the vector is either 1, 0 or 1, minus 1 or something like that.So 1 means purchase and 0 means not purchased or minus 1 means not purchased or somethinglike that and then we find out cosine matrix between the customers.So similarity matrix is cosine where I find out a cosine matrix between the customersand then generate recommendations based on a few customers who are most similar.So cosine is the similarity matrix.I find out, if I am a customer and there are 10 other customers, whom of these customers’are cosine is smaller?The smaller the cosine, the closer they are to me and if they are closer to me then whatis the, sorry, sorry.Cos theta is Cos 0 Sin 0.Cos 0 is small.So smaller the cosine, I would say the closer they are.So generates recommendations based on a few customers who are most similar to the user.That is something that becomes important and then rank each item according to the how manysimilar customers have purchased it.So this is something that we will follow in our problem also.So actually it is opposite.Cosine means the smaller is the cosine value.Cosine means the angle.It actually talks about the angle between the two vectors and smaller is that particularangle the more similar you are and for Cos 0 is 1.Sin 0 is 0.Cos 0 is 1 means specifically, the higher the cosine value the higher is the similarity.Because the theta, the angle is smaller.So that is something that we will try to deal.Computationally expensive because if there are m number of customers and n number ofitems, so you have to create a matrix which is m into n sizes and dimensionally reductionis that is why is an important factor, can increase the performance but can also reducethe quality of the recommendation.So that is also a problem.So for very large data sets, such as 10 million customers and 1 million items, the algorithmencounters severe performance and scaling issues and that is why sometimes you haveto do batch processing beforehand.You cannot do it in real time.You have to do it beforehand.On the other hand, cluster models are different kind of models.The approach is divide the customer base in many segments and treat the task as a classificationproblem.So first job is to break the customers into various segments.So you do not deal with M number of customers, you deal with small m number of clusters.Now small m is lower than capital M. So within the cluster I will not focus on the variabilityof the persons.Assign the user to the segment containing the most similar customers.Fair enough.So it is like, like probably K nearest neighbor model.K means clustering model.So if you have clusters and then you find a new cluster, whichever cluster is most closestto this guy, fit them to that cluster.And uses the purchase and ratings of the customers in the segment to generate recommendations.Now when I create the recommendation calculations, I will not focus on other clusters customer,I will focus on only those customer who are in the same segment and then we will do thecalculations.Cluster models have better online scalability and performance obviously than collaborativefiltering because they compare the user to a controlled number of segments.What is the problem?As I told the quality is an issue.Reduce the quality of the recommendation when you do dimensionality reduction.So here this is also one type of dimensionality reduction.But here you are not reducing the dimensionality in terms of the number of items.You are reducing it in terms of number of users.But that also reduces the cost, reduces the quality of the recommendation that you aregiving.And the recommendations are less relevant because the similar customers that the clustersfind are not the similar, not the most similar customers.So it might be the case that the similar customer that the cluster models find are not actuallyvery close to each other because the method that this clustering method using to createthe clusters are not the purchased data.So that is why the similarity might be different.To improve quality, it needs online segmentation, which is almost as expensive as finding similarcustomers using collaborative filtering.So in other words, frankly speaking, this is more common than this.What do you do in a collaborative filtering?In a user to user collaborative filtering or item to item collaborative filtering, thereare 2 types of collaborative filtering.In one case you find out the similarity between the users and in other case similarity betweenthe items but in both the cases you are recommending some products to some users based on thosematching.So we will try to solve both of them in the next video.So what we do?Let us say these are 17 items.So similarity of item i with item 17.So let us say there are 25 items in total and a, b, c, d, x, y, z are basically usersand these 25 items are there.Fair enough.And then I try to find out let us say item number 17 with, so here it is actually 25is missing.So these values that you are seeing here all these values are basically the number of purchasesthat has been made.So, for now 17, 0, 5, 000, 44444 etc etc.So, 17 this particular product has been bought by these people.Now how will I know that two items are similar?If two items are bought by similar type of people then they are similar.So whoever has seen 17, if they have also seen 19 then 17 and 19 are similar products.So, that is how if I do a column wise calculation, so column wise correlation matrix, I willget the similarity or cosine matrix let us say.I will get the similarity of item 1 with item 2 with item 3 and so on.On the other hand, same thing if I do the row wise, I will find out how customer A andcustomer B are similar.See, customer A has seen, both have not seen 1 and 2.Customer A has seen 3.Customer B has seen 3 four times.4 both of them have not seen.Up to 9 customer B has seen nothing but 4, 5 and 5, 6 and 9 customer A has seen.So, that is a difference.So based on the purchased rate of customer A and purchase rate of customer B, I can findout how customer A and customer B are similar to each other or dissimilar to each other.So as simple as that if you do column wise in this particular matrix, you do column wise,you get item to item collaborative filtering.If you do row wise, you will get user to user collaborative filtering.So, item to item how it works?Matches each of the user’s purchased and rated items to similar items.So these are actually ratings.So each of the items to the similar items combines those similar items to a recommendationlist.Then what is an iterative algorithm?Build a similar items table which will create and provides a better approach by calculatingthe similarity between a single product to all related products.The similarity between two items use the cosine measure.The same thing, each vector corresponding to item rather than a customer.So it corresponds to an item and vector of M dimensions corresponding to customers whohave purchased that item.So see, the advantage is that the number of items in e-commerce firm is generally lesserthan the number of customers.In a already established e-commerce firm less number of products are there, more numberof customers are there.Just give an idea, so how many products you have purchased in an e-commerce in Amazonor Flipkart in the last 1 year let us say?10.So when a person buying 10 items, 15 items, so that is saying that okay, the number ofitems are lower, number of customers are probably, number of items are lower, number of customersare higher because same item have been bought by multiple customers.So generally, that is what is being seen and because that number of products are lower,it is better to do item to item collaborative filtering because that takes less time.Offline computation has to be done because builds a similar item table that calculationwill trigger because lots of time it is extremely time intensive.In practice, it is closer to OMN that means as most customers have very few purchasesand sampling customers can also reduce runtime even further with a little reduction of similarity.So that is something.Online recommendation give a similar item tables, the algorithm finds items similarto each of the user’s purchased and ratings.Aggregate those items and then recommends the most popular correlated items.So these are all the texts.Let us do something hands on and then you will be able to understand what we are tryingto talk about.Now if I compare these 3 basic measures in terms of the scalability and quality, basicallythese are the two issues whether it is time taking, whether it is the quality that youare getting.So user based collaborative filtering requires no offline computation, impractical or largedata sets because it takes lots of time and dimensionally reduction, sampling and partitioningreduces recommendation quality.On the other hand, cluster models can perform much on a computation offline but recommendedquality is relatively poor.So that is a problem.So these two cases it works well but for the moment takes time.The moment you will try to reduce the time taking, its quality gets reduced and in general,quality is low.On the other hand, item to item collaborative filtering, scalability and performance areachieved by creating the expensive similar items table offline.So batch processing is done beforehand.An online component looking up similar item scales independently of the catalog size orthe number of customers.So it does not depend on the number of customers.It only depends on the catalog size.So catalog, so then it is okay.First for extremely large data sets it works.Recommendation quality is excellent since it recommends high correlated similar items.So basically it is saying that this item to item collaborative filtering works betterthan these two.We will solve all these three but we will try to focus on this one more.So some of the results the MovieLens dataset, some people have tried these 3 methods.The MovieLens and the reference is given below.The MovieLens dataset 1 million ratings of 6040 users and 3090 movies.So number of movies are lower than number of users.The best overall results are reached by item to item collaborative filtering.You see that item to item collaborative filtering to 170 second.Much much lower than the other one and a mean average, mean absolute radar means whateverthe recommended people actually bought it or not.The radar is coming up to 63 percent.So it is not 63, 0.6382.In other case, it is 0.66, 0.67 so comparable.And prediction time taken is 3 second.So item to item giving similar result but with much much lesser time.Some related applications can be Pandora, can be Netflix, can be Google YouTube.For example, for Pandora, Pandora is based its recommendation on data for Music GenomeProject.Assigns 400 attributes for each song.Each song they find out 400 attributes.Done by musicians.Take half an hour per song and use this method to find songs which is similar to user’sfavorite songs.What is the method, benefit?Accurate method do not need lots of user information.So because it is item based music and musician are matched.Do not scale very well and often feels that Pandora’s library is somewhat limited.So that is one problem.Netflix’s movie recommendation based on the ratings.Make recommendations by comparing the watching and the searching habits.Sorry, searching habits of similar users as well as by offering movies that share characteristicswith films that a user has rated.So both the purchase behavior of the customer and the content, so create a hybrid model.Collaborative, content based, and knowledge based and demographic techniques serve asthe basis of the recommendation system.So collaborative filtering is used, content is used and also the demographic of the customersis used.So it is ensemble method.It is a user based, item based and content based then create a recommendation enginewith all these three and then match them together and create an ensemble method of 107 differentalgorithm approaches.What is the benefit?So these are large scale implementations.We will do small small examples.Each of these techniques have some short comings, use multiple techniques together to overcome,so one shortcoming of one technique is overcome by the advantage of another technique.So that is how the ensemble methods work.Google YouTube recommendation system how it works?It focus on videos.Brings videos to users which they believe users will be interested in.Increase the number of videos, increase the length of time and maximize the enjoymentand this is their motto.Ultimately, Google can increase revenue by showing more ads.So more the engagement, the more ads they can show and the more revenue they can generate.What is it?Give up the old recommendation system based on random walk.Change to a new one based on Amazon’s item to item collaborative filtering.So old one was some random one and this one is based on collaborative filtering.In 2010, they did this change.Amazon’s item to item collaborative filtering appears to be the best for video recommendation.So this is somehow the background.So we have learned about basically some of the references.We have learned the primary features about item to item, user to use collaborative filtering,content based method and then the clustering method, cluster based recommendation engine.So these are the 4 things that we will try to use in smaller problems in a hands on basisin the next videos.Thank you for being with me with this video.I will come back with the next video in terms of collaborative filtering.First, we will do item to item.Thank you very much.