recommendation system research methodology

• Survey paper
• Open access
• Published: 03 May 2022

A systematic review and research perspective on recommender systems

• Deepjyoti Roy   ORCID: orcid.org/0000-0002-8020-7145 1 &
• Mala Dutta 1  

Journal of Big Data volume  9 , Article number:  59 ( 2022 ) Cite this article

80k Accesses

148 Citations

14 Altmetric

Metrics details

Recommender systems are efficient tools for filtering online information, which is widespread owing to the changing habits of computer users, personalization trends, and emerging access to the internet. Even though the recent recommender systems are eminent in giving precise recommendations, they suffer from various limitations and challenges like scalability, cold-start, sparsity, etc. Due to the existence of various techniques, the selection of techniques becomes a complex work while building application-focused recommender systems. In addition, each technique comes with its own set of features, advantages and disadvantages which raises even more questions, which should be addressed. This paper aims to undergo a systematic review on various recent contributions in the domain of recommender systems, focusing on diverse applications like books, movies, products, etc. Initially, the various applications of each recommender system are analysed. Then, the algorithmic analysis on various recommender systems is performed and a taxonomy is framed that accounts for various components required for developing an effective recommender system. In addition, the datasets gathered, simulation platform, and performance metrics focused on each contribution are evaluated and noted. Finally, this review provides a much-needed overview of the current state of research in this field and points out the existing gaps and challenges to help posterity in developing an efficient recommender system.

Introduction

The recent advancements in technology along with the prevalence of online services has offered more abilities for accessing a huge amount of online information in a faster manner. Users can post reviews, comments, and ratings for various types of services and products available online. However, the recent advancements in pervasive computing have resulted in an online data overload problem. This data overload complicates the process of finding relevant and useful content over the internet. The recent establishment of several procedures having lower computational requirements can however guide users to the relevant content in a much easy and fast manner. Because of this, the development of recommender systems has recently gained significant attention. In general, recommender systems act as information filtering tools, offering users suitable and personalized content or information. Recommender systems primarily aim to reduce the user’s effort and time required for searching relevant information over the internet.

Nowadays, recommender systems are being increasingly used for a large number of applications such as web [ 1 , 67 , 70 ], books [ 2 ], e-learning [ 4 , 16 , 61 ], tourism [ 5 , 8 , 78 ], movies [ 66 ], music [ 79 ], e-commerce, news, specialized research resources [ 65 ], television programs [ 72 , 81 ], etc. It is therefore important to build high-quality and exclusive recommender systems for providing personalized recommendations to the users in various applications. Despite the various advances in recommender systems, the present generation of recommender systems requires further improvements to provide more efficient recommendations applicable to a broader range of applications. More investigation of the existing latest works on recommender systems is required which focus on diverse applications.

There is hardly any review paper that has categorically synthesized and reviewed the literature of all the classification fields and application domains of recommender systems. The few existing literature reviews in the field cover just a fraction of the articles or focus only on selected aspects such as system evaluation. Thus, they do not provide an overview of the application field, algorithmic categorization, or identify the most promising approaches. Also, review papers often neglect to analyze the dataset description and the simulation platforms used. This paper aims to fulfil this significant gap by reviewing and comparing existing articles on recommender systems based on a defined classification framework, their algorithmic categorization, simulation platforms used, applications focused, their features and challenges, dataset description and system performance. Finally, we provide researchers and practitioners with insight into the most promising directions for further investigation in the field of recommender systems under various applications.

In essence, recommender systems deal with two entities—users and items, where each user gives a rating (or preference value) to an item (or product). User ratings are generally collected by using implicit or explicit methods. Implicit ratings are collected indirectly from the user through the user’s interaction with the items. Explicit ratings, on the other hand, are given directly by the user by picking a value on some finite scale of points or labelled interval values. For example, a website may obtain implicit ratings for different items based on clickstream data or from the amount of time a user spends on a webpage and so on. Most recommender systems gather user ratings through both explicit and implicit methods. These feedbacks or ratings provided by the user are arranged in a user-item matrix called the utility matrix as presented in Table 1 .

The utility matrix often contains many missing values. The problem of recommender systems is mainly focused on finding the values which are missing in the utility matrix. This task is often difficult as the initial matrix is usually very sparse because users generally tend to rate only a small number of items. It may also be noted that we are interested in only the high user ratings because only such items would be suggested back to the users. The efficiency of a recommender system greatly depends on the type of algorithm used and the nature of the data source—which may be contextual, textual, visual etc.

Types of recommender systems

Recommender systems are broadly categorized into three different types viz. content-based recommender systems, collaborative recommender systems and hybrid recommender systems. A diagrammatic representation of the different types of recommender systems is given in Fig.  1 .

Content-based recommender system

In content-based recommender systems, all the data items are collected into different item profiles based on their description or features. For example, in the case of a book, the features will be author, publisher, etc. In the case of a movie, the features will be the movie director, actor, etc. When a user gives a positive rating to an item, then the other items present in that item profile are aggregated together to build a user profile. This user profile combines all the item profiles, whose items are rated positively by the user. Items present in this user profile are then recommended to the user, as shown in Fig.  2 .

One drawback of this approach is that it demands in-depth knowledge of the item features for an accurate recommendation. This knowledge or information may not be always available for all items. Also, this approach has limited capacity to expand on the users' existing choices or interests. However, this approach has many advantages. As user preferences tend to change with time, this approach has the quick capability of dynamically adapting itself to the changing user preferences. Since one user profile is specific only to that user, this algorithm does not require the profile details of any other users because they provide no influence in the recommendation process. This ensures the security and privacy of user data. If new items have sufficient description, content-based techniques can overcome the cold-start problem i.e., this technique can recommend an item even when that item has not been previously rated by any user. Content-based filtering approaches are more common in systems like personalized news recommender systems, publications, web pages recommender systems, etc.

Collaborative filtering-based recommender system

Collaborative approaches make use of the measure of similarity between users. This technique starts with finding a group or collection of user X whose preferences, likes, and dislikes are similar to that of user A. X is called the neighbourhood of A. The new items which are liked by most of the users in X are then recommended to user A. The efficiency of a collaborative algorithm depends on how accurately the algorithm can find the neighbourhood of the target user. Traditionally collaborative filtering-based systems suffer from the cold-start problem and privacy concerns as there is a need to share user data. However, collaborative filtering approaches do not require any knowledge of item features for generating a recommendation. Also, this approach can help to expand on the user’s existing interests by discovering new items. Collaborative approaches are again divided into two types: memory-based approaches and model-based approaches.

Memory-based collaborative approaches recommend new items by taking into consideration the preferences of its neighbourhood. They make use of the utility matrix directly for prediction. In this approach, the first step is to build a model. The model is equal to a function that takes the utility matrix as input.

Model = f (utility matrix)

Then recommendations are made based on a function that takes the model and user profile as input. Here we can make recommendations only to users whose user profile belongs to the utility matrix. Therefore, to make recommendations for a new user, the user profile must be added to the utility matrix, and the similarity matrix should be recomputed, which makes this technique computation heavy.

Recommendation = f (defined model, user profile) where user profile  ∈  utility matrix

Memory-based collaborative approaches are again sub-divided into two types: user-based collaborative filtering and item-based collaborative filtering. In the user-based approach, the user rating of a new item is calculated by finding other users from the user neighbourhood who has previously rated that same item. If a new item receives positive ratings from the user neighbourhood, the new item is recommended to the user. Figure  3 depicts the user-based filtering approach.

User-based collaborative filtering

In the item-based approach, an item-neighbourhood is built consisting of all similar items which the user has rated previously. Then that user’s rating for a different new item is predicted by calculating the weighted average of all ratings present in a similar item-neighbourhood as shown in Fig.  4 .

Item-based collaborative filtering

Model-based systems use various data mining and machine learning algorithms to develop a model for predicting the user’s rating for an unrated item. They do not rely on the complete dataset when recommendations are computed but extract features from the dataset to compute a model. Hence the name, model-based technique. These techniques also need two steps for prediction—the first step is to build the model, and the second step is to predict ratings using a function (f) which takes the model defined in the first step and the user profile as input.

Recommendation = f (defined model, user profile) where user profile  ∉  utility matrix

Model-based techniques do not require adding the user profile of a new user into the utility matrix before making predictions. We can make recommendations even to users that are not present in the model. Model-based systems are more efficient for group recommendations. They can quickly recommend a group of items by using the pre-trained model. The accuracy of this technique largely relies on the efficiency of the underlying learning algorithm used to create the model. Model-based techniques are capable of solving some traditional problems of recommender systems such as sparsity and scalability by employing dimensionality reduction techniques [ 86 ] and model learning techniques.

Hybrid filtering

A hybrid technique is an aggregation of two or more techniques employed together for addressing the limitations of individual recommender techniques. The incorporation of different techniques can be performed in various ways. A hybrid algorithm may incorporate the results achieved from separate techniques, or it can use content-based filtering in a collaborative method or use a collaborative filtering technique in a content-based method. This hybrid incorporation of different techniques generally results in increased performance and increased accuracy in many recommender applications. Some of the hybridization approaches are meta-level, feature-augmentation, feature-combination, mixed hybridization, cascade hybridization, switching hybridization and weighted hybridization [ 86 ]. Table 2 describes these approaches.

Recommender system challenges

This section briefly describes the various challenges present in current recommender systems and offers different solutions to overcome these challenges.

Cold start problem

The cold start problem appears when the recommender system cannot draw any inference from the existing data, which is insufficient. Cold start refers to a condition when the system cannot produce efficient recommendations for the cold (or new) users who have not rated any item or have rated a very few items. It generally arises when a new user enters the system or new items (or products) are inserted into the database. Some solutions to this problem are as follows: (a) Ask new users to explicitly mention their item preference. (b) Ask a new user to rate some items at the beginning. (c) Collect demographic information (or meta-data) from the user and recommend items accordingly.

Shilling attack problem

This problem arises when a malicious user fakes his identity and enters the system to give false item ratings [ 87 ]. Such a situation occurs when the malicious user wants to either increase or decrease some item’s popularity by causing a bias on selected target items. Shilling attacks greatly reduce the reliability of the system. One solution to this problem is to detect the attackers quickly and remove the fake ratings and fake user profiles from the system.

Synonymy problem

This problem arises when similar or related items have different entries or names, or when the same item is represented by two or more names in the system [ 78 ]. For example, babywear and baby cloth. Many recommender systems fail to distinguish these differences, hence reducing their recommendation accuracy. To alleviate this problem many methods are used such as demographic filtering, automatic term expansion and Singular Value Decomposition [ 76 ].

Latency problem

The latency problem is specific to collaborative filtering approaches and occurs when new items are frequently inserted into the database. This problem is characterized by the system’s failure to recommend new items. This happens because new items must be reviewed before they can be recommended in a collaborative filtering environment. Using content-based filtering may resolve this issue, but it may introduce overspecialization and decrease the computing time and system performance. To increase performance, the calculations can be done in an offline environment and clustering-based techniques can be used [ 76 ].

Sparsity problem

Data sparsity is a common problem in large scale data analysis, which arises when certain expected values are missing in the dataset. In the case of recommender systems, this situation occurs when the active users rate very few items. This reduces the recommendation accuracy. To alleviate this problem several techniques can be used such as demographic filtering, singular value decomposition and using model-based collaborative techniques.

Grey sheep problem

The grey sheep problem is specific to pure collaborative filtering approaches where the feedback given by one user do not match any user neighbourhood. In this situation, the system fails to accurately predict relevant items for that user. This problem can be resolved by using pure content-based approaches where predictions are made based on the user’s profile and item properties.

Scalability problem

Recommender systems, especially those employing collaborative filtering techniques, require large amounts of training data, which cause scalability problems. The scalability problem arises when the amount of data used as input to a recommender system increases quickly. In this era of big data, more and more items and users are rapidly getting added to the system and this problem is becoming common in recommender systems. Two common approaches used to solve the scalability problem is dimensionality reduction and using clustering-based techniques to find users in tiny clusters instead of the complete database.

Methodology

The purpose of this study is to understand the research trends in the field of recommender systems. The nature of research in recommender systems is such that it is difficult to confine each paper to a specific discipline. This can be further understood by the fact that research papers on recommender systems are scattered across various journals such as computer science, management, marketing, information technology and information science. Hence, this literature review is conducted over a wide range of electronic journals and research databases such as ACM Portal, IEEE/IEE Library, Google Scholars and Science Direct [ 88 ].

The search process of online research articles was performed based on 6 descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*”, “Hybrid Recommend*”. The following research papers described below were excluded from our research:

News articles.

Master’s dissertations.

Non-English papers.

Unpublished papers.

Research papers published before 2011.

We have screened a total of 350 articles based on their abstracts and content. However, only research papers that described how recommender systems can be applied were chosen. Finally, 60 papers were selected from top international journals indexed in Scopus or E-SCI in 2021. We now present the PRISMA flowchart of the inclusion and exclusion process in Fig.  5 .

PRISMA flowchart of the inclusion and exclusion process. Abstract and content not suitable to the study: * The use or application of the recommender system is not specified: **

Each paper was carefully reviewed and classified into 6 categories in the application fields and 3 categories in the techniques used to develop the system. The classification framework is presented in Fig.  6 .

Classification framework

The number of relevant articles come from Expert Systems with Applications (23%), followed by IEEE (17%), Knowledge-Based System (17%) and Others (43%). Table 3 depicts the article distribution by journal title and Table 4 depicts the sector-wise article distribution.

Both forward and backward searching techniques were implemented to establish that the review of 60 chosen articles can represent the domain literature. Hence, this paper can demonstrate its validity and reliability as a literature review.

Review on state-of-the-art recommender systems

This section presents a state-of-art literature review followed by a chronological review of the various existing recommender systems.

Literature review

In 2011, Castellano et al. [ 1 ] developed a “NEuro-fuzzy WEb Recommendation (NEWER)” system for exploiting the possibility of combining computational intelligence and user preference for suggesting interesting web pages to the user in a dynamic environment. It considered a set of fuzzy rules to express the correlations between user relevance and categories of pages. Crespo et al. [ 2 ] presented a recommender system for distance education over internet. It aims to recommend e-books to students using data from user interaction. The system was developed using a collaborative approach and focused on solving the data overload problem in big digital content. Lin et al. [ 3 ] have put forward a recommender system for automatic vending machines using Genetic algorithm (GA), k-means, Decision Tree (DT) and Bayesian Network (BN). It aimed at recommending localized products by developing a hybrid model combining statistical methods, classification methods, clustering methods, and meta-heuristic methods. Wang and Wu [ 4 ] have implemented a ubiquitous learning system for providing personalized learning assistance to the learners by combining the recommendation algorithm with a context-aware technique. It employed the Association Rule Mining (ARM) technique and aimed to increase the effectiveness of the learner’s learning. García-Crespo et al. [ 5 ] presented a “semantic hotel” recommender system by considering the experiences of consumers using a fuzzy logic approach. The system considered both hotel and customer characteristics. Dong et al. [ 6 ] proposed a structure for a service-concept recommender system using a semantic similarity model by integrating the techniques from the view of an ontology structure-oriented metric and a concept content-oriented metric. The system was able to deliver optimal performance when compared with similar recommender systems. Li et al. [ 7 ] developed a Fuzzy linguistic modelling-based recommender system for assisting users to find experts in knowledge management systems. The developed system was applied to the aircraft industry where it demonstrated efficient and feasible performance. Lorenzi et al. [ 8 ] presented an “assumption-based multiagent” system to make travel package recommendations using user preferences in the tourism industry. It performed different tasks like discovering, filtering, and integrating specific information for building a travel package following the user requirement. Huang et al. [ 9 ] proposed a context-aware recommender system through the extraction, evaluation and incorporation of contextual information gathered using the collaborative filtering and rough set model.

In 2012, Chen et al. [ 10 ] presented a diabetes medication recommender model by using “Semantic Web Rule Language (SWRL) and Java Expert System Shell (JESS)” for aggregating suitable prescriptions for the patients. It aimed at selecting the most suitable drugs from the list of specific drugs. Mohanraj et al. [ 11 ] developed the “Ontology-driven bee’s foraging approach (ODBFA)” to accurately predict the online navigations most likely to be visited by a user. The self-adaptive system is intended to capture the various requirements of the online user by using a scoring technique and by performing a similarity comparison. Hsu et al. [ 12 ] proposed a “personalized auxiliary material” recommender system by considering the specific course topics, individual learning styles, complexity of the auxiliary materials using an artificial bee colony algorithm. Gemmell et al. [ 13 ] demonstrated a solution for the problem of resource recommendation in social annotation systems. The model was developed using a linear-weighted hybrid method which was capable of providing recommendations under different constraints. Choi et al. [ 14 ] proposed one “Hybrid Online-Product rEcommendation (HOPE) system” by the integration of collaborative filtering through sequential pattern analysis-based recommendations and implicit ratings. Garibaldi et al. [ 15 ] put forward a technique for incorporating the variability in a fuzzy inference model by using non-stationary fuzzy sets for replicating the variabilities of a human. This model was applied to a decision problem for treatment recommendations of post-operative breast cancer.

In 2013, Salehi and Kmalabadi [ 16 ] proposed an e-learning material recommender system by “modelling of materials in a multidimensional space of material’s attribute”. It employed both content and collaborative filtering. Aher and Lobo [ 17 ] introduced a course recommender system using data mining techniques such as simple K-means clustering and Association Rule Mining (ARM) algorithm. The proposed e-learning system was successfully demonstrated for “MOOC (Massively Open Online Courses)”. Kardan and Ebrahimi [ 18 ] developed a hybrid recommender system for recommending posts in asynchronous discussion groups. The system was built combining both collaborative filtering and content-based filtering. It considered implicit user data to compute the user similarity with various groups, for recommending suitable posts and contents to its users. Chang et al. [ 19 ] adopted a cloud computing technology for building a TV program recommender system. The system designed for digital TV programs was implemented using Hadoop Fair Scheduler (HFC), K-means clustering and k-nearest neighbour (KNN) algorithms. It was successful in processing huge amounts of real-time user data. Lucas et al. [ 20 ] implemented a recommender model for assisting a tourism application by using associative classification and fuzzy logic to predict the context. Niu et al. [ 21 ] introduced “Affivir: An Affect-based Internet Video Recommendation System” which was developed by calculating user preferences and by using spectral clustering. This model recommended videos with similar effects, which was processed to get optimal results with dynamic adjustments of recommendation constraints.

In 2014, Liu et al. [ 22 ] implemented a new route recommendation model for offering personalized and real-time route recommendations for self-driven tourists to minimize the queuing time and traffic jams infamous tourist places. Recommendations were carried out by considering the preferences of users. Bakshi et al. [ 23 ] proposed an unsupervised learning-based recommender model for solving the scalability problem of recommender systems. The algorithm used transitive similarities along with Particle Swarm Optimization (PSO) technique for discovering the global neighbours. Kim and Shim [ 24 ] proposed a recommender system based on “latent Dirichlet allocation using probabilistic modelling for Twitter” that could recommend the top-K tweets for a user to read, and the top-K users to follow. The model parameters were learned from an inference technique by using the differential Expectation–Maximization (EM) algorithm. Wang et al. [ 25 ] developed a hybrid-movie recommender model by aggregating a genetic algorithm (GA) with improved K-means and Principal Component Analysis (PCA) technique. It was able to offer intelligent movie recommendations with personalized suggestions. Kolomvatsos et al. [ 26 ] proposed a recommender system by considering an optimal stopping theory for delivering books or music recommendations to the users. Gottschlich et al. [ 27 ] proposed a decision support system for stock investment recommendations. It computed the output by considering the overall crowd’s recommendations. Torshizi et al. [ 28 ] have introduced a hybrid recommender system to determine the severity level of a medical condition. It could recommend suitable therapies for patients suffering from Benign Prostatic Hyperplasia.

In 2015, Zahálka et al. [ 29 ] proposed a venue recommender: “City Melange”. It was an interactive content-based model which used the convolutional deep-net features of the visual domain and the linear Support Vector Machine (SVM) model to capture the semantic information and extract latent topics. Sankar et al. [ 30 ] have proposed a stock recommender system based on the stock holding portfolio of trusted mutual funds. The system employed the collaborative filtering approach along with social network analysis for offering a decision support system to build a trust-based recommendation model. Chen et al. [ 31 ] have put forward a novel movie recommender system by applying the “artificial immune network to collaborative filtering” technique. It computed the affinity of an antigen and the affinity between an antibody and antigen. Based on this computation a similarity estimation formula was introduced which was used for the movie recommendation process. Wu et al. [ 32 ] have examined the technique of data fusion for increasing the efficiency of item recommender systems. It employed a hybrid linear combination model and used a collaborative tagging system. Yeh and Cheng [ 33 ] have proposed a recommender system for tourist attractions by constructing the “elicitation mechanism using the Delphi panel method and matrix construction mechanism using the repertory grids”, which was developed by considering the user preference and expert knowledge.

In 2016, Liao et al. [ 34 ] proposed a recommender model for online customers using a rough set association rule. The model computed the probable behavioural variations of online consumers and provided product category recommendations for e-commerce platforms. Li et al. [ 35 ] have suggested a movie recommender system based on user feedback collected from microblogs and social networks. It employed the sentiment-aware association rule mining algorithm for recommendations using the prior information of frequent program patterns, program metadata similarity and program view logs. Wu et al. [ 36 ] have developed a recommender system for social media platforms by aggregating the technique of Social Matrix Factorization (SMF) and Collaborative Topic Regression (CTR). The model was able to compute the ratings of users to items for making recommendations. For improving the recommendation quality, it gathered information from multiple sources such as item properties, social networks, feedback, etc. Adeniyi et al. [ 37 ] put forward a study of automated web-usage data mining and developed a recommender system that was tested in both real-time and online for identifying the visitor’s or client’s clickstream data.

In 2017, Rawat and Kankanhalli [ 38 ] have proposed a viewpoint recommender system called “ClickSmart” for assisting mobile users to capture high-quality photographs at famous tourist places. Yang et al. [ 39 ] proposed a gradient boosting-based job recommendation system for satisfying the cost-sensitive requirements of the users. The hybrid algorithm aimed to reduce the rate of unnecessary job recommendations. Lee et al. [ 40 ] proposed a music streaming recommender system based on smartphone activity usage. The proposed system benefitted by using feature selection approaches with machine learning techniques such as Naive Bayes (NB), Support Vector Machine (SVM), Multi-layer Perception (MLP), Instance-based k -Nearest Neighbour (IBK), and Random Forest (RF) for performing the activity detection from the mobile signals. Wei et al. [ 41 ] have proposed a new stacked denoising autoencoder (SDAE) based recommender system for cold items. The algorithm employed deep learning and collaborative filtering method to predict the unknown ratings.

In 2018, Li et al. [ 42 ] have developed a recommendation algorithm using Weighted Linear Regression Models (WLRRS). The proposed system was put to experiment using the MovieLens dataset and it presented better classification and predictive accuracy. Mezei and Nikou [ 43 ] presented a mobile health and wellness recommender system based on fuzzy optimization. It could recommend a collection of actions to be taken by the user to improve the user’s health condition. Recommendations were made considering the user’s physical activities and preferences. Ayata et al. [ 44 ] proposed a music recommendation model based on the user emotions captured through wearable physiological sensors. The emotion detection algorithm employed different machine learning algorithms like SVM, RF, KNN and decision tree (DT) algorithms to predict the emotions from the changing electrical signals gathered from the wearable sensors. Zhao et al. [ 45 ] developed a multimodal learning-based, social-aware movie recommender system. The model was able to successfully resolve the sparsity problem of recommender systems. The algorithm developed a heterogeneous network by exploiting the movie-poster image and textual description of each movie based on the social relationships and user ratings.

In 2019, Hammou et al. [ 46 ] proposed a Big Data recommendation algorithm capable of handling large scale data. The system employed random forest and matrix factorization through a data partitioning scheme. It was then used for generating recommendations based on user rating and preference for each item. The proposed system outperformed existing systems in terms of accuracy and speed. Zhao et al. [ 47 ] have put forward a hybrid initialization method for social network recommender systems. The algorithm employed denoising autoencoder (DAE) neural network-based initialization method (ANNInit) and attribute mapping. Bhaskaran and Santhi [ 48 ] have developed a hybrid, trust-based e-learning recommender system using cloud computing. The proposed algorithm was capable of learning online user activities by using the Firefly Algorithm (FA) and K-means clustering. Afolabi and Toivanen [ 59 ] have suggested an integrated recommender model based on collaborative filtering. The proposed model “Connected Health for Effective Management of Chronic Diseases”, aimed for integrating recommender systems for better decision-making in the process of disease management. He et al. [ 60 ] proposed a movie recommender system called “HI2Rec” which explored the usage of collaborative filtering and heterogeneous information for making movie recommendations. The model used the knowledge representation learning approach to embed movie-related information gathered from different sources.

In 2020, Han et al. [ 49 ] have proposed one Internet of Things (IoT)-based cancer rehabilitation recommendation system using the Beetle Antennae Search (BAS) algorithm. It presented the patients with a solution for the problem of optimal nutrition program by considering the objective function as the recurrence time. Kang et al. [ 50 ] have presented a recommender system for personalized advertisements in Online Broadcasting based on a tree model. Recommendations were generated in real-time by considering the user preferences to minimize the overhead of preference prediction and using a HashMap along with the tree characteristics. Ullah et al. [ 51 ] have implemented an image-based service recommendation model for online shopping based random forest and Convolutional Neural Networks (CNN). The model used JPEG coefficients to achieve an accurate prediction rate. Cai et al. [ 52 ] proposed a new hybrid recommender model using a many-objective evolutionary algorithm (MaOEA). The proposed algorithm was successful in optimizing the novelty, diversity, and accuracy of recommendations. Esteban et al. [ 53 ] have implemented a hybrid multi-criteria recommendation system concerned with students’ academic performance, personal interests, and course selection. The system was developed using a Genetic Algorithm (GA) and aimed at helping university students. It combined both course information and student information for increasing system performance and the reliability of the recommendations. Mondal et al. [ 54 ] have built a multilayer, graph data model-based doctor recommendation system by exploiting the trust concept between a patient-doctor relationship. The proposed system showed good results in practical applications.

In 2021, Dhelim et al. [ 55 ] have developed a personality-based product recommending model using the techniques of meta path discovery and user interest mining. This model showed better results when compared to session-based and deep learning models. Bhalse et al. [ 56 ] proposed a web-based movie recommendation system based on collaborative filtering using Singular Value Decomposition (SVD), collaborative filtering and cosine similarity (CS) for addressing the sparsity problem of recommender systems. It suggested a recommendation list by considering the content information of movies. Similarly, to solve both sparsity and cold-start problems Ke et al. [ 57 ] proposed a dynamic goods recommendation system based on reinforcement learning. The proposed system was capable of learning from the reduced entropy loss error on real-time applications. Chen et al. [ 58 ] have presented a movie recommender model combining various techniques like user interest with category-level representation, neighbour-assisted representation, user interest with latent representation and item-level representation using Feed-forward Neural Network (FNN).

Comparative chronological review

A comparative chronological review to compare the total contributions on various recommender systems in the past 10 years is given in Fig.  7 .

Comparative chronological review of recommender systems under diverse applications

This review puts forward a comparison of the number of research works proposed in the domain of recommender systems from the year 2011 to 2021 using various deep learning and machine learning-based approaches. Research articles are categorized based on the recommender system classification framework as shown in Table 5 . The articles are ordered according to their year of publication. There are two key concepts: Application fields and techniques used. The application fields of recommender systems are divided into six different fields, viz. entertainment, health, tourism, web/e-commerce, education and social media/others.

Algorithmic categorization, simulation platforms and applications considered for various recommender systems

This section analyses different methods like deep learning, machine learning, clustering and meta-heuristic-based-approaches used in the development of recommender systems. The algorithmic categorization of different recommender systems is given in Fig.  8 .

Algorithmic categorization of different recommender systems

Categorization is done based on content-based, collaborative filtering-based, and optimization-based approaches. In [ 8 ], a content-based filtering technique was employed for increasing the ability to trust other agents and for improving the exchange of information by trust degree. In [ 16 ], it was applied to enhance the quality of recommendations using the account attributes of the material. It achieved better performance concerning with F1-score, recall and precision. In [ 18 ], this technique was able to capture the implicit user feedback, increasing the overall accuracy of the proposed model. The content-based filtering in [ 30 ] was able to increase the accuracy and performance of a stock recommender system by using the “trust factor” for making decisions.

Different collaborative filtering approaches are utilized in recent studies, which are categorized as follows:

Model-based techniques

Neuro-Fuzzy [ 1 ] based technique helps in discovering the association between user categories and item relevance. It is also simple to understand. K-Means Clustering [ 2 , 19 , 25 , 48 ] is efficient for large scale datasets. It is simple to implement and gives a fast convergence rate. It also offers automatic recovery from failures. The decision tree [ 2 , 44 ] technique is easy to interpret. It can be used for solving the classic regression and classification problems in recommender systems. Bayesian Network [ 3 ] is a probabilistic technique used to solve classification challenges. It is based on the theory of Bayes theorem and conditional probability. Association Rule Mining (ARM) techniques [ 4 , 17 , 35 ] extract rules for projecting the occurrence of an item by considering the existence of other items in a transaction. This method uses the association rules to create a more suitable representation of data and helps in increasing the model performance and storage efficiency. Fuzzy Logic [ 5 , 7 , 15 , 20 , 28 , 43 ] techniques use a set of flexible rules. It focuses on solving complex real-time problems having an inaccurate spectrum of data. This technique provides scalability and helps in increasing the overall model performance for recommender systems. The semantic similarity [ 6 ] technique is used for describing a topological similarity to define the distance among the concepts and terms through ontologies. It measures the similarity information for increasing the efficiency of recommender systems. Rough set [ 9 , 34 ] techniques use probability distributions for solving the challenges of existing recommender models. Semantic web rule language [ 10 ] can efficiently extract the dataset features and increase the model efficiency. Linear programming-based approaches [ 13 , 42 ] are employed for achieving quality decision making in recommender models. Sequential pattern analysis [ 14 ] is applied to find suitable patterns among data items. This helps in increasing model efficiency. The probabilistic model [ 24 ] is a famous tool to handle uncertainty in risk computations and performance assessment. It offers better decision-making capabilities. K-nearest neighbours (KNN) [ 19 , 37 , 44 ] technique provides faster computation time, simplicity and ease of interpretation. They are good for classification and regression-based problems and offers more accuracy. Spectral clustering [ 21 ] is also called graph clustering or similarity-based clustering, which mainly focuses on reducing the space dimensionality in identifying the dataset items. Stochastic learning algorithm [ 26 ] solves the real-time challenges of recommender systems. Linear SVM [ 29 , 44 ] efficiently solves the high dimensional problems related to recommender systems. It is a memory-efficient method and works well with a large number of samples having relative separation among the classes. This method has been shown to perform well even when new or unfamiliar data is added. Relational Functional Gradient Boosting [ 39 ] technique efficiently works on the relational dependency of data, which is useful for statical relational learning for collaborative-based recommender systems. Ensemble learning [ 40 ] combines the forecast of two or more models and aims to achieve better performance than any of the single contributing models. It also helps in reducing overfitting problems, which are common in recommender systems.

SDAE [ 41 ] is used for learning the non-linear transformations with different filters for finding suitable data. This aids in increasing the performance of recommender models. Multimodal network learning [ 45 ] is efficient for multi-modal data, representing a combined representation of diverse modalities. Random forest [ 46 , 51 ] is a commonly used approach in comparison with other classifiers. It has been shown to increase accuracy when handling big data. This technique is a collection of decision trees to minimize variance through training on diverse data samples. ANNInit [ 47 ] is a type of artificial neural network-based technique that has the capability of self-learning and generating efficient results. It is independent of the data type and can learn data patterns automatically. HashMap [ 50 ] gives faster access to elements owing to the hashing methodology, which decreases the data processing time and increases the performance of the system. CNN [ 51 ] technique can automatically fetch the significant features of a dataset without any supervision. It is a computationally efficient method and provides accurate recommendations. This technique is also simple and fast for implementation. Multilayer graph data model [ 54 ] is efficient for real-time applications and minimizes the access time through mapping the correlation as edges among nodes and provides superior performance. Singular Value Decomposition [ 56 ] can simplify the input data and increase the efficiency of recommendations by eliminating the noise present in data. Reinforcement learning [ 57 ] is efficient for practical scenarios of recommender systems having large data sizes. It is capable of boosting the model performance by increasing the model accuracy even for large scale datasets. FNN [ 58 ] is one of the artificial neural network techniques which can learn non-linear and complex relationships between items. It has demonstrated a good performance increase when employed in different recommender systems. Knowledge representation learning [ 60 ] systems aim to simplify the model development process by increasing the acquisition efficiency, inferential efficiency, inferential adequacy and representation adequacy. User-based approaches [ 2 , 55 , 59 ] specialize in detecting user-related meta-data which is employed to increase the overall model performance. This technique is more suitable for real-time applications where it can capture user feedback and use it to increase the user experience.

Optimization-based techniques

The Foraging Bees [ 11 ] technique enables both functional and combinational optimization for random searching in recommender models. Artificial bee colony [ 12 ] is a swarm-based meta-heuristic technique that provides features like faster convergence rate, the ability to handle the objective with stochastic nature, ease for incorporating with other algorithms, usage of fewer control parameters, strong robustness, high flexibility and simplicity. Particle Swarm Optimization [ 23 ] is a computation optimization technique that offers better computational efficiency, robustness in control parameters, and is easy and simple to implement in recommender systems. Portfolio optimization algorithm [ 27 ] is a subclass of optimization algorithms that find its application in stock investment recommender systems. It works well in real-time and helps in the diversification of the portfolio for maximum profit. The artificial immune system [ 31 ]a is computationally intelligent machine learning technique. This technique can learn new patterns in the data and optimize the overall system parameters. Expectation maximization (EM) [ 32 , 36 , 38 ] is an iterative algorithm that guarantees the likelihood of finding the maximum parameters when the input variables are unknown. Delphi panel and repertory grid [ 33 ] offers efficient decision making by solving the dimensionality problem and data sparsity issues of recommender systems. The Firefly algorithm (FA) [ 48 ] provides fast results and increases recommendation efficiency. It is capable of reducing the number of iterations required to solve specific recommender problems. It also provides both local and global sets of solutions. Beetle Antennae Search (BAS) [ 49 ] offers superior search accuracy and maintains less time complexity that promotes the performance of recommendations. Many-objective evolutionary algorithm (MaOEA) [ 52 ] is applicable for real-time, multi-objective, search-related recommender systems. The introduction of a local search operator increases the convergence rate and gets suitable results. Genetic Algorithm (GA) [ 2 , 22 , 25 , 53 ] based techniques are used to solve the multi-objective optimization problems of recommender systems. They employ probabilistic transition rules and have a simpler operation that provides better recommender performance.

Features and challenges

The features and challenges of the existing recommender models are given in Table 6 .

Simulation platforms

The various simulation platforms used for developing different recommender systems with different applications are given in Fig.  9 .

Simulation platforms used for developing different recommender systems

Here, the Java platform is used in 20% of the contributions, MATLAB is implemented in 7% of the contributions, different fold cross-validation are used in 8% of the contributions, 7% of the contributions are utilized by the python platform, 3% of the contributions employ R-programming and 1% of the contributions are developed by Tensorflow, Weka and Android environments respectively. Other simulation platforms like Facebook, web UI (User Interface), real-time environments, etc. are used in 50% of the contributions. Table 7 describes some simulation platforms commonly used for developing recommender systems.

Application focused and dataset description

This section provides an analysis of the different applications focused on a set of recent recommender systems and their dataset details.

Recent recommender systems were analysed and found that 11% of the contributions are focused on the domain of healthcare, 10% of the contributions are on movie recommender systems, 5% of the contributions come from music recommender systems, 6% of the contributions are focused on e-learning recommender systems, 8% of the contributions are used for online product recommender systems, 3% of the contributions are focused on book recommendations and 1% of the contributions are focused on Job and knowledge management recommender systems. 5% of the contributions concentrated on social network recommender systems, 10% of the contributions are focused on tourist and hotels recommender systems, 6% of the contributions are employed for stock recommender systems, and 3% of the contributions contributed for video recommender systems. The remaining 12% of contributions are miscellaneous recommender systems like Twitter, venue-based recommender systems, etc. Similarly, different datasets are gathered for recommender systems based on their application types. A detailed description is provided in Table 8 .

Performance analysis of state-of-art recommender systems

The performance evaluation metrics used for the analysis of different recommender systems is depicted in Table 9 . From the set of research works, 35% of the works use recall measure, 16% of the works employ Mean Absolute Error (MAE), 11% of the works take Root Mean Square Error (RMSE), 41% of the papers consider precision, 30% of the contributions analyse F1-measure, 31% of the works apply accuracy and 6% of the works employ coverage measure to validate the performance of the recommender systems. Moreover, some additional measures are also considered for validating the performance in a few applications.

Research gaps and challenges

In the recent decade, recommender systems have performed well in solving the problem of information overload and has become the more appropriate tool for multiple areas such as psychology, mathematics, computer science, etc. [ 80 ]. However, current recommender systems face a variety of challenges which are stated as follows, and discussed below:

Deployment challenges such as cold start, scalability, sparsity, etc. are already discussed in Sect. 3.

Challenges faced when employing different recommender algorithms for different applications.

Challenges in collecting implicit user data

Challenges in handling real-time user feedback.

Challenges faced in choosing the correct implementation techniques.

Challenges faced in measuring system performance.

Challenges in implementing recommender system for diverse applications.

Numerous recommender algorithms have been proposed on novel emerging dimensions which focus on addressing the existing limitations of recommender systems. A good recommender system must increase the recommendation quality based on user preferences. However, a specific recommender algorithm is not always guaranteed to perform equally for different applications. This encourages the possibility of employing different recommender algorithms for different applications, which brings along a lot of challenges. There is a need for more research to alleviate these challenges. Also, there is a large scope of research in recommender applications that incorporate information from different interactive online sites like Facebook, Twitter, shopping sites, etc. Some other areas for emerging research may be in the fields of knowledge-based recommender systems, methods for seamlessly processing implicit user data and handling real-time user feedback to recommend items in a dynamic environment.

Some of the other research areas like deep learning-based recommender systems, demographic filtering, group recommenders, cross-domain techniques for recommender systems, and dimensionality reduction techniques are also further required to be studied [ 83 ]. Deep learning-based recommender systems have recently gained much popularity. Future research areas in this field can integrate the well-performing deep learning models with new variants of hybrid meta-heuristic approaches.

During this review, it was observed that even though recent recommender systems have demonstrated good performance, there is no single standardized criteria or method which could be used to evaluate the performance of all recommender systems. System performance is generally measured by different evaluation matrices which makes it difficult to compare. The application of recommender systems in real-time applications is growing. User satisfaction and personalization play a very important role in the success of such recommender systems. There is a need for some new evaluation criteria which can evaluate the level of user satisfaction in real-time. New research should focus on capturing real-time user feedback and use the information to change the recommendation process accordingly. This will aid in increasing the quality of recommendations.

Conclusion and future scope

Recommender systems have attracted the attention of researchers and academicians. In this paper, we have identified and prudently reviewed research papers on recommender systems focusing on diverse applications, which were published between 2011 and 2021. This review has gathered diverse details like different application fields, techniques used, simulation tools used, diverse applications focused, performance metrics, datasets used, system features, and challenges of different recommender systems. Further, the research gaps and challenges were put forward to explore the future research perspective on recommender systems. Overall, this paper provides a comprehensive understanding of the trend of recommender systems-related research and to provides researchers with insight and future direction on recommender systems. The results of this study have several practical and significant implications:

Based on the recent-past publication rates, we feel that the research of recommender systems will significantly grow in the future.

A large number of research papers were identified in movie recommendations, whereas health, tourism and education-related recommender systems were identified in very few numbers. This is due to the availability of movie datasets in the public domain. Therefore, it is necessary to develop datasets in other fields also.

There is no standard measure to compute the performance of recommender systems. Among 60 papers, 21 used recall, 10 used MAE, 25 used precision, 18 used F1-measure, 19 used accuracy and only 7 used RMSE to calculate system performance. Very few systems were found to excel in two or more matrices.

Java and Python (with a combined contribution of 27%) are the most common programming languages used to develop recommender systems. This is due to the availability of a large number of standard java and python libraries which aid in the development process.

Recently a large number of hybrid and optimizations techniques are being proposed for recommender systems. The performance of a recommender system can be greatly improved by applying optimization techniques.

There is a large scope of research in using neural networks and deep learning-based methods for developing recommender systems. Systems developed using these methods are found to achieve high-performance accuracy.

This research will provide a guideline for future research in the domain of recommender systems. However, this research has some limitations. Firstly, due to the limited amount of manpower and time, we have only reviewed papers published in journals focusing on computer science, management and medicine. Secondly, we have reviewed only English papers. New research may extend this study to cover other journals and non-English papers. Finally, this review was conducted based on a search on only six descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*” and “Hybrid Recommend*”. Research papers that did not include these keywords were not considered. Future research can include adding some additional descriptors and keywords for searching. This will allow extending the research to cover more diverse articles on recommender systems.

Availability of data and materials

Not applicable.

Castellano G, Fanelli AM, Torsello MA. NEWER: A system for neuro-fuzzy web recommendation. Appl Soft Comput. 2011;11:793–806.

Article   Google Scholar  

Crespo RG, Martínez OS, Lovelle JMC, García-Bustelo BCP, Gayo JEL, Pablos PO. Recommendation system based on user interaction data applied to intelligent electronic books. Computers Hum Behavior. 2011;27:1445–9.

Lin FC, Yu HW, Hsu CH, Weng TC. Recommendation system for localized products in vending machines. Expert Syst Appl. 2011;38:9129–38.

Wang SL, Wu CY. Application of context-aware and personalized recommendation to implement an adaptive ubiquitous learning system. Expert Syst Appl. 2011;38:10831–8.

García-Crespo Á, López-Cuadrado JL, Colomo-Palacios R, González-Carrasco I, Ruiz-Mezcua B. Sem-Fit: A semantic based expert system to provide recommendations in the tourism domain. Expert Syst Appl. 2011;38:13310–9.

Dong H, Hussain FK, Chang E. A service concept recommendation system for enhancing the dependability of semantic service matchmakers in the service ecosystem environment. J Netw Comput Appl. 2011;34:619–31.

Li M, Liu L, Li CB. An approach to expert recommendation based on fuzzy linguistic method and fuzzy text classification in knowledge management systems. Expert Syst Appl. 2011;38:8586–96.

Lorenzi F, Bazzan ALC, Abel M, Ricci F. Improving recommendations through an assumption-based multiagent approach: An application in the tourism domain. Expert Syst Appl. 2011;38:14703–14.

Huang Z, Lu X, Duan H. Context-aware recommendation using rough set model and collaborative filtering. Artif Intell Rev. 2011;35:85–99.

Chen RC, Huang YH, Bau CT, Chen SM. A recommendation system based on domain ontology and SWRL for anti-diabetic drugs selection. Expert Syst Appl. 2012;39:3995–4006.

Mohanraj V, Chandrasekaran M, Senthilkumar J, Arumugam S, Suresh Y. Ontology driven bee’s foraging approach based self-adaptive online recommendation system. J Syst Softw. 2012;85:2439–50.

Hsu CC, Chen HC, Huang KK, Huang YM. A personalized auxiliary material recommendation system based on learning style on facebook applying an artificial bee colony algorithm. Comput Math Appl. 2012;64:1506–13.

Gemmell J, Schimoler T, Mobasher B, Burke R. Resource recommendation in social annotation systems: A linear-weighted hybrid approach. J Comput Syst Sci. 2012;78:1160–74.

Article   MathSciNet   Google Scholar  

Choi K, Yoo D, Kim G, Suh Y. A hybrid online-product recommendation system: Combining implicit rating-based collaborative filtering and sequential pattern analysis. Electron Commer Res Appl. 2012;11:309–17.

Garibaldi JM, Zhou SM, Wang XY, John RI, Ellis IO. Incorporation of expert variability into breast cancer treatment recommendation in designing clinical protocol guided fuzzy rule system models. J Biomed Inform. 2012;45:447–59.

Salehi M, Kmalabadi IN. A hybrid attribute–based recommender system for e–learning material recommendation. IERI Procedia. 2012;2:565–70.

Aher SB, Lobo LMRJ. Combination of machine learning algorithms for recommendation of courses in e-learning System based on historical data. Knowl-Based Syst. 2013;51:1–14.

Kardan AA, Ebrahimi M. A novel approach to hybrid recommendation systems based on association rules mining for content recommendation in asynchronous discussion groups. Inf Sci. 2013;219:93–110.

Chang JH, Lai CF, Wang MS, Wu TY. A cloud-based intelligent TV program recommendation system. Comput Electr Eng. 2013;39:2379–99.

Lucas JP, Luz N, Moreno MN, Anacleto R, Figueiredo AA, Martins C. A hybrid recommendation approach for a tourism system. Expert Syst Appl. 2013;40:3532–50.

Niu J, Zhu L, Zhao X, Li H. Affivir: An affect-based Internet video recommendation system. Neurocomputing. 2013;120:422–33.

Liu L, Xu J, Liao SS, Chen H. A real-time personalized route recommendation system for self-drive tourists based on vehicle to vehicle communication. Expert Syst Appl. 2014;41:3409–17.

Bakshi S, Jagadev AK, Dehuri S, Wang GN. Enhancing scalability and accuracy of recommendation systems using unsupervised learning and particle swarm optimization. Appl Soft Comput. 2014;15:21–9.

Kim Y, Shim K. TWILITE: A recommendation system for twitter using a probabilistic model based on latent Dirichlet allocation. Inf Syst. 2014;42:59–77.

Wang Z, Yu X, Feng N, Wang Z. An improved collaborative movie recommendation system using computational intelligence. J Vis Lang Comput. 2014;25:667–75.

Kolomvatsos K, Anagnostopoulos C, Hadjiefthymiades S. An efficient recommendation system based on the optimal stopping theory. Expert Syst Appl. 2014;41:6796–806.

Gottschlich J, Hinz O. A decision support system for stock investment recommendations using collective wisdom. Decis Support Syst. 2014;59:52–62.

Torshizi AD, Zarandi MHF, Torshizi GD, Eghbali K. A hybrid fuzzy-ontology based intelligent system to determine level of severity and treatment recommendation for benign prostatic hyperplasia. Comput Methods Programs Biomed. 2014;113:301–13.

Zahálka J, Rudinac S, Worring M. Interactive multimodal learning for venue recommendation. IEEE Trans Multimedia. 2015;17:2235–44.

Sankar CP, Vidyaraj R, Kumar KS. Trust based stock recommendation system – a social network analysis approach. Procedia Computer Sci. 2015;46:299–305.

Chen MH, Teng CH, Chang PC. Applying artificial immune systems to collaborative filtering for movie recommendation. Adv Eng Inform. 2015;29:830–9.

Wu H, Pei Y, Li B, Kang Z, Liu X, Li H. Item recommendation in collaborative tagging systems via heuristic data fusion. Knowl-Based Syst. 2015;75:124–40.

Yeh DY, Cheng CH. Recommendation system for popular tourist attractions in Taiwan using delphi panel and repertory grid techniques. Tour Manage. 2015;46:164–76.

Liao SH, Chang HK. A rough set-based association rule approach for a recommendation system for online consumers. Inf Process Manage. 2016;52:1142–60.

Li H, Cui J, Shen B, Ma J. An intelligent movie recommendation system through group-level sentiment analysis in microblogs. Neurocomputing. 2016;210:164–73.

Wu H, Yue K, Pei Y, Li B, Zhao Y, Dong F. Collaborative topic regression with social trust ensemble for recommendation in social media systems. Knowl-Based Syst. 2016;97:111–22.

Adeniyi DA, Wei Z, Yongquan Y. Automated web usage data mining and recommendation system using K-Nearest Neighbor (KNN) classification method. Appl Computing Inform. 2016;12:90–108.

Rawat YS, Kankanhalli MS. ClickSmart: A context-aware viewpoint recommendation system for mobile photography. IEEE Trans Circuits Syst Video Technol. 2017;27:149–58.

Yang S, Korayem M, Aljadda K, Grainger T, Natarajan S. Combining content-based and collaborative filtering for job recommendation system: A cost-sensitive Statistical Relational Learning approach. Knowl-Based Syst. 2017;136:37–45.

Lee WP, Chen CT, Huang JY, Liang JY. A smartphone-based activity-aware system for music streaming recommendation. Knowl-Based Syst. 2017;131:70–82.

Wei J, He J, Chen K, Zhou Y, Tang Z. Collaborative filtering and deep learning based recommendation system for cold start items. Expert Syst Appl. 2017;69:29–39.

Li C, Wang Z, Cao S, He L. WLRRS: A new recommendation system based on weighted linear regression models. Comput Electr Eng. 2018;66:40–7.

Mezei J, Nikou S. Fuzzy optimization to improve mobile health and wellness recommendation systems. Knowl-Based Syst. 2018;142:108–16.

Ayata D, Yaslan Y, Kamasak ME. Emotion based music recommendation system using wearable physiological sensors. IEEE Trans Consum Electron. 2018;64:196–203.

Zhao Z, Yang Q, Lu H, Weninger T. Social-aware movie recommendation via multimodal network learning. IEEE Trans Multimedia. 2018;20:430–40.

Hammou BA, Lahcen AA, Mouline S. An effective distributed predictive model with matrix factorization and random forest for big data recommendation systems. Expert Syst Appl. 2019;137:253–65.

Zhao J, Geng X, Zhou J, Sun Q, Xiao Y, Zhang Z, Fu Z. Attribute mapping and autoencoder neural network based matrix factorization initialization for recommendation systems. Knowl-Based Syst. 2019;166:132–9.

Bhaskaran S, Santhi B. An efficient personalized trust based hybrid recommendation (TBHR) strategy for e-learning system in cloud computing. Clust Comput. 2019;22:1137–49.

Han Y, Han Z, Wu J, Yu Y, Gao S, Hua D, Yang A. Artificial intelligence recommendation system of cancer rehabilitation scheme based on IoT technology. IEEE Access. 2020;8:44924–35.

Kang S, Jeong C, Chung K. Tree-based real-time advertisement recommendation system in online broadcasting. IEEE Access. 2020;8:192693–702.

Ullah F, Zhang B, Khan RU. Image-based service recommendation system: A JPEG-coefficient RFs approach. IEEE Access. 2020;8:3308–18.

Cai X, Hu Z, Zhao P, Zhang W, Chen J. A hybrid recommendation system with many-objective evolutionary algorithm. Expert Syst Appl. 2020. https://doi.org/10.1016/j.eswa.2020.113648 .

Esteban A, Zafra A, Romero C. Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization. Knowledge-Based Syst. 2020;194:105385.

Mondal S, Basu A, Mukherjee N. Building a trust-based doctor recommendation system on top of multilayer graph database. J Biomed Inform. 2020;110:103549.

Dhelim S, Ning H, Aung N, Huang R, Ma J. Personality-aware product recommendation system based on user interests mining and metapath discovery. IEEE Trans Comput Soc Syst. 2021;8:86–98.

Bhalse N, Thakur R. Algorithm for movie recommendation system using collaborative filtering. Materials Today: Proceedings. 2021. https://doi.org/10.1016/j.matpr.2021.01.235 .

Ke G, Du HL, Chen YC. Cross-platform dynamic goods recommendation system based on reinforcement learning and social networks. Appl Soft Computing. 2021;104:107213.

Chen X, Liu D, Xiong Z, Zha ZJ. Learning and fusing multiple user interest representations for micro-video and movie recommendations. IEEE Trans Multimedia. 2021;23:484–96.

Afolabi AO, Toivanen P. Integration of recommendation systems into connected health for effective management of chronic diseases. IEEE Access. 2019;7:49201–11.

He M, Wang B, Du X. HI2Rec: Exploring knowledge in heterogeneous information for movie recommendation. IEEE Access. 2019;7:30276–84.

Bobadilla J, Serradilla F, Hernando A. Collaborative filtering adapted to recommender systems of e-learning. Knowl-Based Syst. 2009;22:261–5.

Russell S, Yoon V. Applications of wavelet data reduction in a recommender system. Expert Syst Appl. 2008;34:2316–25.

Campos LM, Fernández-Luna JM, Huete JF. A collaborative recommender system based on probabilistic inference from fuzzy observations. Fuzzy Sets Syst. 2008;159:1554–76.

Funk M, Rozinat A, Karapanos E, Medeiros AKA, Koca A. In situ evaluation of recommender systems: Framework and instrumentation. Int J Hum Comput Stud. 2010;68:525–47.

Porcel C, Moreno JM, Herrera-Viedma E. A multi-disciplinar recommender system to advice research resources in University Digital Libraries. Expert Syst Appl. 2009;36:12520–8.

Bobadilla J, Serradilla F, Bernal J. A new collaborative filtering metric that improves the behavior of recommender systems. Knowl-Based Syst. 2010;23:520–8.

Ochi P, Rao S, Takayama L, Nass C. Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int J Hum Comput Stud. 2010;68:472–82.

Olmo FH, Gaudioso E. Evaluation of recommender systems: A new approach. Expert Syst Appl. 2008;35:790–804.

Zhen L, Huang GQ, Jiang Z. An inner-enterprise knowledge recommender system. Expert Syst Appl. 2010;37:1703–12.

Göksedef M, Gündüz-Öğüdücü S. Combination of web page recommender systems. Expert Syst Appl. 2010;37(4):2911–22.

Shao B, Wang D, Li T, Ogihara M. Music recommendation based on acoustic features and user access patterns. IEEE Trans Audio Speech Lang Process. 2009;17:1602–11.

Shin C, Woo W. Socially aware tv program recommender for multiple viewers. IEEE Trans Consum Electron. 2009;55:927–32.

Lopez-Carmona MA, Marsa-Maestre I, Perez JRV, Alcazar BA. Anegsys: An automated negotiation based recommender system for local e-marketplaces. IEEE Lat Am Trans. 2007;5:409–16.

Yap G, Tan A, Pang H. Discovering and exploiting causal dependencies for robust mobile context-aware recommenders. IEEE Trans Knowl Data Eng. 2007;19:977–92.

Meo PD, Quattrone G, Terracina G, Ursino D. An XML-based multiagent system for supporting online recruitment services. IEEE Trans Syst Man Cybern. 2007;37:464–80.

Khusro S, Ali Z, Ullah I. Recommender systems: Issues, challenges, and research opportunities. Inform Sci Appl. 2016. https://doi.org/10.1007/978-981-10-0557-2_112 .

Blanco-Fernandez Y, Pazos-Arias JJ, Gil-Solla A, Ramos-Cabrer M, Lopez-Nores M. Providing entertainment by content-based filtering and semantic reasoning in intelligent recommender systems. IEEE Trans Consum Electron. 2008;54:727–35.

Isinkaye FO, Folajimi YO, Ojokoh BA. Recommendation systems: Principles, methods and evaluation. Egyptian Inform J. 2015;16:261–73.

Yoshii K, Goto M, Komatani K, Ogata T, Okuno HG. An efficient hybrid music recommender system using an incrementally trainable probabilistic generative model. IEEE Trans Audio Speech Lang Process. 2008;16:435–47.

Wei YZ, Moreau L, Jennings NR. Learning users’ interests by quality classification in market-based recommender systems. IEEE Trans Knowl Data Eng. 2005;17:1678–88.

Bjelica M. Towards TV recommender system: experiments with user modeling. IEEE Trans Consum Electron. 2010;56:1763–9.

Setten MV, Veenstra M, Nijholt A, Dijk BV. Goal-based structuring in recommender systems. Interact Comput. 2006;18:432–56.

Adomavicius G, Tuzhilin A. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng. 2005;17:734–49.

Symeonidis P, Nanopoulos A, Manolopoulos Y. Providing justifications in recommender systems. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans. 2009;38:1262–72.

Zhan J, Hsieh C, Wang I, Hsu T, Liau C, Wang D. Privacy preserving collaborative recommender systems. IEEE Trans Syst Man Cybernet. 2010;40:472–6.

Burke R. Hybrid recommender systems: survey and experiments. User Model User-Adap Inter. 2002;12:331–70.

Article   MATH   Google Scholar  

Gunes I, Kaleli C, Bilge A, Polat H. Shilling attacks against recommender systems: a comprehensive survey. Artif Intell Rev. 2012;42:767–99.

Park DH, Kim HK, Choi IY, Kim JK. A literature review and classification of recommender systems research. Expert Syst Appl. 2012;39:10059–72.

Download references

Acknowledgements

We thank our colleagues from Assam Down Town University who provided insight and expertise that greatly assisted this research, although they may not agree with all the interpretations and conclusions of this paper.

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and affiliations.

Department of Computer Science & Engineering, Assam Down Town University, Panikhaiti, Guwahati, 781026, Assam, India

Deepjyoti Roy & Mala Dutta

You can also search for this author in PubMed   Google Scholar

Contributions

DR carried out the review study and analysis of the existing algorithms in the literature. MD has been involved in drafting the manuscript or revising it critically for important intellectual content. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Deepjyoti Roy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Roy, D., Dutta, M. A systematic review and research perspective on recommender systems. J Big Data 9 , 59 (2022). https://doi.org/10.1186/s40537-022-00592-5

Download citation

Received : 04 October 2021

Accepted : 28 March 2022

Published : 03 May 2022

DOI : https://doi.org/10.1186/s40537-022-00592-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Recommender system
• Machine learning
• Content-based filtering
• Collaborative filtering
• Deep learning

Evaluating Recommender Systems: Survey and Framework

New Citation Alert added!

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, 1 introduction, 2 conceptual basis, 2.1 recommender systems.

2.2 Evaluation of Recommender Systems

3 recommender systems evaluation: a review.

3.1 Evaluation Objectives

3.2 Evaluation Design Space: Evaluation Principles

3.3 evaluation design space: experiment type.

3.3.1 Offline Evaluation.

3.3.2 user study., 3.3.3 online evaluation., 3.4 evaluation design space: evaluation aspects, 3.4.1 types of data..

3.4.2 Data Collection.

3.4.3 Data Quality and Biases.

3.4.4 evaluation metrics..

3.4.5 Evaluation System.

4 mapping a fictitious case to fevr, 4.1 mapping to fevr.

4.2 Limitations and Discussion

5 discussion, conclusion, and future directions.

• Behare N Jeet D (2024) The Art and Science of User Engagement The Rise of Over-the-Top (OTT) Media and Implications for Media Consumption and Production 10.4018/979-8-3693-0116-6.ch009 (130-159) Online publication date: 26-Jan-2024 https://doi.org/10.4018/979-8-3693-0116-6.ch009
• de Miguel J Velázquez-Iturbide J (2024) What Is an Algorithm? Encyclopedia of Information Science and Technology, Sixth Edition 10.4018/978-1-6684-7366-5.ch072 (1-17) Online publication date: 1-Jul-2024 https://doi.org/10.4018/978-1-6684-7366-5.ch072
• Slade E Rennick-Egglestone S Ng F Kotera Y Llewellyn-Beardsley J Newby C Glover T Keppens J Slade M (2024) The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Use and Performance JMIR Mental Health 10.2196/45754 11 (e45754) Online publication date: 29-Mar-2024 https://doi.org/10.2196/45754
• Show More Cited By

Index Terms

Human-centered computing

Human computer interaction (HCI)

HCI design and evaluation methods

Information systems

Information retrieval

Evaluation of retrieval results

Retrieval tasks and goals

Recommender systems

Recommendations

A survey of serendipity in recommender systems.

We summarize most efforts on serendipity in recommender systems.We compare definitions of serendipity in recommender systems.We classify the state-of-the-art serendipity-oriented recommendation algorithms.We review methods to assess serendipity in ...

Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols

Exploiting temporal context has been proved to be an effective approach to improve recommendation performance, as shown, e.g. in the Netflix Prize competition. Time-aware recommender systems (TARS) are indeed receiving increasing attention. A wide range ...

Evaluating collaborative filtering recommender systems

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being ...

Information

Published in.

University of Sydney, Australia

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates, author tags.

• Framework for EValuating Recommender systems

Funding Sources

• Austrian Science Fund (FWF)

Contributors

Other metrics, bibliometrics, article metrics.

• 61 Total Citations View Citations
• 19,620 Total Downloads
• Downloads (Last 12 months) 13,585
• Downloads (Last 6 weeks) 1,268
• Bertino A (2024) Serendipität und Open Data. Richtlinien für die Implementierung von Empfehlungssystemen auf Entdeckungsplattformen Bibliothek Forschung und Praxis 10.1515/bfp-2023-0059 Online publication date: 1-Jul-2024 https://doi.org/10.1515/bfp-2023-0059
• Fan Y Ji Y Zhang J Sun A (2024) Our Model Achieves Excellent Performance on MovieLens: What Does It Mean? ACM Transactions on Information Systems 10.1145/3675163 Online publication date: 1-Jul-2024 https://dl.acm.org/doi/10.1145/3675163
• Zeng Z Wang Y Zhao Y Shi W (2024) A Survey of Music Recommendation Systems Proceedings of the 5th International Conference on Computer Information and Big Data Applications 10.1145/3671151.3671243 (507-519) Online publication date: 26-Apr-2024 https://dl.acm.org/doi/10.1145/3671151.3671243
• Deffayet R Thonet T Hwang D Lehoux V Renders J de Rijke M (2024) SARDINE: Simulator for Automated Recommendation in Dynamic and Interactive Environments ACM Transactions on Recommender Systems 10.1145/3656481 2 :3 (1-34) Online publication date: 5-Jun-2024 https://dl.acm.org/doi/10.1145/3656481
• Jeunen O Potapov I Ustimenko A Baeza-Yates R Bonchi F (2024) On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining 10.1145/3637528.3671687 (1222-1233) Online publication date: 25-Aug-2024 https://dl.acm.org/doi/10.1145/3637528.3671687
• Vrijenhoek S Daniil S Sandel J Hollink L (2024) Diversity of What? On the Different Conceptualizations of Diversity in Recommender Systems Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency 10.1145/3630106.3658926 (573-584) Online publication date: 3-Jun-2024 https://dl.acm.org/doi/10.1145/3630106.3658926
• Bauer C Zangerle E Said A (2024) Exploring the Landscape of Recommender Systems Evaluation: Practices and Perspectives ACM Transactions on Recommender Systems 10.1145/3629170 2 :1 (1-31) Online publication date: 7-Mar-2024 https://dl.acm.org/doi/10.1145/3629170

View options

View or Download as a PDF file.

View online with eReader .

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Journal Proposal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

A survey of recommendation systems: recommendation models, techniques, and application fields.

Graphical Abstract

1. Introduction

2. literature surveys, 2.1. recommendation models, 2.1.1. content-based filtering, 2.1.2. collaborative filtering, 2.1.3. hybrid system, 2.1.4. qualitative evaluation metrics of recommendation systems, 2.1.5. research trend of recommendation models, 2.2. recommendation techniques, 2.2.1. text mining, 2.2.2. knn (k-nearest neighbor), 2.2.3. clustering, 2.2.4. matrix factorization, 2.2.5. neural network, 2.2.6. research trends of recommendation system techniques, 2.3. application fields, 2.3.1. streaming service, 2.3.2. social network service, 2.3.3. tourism service, 2.3.4. e-commerce service, 2.3.5. healthcare service, 2.3.6. education service, 2.3.7. academic information service, 2.4. recommendation system research trend, 2.4.1. streaming service and research trend, 2.4.2. social network service and research trends, 2.4.3. tourism service and research trend, 2.4.4. e-commerce service and research trends, 2.4.5. healthcare service and research trends, 2.4.6. education service and research trends, 2.4.7. summary of research trends, 3. conclusions, author contributions, data availability statement, conflicts of interest.

• Beheshti, A.; Yakhchi, S.; Mousaeirad, S.; Ghafari, S.M.; Goluguri, S.R.; Edrisi, M.A. Towards Cognitive Recommender Systems. Algorithms 2020 , 13 , 176. [ Google Scholar ] [ CrossRef ]
• Abbasi-Moud, Z.; Vahdat-Nejad, H.; Sadri, J. Tourism Recommendation System Based on Semantic Clustering and Sentiment Analysis. Expert Syst. Appl. 2021 , 167 , 114324. [ Google Scholar ] [ CrossRef ]
• Liu, F.; Lee, H.J. Use of Social Network Information to Enhance Collaborative Filtering Performance. Expert Syst. Appl. 2010 , 37 , 4772–4778. [ Google Scholar ] [ CrossRef ]
• Yang, B.; Lei, Y.; Liu, J.; Li, W. Social Collaborative Filtering by Trust. IEEE Trans. Pattern Anal. Mach. Intell. 2017 , 39 , 1633–1647. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Amato, F.; Moscato, V.; Picariello, A.; Piccialli, F. SOS: A Multimedia Recommender System for Online Social Networks. Future Gener. Comput. Syst. 2019 , 93 , 914–923. [ Google Scholar ] [ CrossRef ]
• Capdevila, J.; Arias, M.; Arratia, A. GeoSRS: A Hybrid Social Recommender System for Geolocated Data. Inf. Syst. 2016 , 57 , 111–128. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Tarus, J.K.; Niu, Z.; Yousif, A. A Hybrid Knowledge-Based Recommender System for e-Learning Based on Ontology and Sequential Pattern Mining. Future Gener. Comput. Syst. 2017 , 72 , 37–48. [ Google Scholar ] [ CrossRef ]
• Choi, S.-M.; Ko, S.-K.; Han, Y.-S. A Movie Recommendation Algorithm Based on Genre Correlations. Expert Syst. Appl. 2012 , 39 , 8079–8085. [ Google Scholar ] [ CrossRef ]
• Walek, B.; Fojtik, V. A Hybrid Recommender System for Recommending Relevant Movies Using an Expert System. Expert Syst. Appl. 2020 , 158 , 113452. [ Google Scholar ] [ CrossRef ]
• Pan, Y.; He, F.; Yu, H. Learning Social Representations with Deep Autoencoder for Recommender System. World Wide Web 2020 , 23 , 2259–2279. [ Google Scholar ] [ CrossRef ]
• Aivazoglou, M.; Roussos, A.O.; Margaris, D.; Vassilakis, C.; Ioannidis, S.; Polakis, J.; Spiliotopoulos, D. A Fine-Grained Social Network Recommender System. Soc. Netw. Anal. Min. 2020 , 10 , 8. [ Google Scholar ] [ CrossRef ]
• García-Sánchez, F.; Colomo-Palacios, R.; Valencia-García, R. A Social-Semantic Recommender System for Advertisements. Inf. Process. Manag. 2020 , 57 , 102153. [ Google Scholar ] [ CrossRef ]
• Bollen, D.; Knijnenburg, B.P.; Willemsen, M.C.; Graus, M. Understanding Choice Overload in Recommender Systems. In Proceedings of the Fourth ACM Conference on Recommender Systems, Barcelona, Spain, 26–30 September 2010; pp. 63–70. [ Google Scholar ]
• Im, I.; Hars, A. Does a One-Size Recommendation System Fit All? The Effectiveness of Collaborative Filtering Based Recommendation Systems across Different Domains and Search Modes. ACM Trans. Inf. Syst. 2007 , 26 , 4. [ Google Scholar ] [ CrossRef ]
• Wu, M.-L.; Chang, C.-H.; Liu, R.-Z. Integrating Content-Based Filtering with Collaborative Filtering Using Co-Clustering with Augmented Matrices. Expert Syst. Appl. 2014 , 41 , 2754–2761. [ Google Scholar ] [ CrossRef ]
• Van Meteren, R.; Van Someren, M. Using content-based filtering for recommendation. In Proceedings of the Machine Learning in the New Information Age MLnet/ECML2000 Workshop, Barcelona, Spain, 30 May 2000; pp. 47–56. [ Google Scholar ]
• Iyengar, S.S.; Lepper, M.R. When Choice Is Demotivating: Can One Desire Too Much of a Good Thing? J. Personal. Soc. Psychol. 2000 , 79 , 995–1006. [ Google Scholar ] [ CrossRef ]
• Loeb, S.; Terry, D. Information Filtering. Commun. ACM 1992 , 35 , 26–28. [ Google Scholar ] [ CrossRef ]
• Cantador, I.; Fernández, M.; Vallet, D.; Castells, P.; Picault, J.; Ribière, M. A Multi-Purpose Ontology-Based Approach for Personalised Content Filtering and Retrieval. In Advances in Semantic Media Adaptation and Personalization ; Wallace, M., Angelides, M.C., Mylonas, P., Eds.; Studies in Computational Intelligence; Springer: Berlin/Heidelberg, Germany, 2008; Volume 93, pp. 25–51. ISBN 978-3-540-76359-8. [ Google Scholar ]
• Salter, J.; Antonopoulos, N. CinemaScreen Recommender Agent: Combining Collaborative and Content-Based Filtering. IEEE Intell. Syst. 2006 , 21 , 35–41. [ Google Scholar ] [ CrossRef ]
• Iwahama, K.; Hijikata, Y.; Nishida, S. Content-Based Filtering System for Music Data. In Proceedings of the 2004 International Symposium on Applications and the Internet Workshops. 2004 Workshops, Tokyo, Japan, 26–30 January 2004; 2004; pp. 480–487. [ Google Scholar ]
• Weihong, H.; Yi, C. An E-Commerce Recommender System Based on Content-Based Filtering. Wuhan Univ. J. Nat. Sci. 2006 , 11 , 1091–1096. [ Google Scholar ] [ CrossRef ]
• Ghauth, K.I.; Abdullah, N.A. Learning Materials Recommendation Using Good Learners’ Ratings and Content-Based Filtering. Educ. Technol. Res. Dev. 2010 , 58 , 711–727. [ Google Scholar ] [ CrossRef ]
• Di Noia, T.; Mirizzi, R.; Ostuni, V.; Romito, D.; Zanker, M. Linked open data to support content-based recommender systems. In Proceedings of the 8th International Conference on Semantic Systems, Graz, Austria, 5–7 September 2012; pp. 1–8. [ Google Scholar ]
• Kompan, M.; Bieliková, M. Content-Based News Recommendation. In E-Commerce and Web Technologies ; Buccafurri, F., Semeraro, G., Eds.; Lecture Notes in Business Information Processing; Springer: Berlin/Heidelberg, Germany, 2010; Volume 61, pp. 61–72. ISBN 978-3-642-15207-8. [ Google Scholar ]
• Son, J.; Kim, S.B. Content-Based Filtering for Recommendation Systems Using Multiattribute Networks. Expert Syst. Appl. 2017 , 89 , 404–412. [ Google Scholar ] [ CrossRef ]
• Goldberg, D.; Nichols, D.; Oki, B.M.; Terry, D. Using Collaborative Filtering to Weave an Information Tapestry. Commun. ACM 1992 , 35 , 61–70. [ Google Scholar ] [ CrossRef ]
• Resnick, P.; Iacovou, N.; Suchak, M.; Bergstrom, P.; Riedl, J. GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, Chapel Hill, NC, USA, 22–26 October 1994; pp. 175–186. [ Google Scholar ]
• Park, S.-H.; Han, S.P. Empirical Analysis of the Impact of Product Diversity on Long-Term Performance of Recommender Systems. In Proceedings of the 14th Annual International Conference on Electronic Commerce—ICEC ’12, Singapore, 7–8 August 2012; pp. 280–281. [ Google Scholar ]
• Koren, Y.; Bell, R. Advances in collaborative filtering. In Recommender Systems Handbook ; Springer: Boston, MA, USA, 2015; pp. 77–118. [ Google Scholar ]
• Nilashi, M.; Ibrahim, O.; Bagherifard, K. A Recommender System Based on Collaborative Filtering Using Ontology and Dimensionality Reduction Techniques. Expert Syst. Appl. 2018 , 92 , 507–520. [ Google Scholar ] [ CrossRef ]
• Ansari, A.; Essegaier, S.; Kohli, R. Internet Recommendation Systems. J. Mark. Res. 2000 , 37 , 363–375. [ Google Scholar ] [ CrossRef ]
• Papagelis, M.; Plexousakis, D.; Kutsuras, T. Alleviating the Sparsity Problem of Collaborative Filtering Using Trust Inferences. In Trust Management ; Herrmann, P., Issarny, V., Shiu, S., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2005; Volume 3477, pp. 224–239. ISBN 978-3-540-26042-4. [ Google Scholar ]
• Wei, J.; He, J.; Chen, K.; Zhou, Y.; Tang, Z. Collaborative Filtering and Deep Learning Based Recommendation System for Cold Start Items. Expert Syst. Appl. 2017 , 69 , 29–39. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Gras, B.; Brun, A.; Boyer, A. Identifying Grey Sheep Users in Collaborative Filtering: A Distribution-Based Technique. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, Halifax, NS, Canada, 13–16 July 2016; pp. 17–26. [ Google Scholar ]
• Barragáns-Martínez, A.B.; Costa-Montenegro, E.; Burguillo, J.C.; Rey-López, M.; Mikic-Fonte, F.A.; Peleteiro, A. A Hybrid Content-Based and Item-Based Collaborative Filtering Approach to Recommend TV Programs Enhanced with Singular Value Decomposition. Inf. Sci. 2010 , 180 , 4290–4311. [ Google Scholar ] [ CrossRef ]
• Liu, H.; Hu, Z.; Mian, A.; Tian, H.; Zhu, X. A New User Similarity Model to Improve the Accuracy of Collaborative Filtering. Knowl.-Based Syst. 2014 , 56 , 156–166. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Shi, Y.; Larson, M.; Hanjalic, A. Collaborative Filtering beyond the User-Item Matrix: A Survey of the State of the Art and Future Challenges. ACM Comput. Surv. 2014 , 47 , 1–45. [ Google Scholar ] [ CrossRef ]
• Kim, H.-N.; Ji, A.-T.; Ha, I.; Jo, G.-S. Collaborative Filtering Based on Collaborative Tagging for Enhancing the Quality of Recommendation. Electron. Commer. Res. Appl. 2010 , 9 , 73–83. [ Google Scholar ] [ CrossRef ]
• Bobadilla, J.; Hernando, A.; Ortega, F.; Gutiérrez, A. Collaborative Filtering Based on Significances. Inf. Sci. 2012 , 185 , 1–17. [ Google Scholar ] [ CrossRef ]
• Koohi, H.; Kiani, K. User Based Collaborative Filtering Using Fuzzy C-Means. Measurement 2016 , 91 , 134–139. [ Google Scholar ] [ CrossRef ]
• Choi, K.; Suh, Y. A New Similarity Function for Selecting Neighbors for Each Target Item in Collaborative Filtering. Knowl.-Based Syst. 2013 , 37 , 146–153. [ Google Scholar ] [ CrossRef ]
• Basilico, J.; Hofmann, T. Unifying Collaborative and Content-Based Filtering. In Proceedings of the Twenty-First International Conference on Machine Learning—ICML ’04, Banff, AB, Canada, 4–8 July 2004; 2004; p. 9. [ Google Scholar ]
• Burke, R. Hybrid recommender systems: Survey and experiments. User Model. User-Adapt. Interact. 2002 , 12 , 331–370. [ Google Scholar ] [ CrossRef ]
• Kim, Y.M.; Choi, S. Scalable Variational Bayesian Matrix Factorization with Side Information. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Reykjavik, Iceland, 22–25 April 2014; pp. 493–502. [ Google Scholar ]
• Strub, F.; Gaudel, R.; Mary, J. Hybrid Recommender System Based on Autoencoders. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, Boston, MA, USA, 15 September 2016; pp. 11–16. [ Google Scholar ]
• Zhao, H.; Yao, Q.; Song, Y.; Kwok, J.T.; Lee, D.L. Side Information Fusion for Recommender Systems over Heterogeneous Information Network. ACM Trans. Knowl. Discov. Data 2021 , 15 , 1–32. [ Google Scholar ] [ CrossRef ]
• Fayyaz, Z.; Ebrahimian, M.; Nawara, D.; Ibrahim, A.; Kashef, R. Recommendation Systems: Algorithms, Challenges, Metrics, and Business Opportunities. Appl. Sci. 2020 , 10 , 7748. [ Google Scholar ] [ CrossRef ]
• Bag, S.; Kumar, S.; Awasthi, A.; Tiwari, M.K. A Noise Correction-Based Approach to Support a Recommender System in a Highly Sparse Rating Environment. Decis. Support Syst. 2019 , 118 , 46–57. [ Google Scholar ] [ CrossRef ]
• Khelloufi, A.; Ning, H.; Dhelim, S.; Qiu, T.; Ma, J.; Huang, R.; Atzori, L. A Social-Relationships-Based Service Recommendation System for SIoT Devices. IEEE Internet Things J. 2021 , 8 , 1859–1870. [ Google Scholar ] [ CrossRef ]
• Liu, Y.; Cheng, J.; Yan, C.; Wu, X.; Chen, F. Research on the Matthews Correlation Coefficients Metrics of Personalized Recommendation Algorithm Evaluation. Int. J. Hybrid Inf. Technol. 2015 , 8 , 163–172. [ Google Scholar ] [ CrossRef ]
• Krichene, W.; Rendle, S. On Sampled Metrics for Item Recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual Event, CA, USA, 23 August 2020; pp. 1748–1757. [ Google Scholar ]
• Hsu, F.-M.; Lin, Y.-T.; Ho, T.-K. Design and Implementation of an Intelligent Recommendation System for Tourist Attractions: The Integration of EBM Model, Bayesian Network and Google Maps. Expert Syst. Appl. 2012 , 39 , 3257–3264. [ Google Scholar ] [ CrossRef ]
• Parra, D.; Sahebi, S. Recommender Systems: Sources of Knowledge and Evaluation Metrics. In Advanced Techniques in Web Intelligence-2 ; Velásquez, J.D., Palade, V., Jain, L.C., Eds.; Studies in Computational Intelligence; Springer: Berlin/Heidelberg, Germany, 2013; Volume 452, pp. 149–175. ISBN 978-3-642-33325-5. [ Google Scholar ]
• Kawasaki, M.; Hasuike, T. A Recommendation System by Collaborative Filtering Including Information and Characteristics on Users and Items. In Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence, Honolulu, HI, USA, 27 November–1 December 2017; pp. 1–8. [ Google Scholar ]
• Hendler, J. Web 3.0 emerging. Computer 2009 , 42 , 111–113. [ Google Scholar ] [ CrossRef ]
• Di Noia, T.; Mirizzi, R.; Ostuni, V.C.; Romito, D. Exploiting the Web of Data in Model-Based Recommender Systems. In Proceedings of the Sixth ACM Conference on Recommender Systems—RecSys ’12, New York, NY, USA, 9–13 September 2012; p. 253. [ Google Scholar ]
• Lika, B.; Kolomvatsos, K.; Hadjiefthymiades, S. Facing the Cold Start Problem in Recommender Systems. Expert Syst. Appl. 2014 , 41 , 2065–2073. [ Google Scholar ] [ CrossRef ]
• Gündüz, S.; Özsu, M.T. A web page prediction model based on click-stream tree representation of user behavior. In Proceedings of the 9th ACM International Conference on Knowledge Discovery and Data Mining (KDD), Washington, DC, USA, 24–27 August 2003; pp. 535–540. [ Google Scholar ]
• Gómez-Pérez, A.; Corcho, O. Ontology languages for the Semantic Web. IEEE Intell. Syst. 2002 , 17 , 54–60. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Yu, Eunji; Kim, Jung-Cheol; Lee, Choon Yeul; Kim, Namgyu Using Ontologies for Semantic Text Mining. J. Inf. Syst. 2012 , 21 , 137–161. [ CrossRef ] [ Green Version ]
• Noy, N.F.; McGuinness, D.L. Ontology Development 101: A Guide to Creating Your First Ontology ; Stanford University: Stanford, CA, USA, 2001. [ Google Scholar ]
• Abowd, G.D.; Dey, A.K.; Brown, P.J.; Davies, N.; Smith, M.; Steggles, P. Towards a Better Understanding of Context and Context-Awareness. In Handheld and Ubiquitous Computing ; Gellersen, H.-W., Ed.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 1999; Volume 1707, pp. 304–307. ISBN 978-3-540-66550-2. [ Google Scholar ]
• Zadeh, L.A. The Concept of a Linguistic Variable and Its Application to Approximate Reasoning—I. Inf. Sci. 1975 , 8 , 199–249. [ Google Scholar ] [ CrossRef ]
• Porcel, C.; Herrera-Viedma, E. Dealing with Incomplete Information in a Fuzzy Linguistic Recommender System to Disseminate Information in University Digital Libraries. Knowl.-Based Syst. 2010 , 23 , 32–39. [ Google Scholar ] [ CrossRef ]
• Trstenjak, B.; Mikac, S.; Donko, D. KNN with TF-IDF Based Framework for Text Categorization. Procedia Eng. 2014 , 69 , 1356–1364. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Guo, G.; Wang, H.; Bell, D.; Bi, Y.; Greer, K. KNN Model-Based Approach in Classification. In On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE. ; Meersman, R., Tari, Z., Schmidt, D.C., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2003; Volume 2888, pp. 986–996. ISBN 978-3-540-20498-5. [ Google Scholar ]
• Jannach, D.; Lerche, L.; Kamehkhosh, I.; Jugovac, M. What Recommenders Recommend: An Analysis of Recommendation Biases and Possible Countermeasures. User Model User-Adap. Inter. 2015 , 25 , 427–491. [ Google Scholar ] [ CrossRef ]
• Shepitsen, A.; Gemmell, J.; Mobasher, B.; Burke, R. Personalized Recommendation in Social Tagging Systems Using Hierarchical Clustering. In Proceedings of the 2008 ACM Conference on Recommender Systems—RecSys ’08, Lausanne, Switzerland, 23–25 October 2008; p. 259. [ Google Scholar ]
• Bilge, A.; Polat, H. A Comparison of Clustering-Based Privacy-Preserving Collaborative Filtering Schemes. Appl. Soft Comput. 2013 , 13 , 2478–2489. [ Google Scholar ] [ CrossRef ]
• Gan, G.; Ma, C.; Wu, J. Data Clustering: Theory, Algorithms, and Applications ; Society for Industrial and Applied Mathematics: Philadelphia, VA, USA, 2007. [ Google Scholar ]
• Gong, S. A Collaborative Filtering Recommendation Algorithm Based on User Clustering and Item Clustering. J. Softw. 2010 , 5 , 745–752. [ Google Scholar ] [ CrossRef ]
• Mnih, A.; Salakhutdinov, R.R. Probabilistic matrix factorization. In Advances in Neural Information Processing Systems ; IEEE: Piscatawy, NJ, USA, 2008; pp. 1257–1264. [ Google Scholar ]
• Koren, Y.; Bell, R.; Volinsky, C. Matrix Factorization Techniques for Recommender Systems. Computer 2009 , 42 , 30–37. [ Google Scholar ] [ CrossRef ]
• Kim, D.; Park, C.; Oh, J.; Lee, S.; Yu, H. Convolutional Matrix Factorization for Document Context-Aware Recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, 15–19 September 2016; pp. 233–240. [ Google Scholar ]
• Koren, Y. Factorization meets the neighborhood: A multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 24–27 August 2008; pp. 426–434. [ Google Scholar ]
• Ma, H.; Yang, H.; Lyu, M.R.; King, I. SoRec: Social Recommendation Using Probabilistic Matrix Factorization. In Proceedings of the 17th ACM Conference on Information and Knowledge Mining—CIKM ’08, Napa Valley, CA, USA, 26–30 October 2008; p. 931. [ Google Scholar ]
• Kim, Y.; Shim, K. TWILITE: A Recommendation System for Twitter Using a Probabilistic Model Based on Latent Dirichlet Allocation. Inf. Syst. 2014 , 42 , 59–77. [ Google Scholar ] [ CrossRef ]
• Zhang, Y. GroRec: A Group-Centric Intelligent Recommender System Integrating Social, Mobile and Big Data Technologies. IEEE Trans. Serv. Comput. 2016 , 9 , 786–795. [ Google Scholar ] [ CrossRef ]
• Seo, Y.-S.; Huh, J.-H. GUI-Based Software Modularization through Module Clustering in Edge Computing Based IoT Environments. J. Ambient Intell. Human. Comput. 2019, pp. 1–15. Available online: https://link.springer.com/article/10.1007%2Fs12652-019-01455-3 (accessed on 6 September 2019).
• Van den Oord, A.; Dieleman, S.; Schrauwen, B. Deep Content-Based Music Recommendation. In Proceedings of the 26th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 15–19 December 2013; pp. 2643–2651. [ Google Scholar ]
• Wang, H.; Wang, N.; Yeung, D.-Y. Collaborative Deep Learning for Recommender Systems. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, Australia, 10–13 August 2015; pp. 1235–1244. [ Google Scholar ]
• He, X.; Liao, L.; Zhang, H.; Nie, L.; Hu, X.; Chua, T.-S. Neural Collaborative Filtering. In Proceedings of the 26th International Conference on World Wide Web, Perth, Australia, 3–7 April 2017; International World Wide Web Conferences Steering Committee; pp. 173–182. [ Google Scholar ]
• Gomez-Uribe, C.A.; Hunt, N. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Trans. Manage. Inf. Syst. 2016 , 6 , 1–19. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Koren, Y. Factor in the Neighbors: Scalable and Accurate Collaborative Filtering. ACM Trans. Knowl. Discov. Data 2010 , 4 , 1–24. [ Google Scholar ] [ CrossRef ]
• Bell, R.M.; Koren, Y. Lessons from the Netflix Prize Challenge. SIGKDD Explor. Newsl. 2007 , 9 , 75–79. [ Google Scholar ] [ CrossRef ]
• McFee, B.; Barrington, L.; Lanckriet, G. Learning Content Similarity for Music Recommendation. IEEE Trans. Audio Speech Lang. Process. 2012 , 20 , 2207–2218. [ Google Scholar ] [ CrossRef ]
• Odić, A.; Tkalčič, M.; Tasič, J.F.; Košir, A. Predicting and Detecting the Relevant Contextual Information in a Movie-Recommender System. Interact. Comput. 2013 , 25 , 74–90. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Wang, X.; Wang, Y. Improving Content-Based and Hybrid Music Recommendation Using Deep Learning. In Proceedings of the 22nd ACM International Conference on Multimedia, rlando, FL, USA, 3–7 November 2014; pp. 627–636. [ Google Scholar ]
• Bogdanov, D.; Haro, M.; Fuhrmann, F.; Xambó, A.; Gómez, E.; Herrera, P. Semantic Audio Content-Based Music Recommendation and Visualization Based on User Preference Examples. Inf. Process. Manag. 2013 , 49 , 13–33. [ Google Scholar ] [ CrossRef ]
• Colombo-Mendoza, L.O.; Valencia-García, R.; Rodríguez-González, A.; Alor-Hernández, G.; Samper-Zapater, J.J. RecomMetz: A Context-Aware Knowledge-Based Mobile Recommender System for Movie Showtimes. Expert Syst. Appl. 2015 , 42 , 1202–1222. [ Google Scholar ] [ CrossRef ]
• Vall, A.; Dorfer, M.; Eghbal-zadeh, H.; Schedl, M.; Burjorjee, K.; Widmer, G. Feature-Combination Hybrid Recommender Systems for Automated Music Playlist Continuation. User Model User-Adap. Inter. 2019 , 29 , 527–572. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Huh, J.-H.; Otgonchimeg, S.; Seo, K. Advanced Metering Infrastructure Design and Test Bed Experiment Using Intelligent Agents: Focusing on the PLC Network Base Technology for Smart Grid System. J. Supercomput. 2016 , 72 , 1862–1877. [ Google Scholar ] [ CrossRef ]
• Lee, S.; Jeong, H.; Ko, H. Classical Music Specific Mood Automatic Recognition Model Proposal. Electronics 2021 , 10 , 2489. [ Google Scholar ] [ CrossRef ]
• Wang, Z.; Yu, X.; Feng, N.; Wang, Z. An Improved Collaborative Movie Recommendation System Using Computational Intelligence. J. Vis. Lang. Comput. 2014 , 25 , 667–675. [ Google Scholar ] [ CrossRef ]
• Sánchez-Moreno, D.; Gil González, A.B.; Muñoz Vicente, M.D.; López Batista, V.F.; Moreno García, M.N. A Collaborative Filtering Method for Music Recommendation Using Playing Coefficients for Artists and Users. Expert Syst. Appl. 2016 , 66 , 234–244. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• He, J.; Chu, W.W. A Social Network-Based Recommender System (SNRS). In Data Mining for Social Network Data ; Memon, N., Xu, J.J., Hicks, D.L., Chen, H., Eds.; Annals of Information Systems; Springer: Boston, MA, USA, 2010; Volume 12, pp. 47–74. ISBN 978-1-4419-6286-7. [ Google Scholar ]
• Tsur, O.; Rappoport, A. What’s in a Hashtag? Content Based Prediction of the Spread of Ideas in Microblogging Communities. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining—WSDM ’12, New York, NY, USA, 8–12 February 2012; p. 643. [ Google Scholar ]
• Wang, Z.; Sun, L.; Zhu, W.; Yang, S.; Li, H.; Wu, D. Joint Social and Content Recommendation for User-Generated Videos in Online Social Network. IEEE Trans. Multimed. 2013 , 15 , 698–709. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Kazienko, P.; Musial, K.; Kajdanowicz, T. Multidimensional Social Network in the Social Recommender System. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 2011 , 41 , 746–759. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Sedhain, S.; Menon, A.K.; Sanner, S.; Xie, L.; Braziunas, D. Low-rank linear cold-start recommendation from social data. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, Menlo Park, CA, USA, 4–9 February 2017; pp. 1502–1508. [ Google Scholar ]
• Davoodi, E.; Kianmehr, K.; Afsharchi, M. A Semantic Social Network-Based Expert Recommender System. Appl. Intell. 2013 , 39 , 1–13. [ Google Scholar ] [ CrossRef ]
• Kesorn, K.; Juraphanthong, W.; Salaiwarakul, A. Personalized Attraction Recommendation System for Tourists Through Check-In Data. IEEE Access 2017 , 5 , 26703–26721. [ Google Scholar ] [ CrossRef ]
• Sun, Y.; Fan, H.; Bakillah, M.; Zipf, A. Road-Based Travel Recommendation Using Geo-Tagged Images. Comput. Environ. Urban Syst. 2015 , 53 , 110–122. [ Google Scholar ] [ CrossRef ]
• Smirnov, A.V.; Kashevnik, A.M.; Ponomarev, A. Context-Based Infomobility System for Cultural Heritage Recommendation: Tourist Assistant—TAIS. Pers. Ubiquit. Comput. 2017 , 21 , 297–311. [ Google Scholar ] [ CrossRef ]
• Tsai, C.-Y.; Chung, S.-H. A Personalized Route Recommendation Service for Theme Parks Using RFID Information and Tourist Behavior. Decis. Support Syst. 2012 , 52 , 514–527. [ Google Scholar ] [ CrossRef ]
• Ruotsalo, T.; Haav, K.; Stoyanov, A.; Roche, S.; Fani, E.; Deliai, R.; Mäkelä, E.; Kauppinen, T.; Hyvönen, E. SMARTMUSEUM: A Mobile Recommender System for the Web of Data. J. Web Semant. 2013 , 20 , 50–67. [ Google Scholar ] [ CrossRef ]
• Al-Hassan, M.; Lu, H.; Lu, J. A Semantic Enhanced Hybrid Recommendation Approach: A Case Study of e-Government Tourism Service Recommendation System. Decis. Support Syst. 2015 , 72 , 97–109. [ Google Scholar ] [ CrossRef ]
• Nilashi, M.; bin Ibrahim, O.; Ithnin, N.; Sarmin, N.H. A Multi-Criteria Collaborative Filtering Recommender System for the Tourism Domain Using Expectation Maximization (EM) and PCA–ANFIS. Electron. Commer. Res. Appl. 2015 , 14 , 542–562. [ Google Scholar ] [ CrossRef ]
• Logesh, R.; Subramaniyaswamy, V. Exploring Hybrid Recommender Systems for Personalized Travel Applications. In Cognitive Informatics and Soft Computing ; Mallick, P.K., Balas, V.E., Bhoi, A.K., Zobaa, A.F., Eds.; Advances in Intelligent Systems and Computing; Springer: Singapore, 2019; Volume 768, pp. 535–544. ISBN 9789811306167. [ Google Scholar ]
• Colomo-Palacios, R.; García-Peñalvo, F.J.; Stantchev, V.; Misra, S. Towards a Social and Context-Aware Mobile Recommendation System for Tourism. Pervasive Mob. Comput. 2017 , 38 , 505–515. [ Google Scholar ] [ CrossRef ]
• Galhotra, B.; Dewan, A. Impact of COVID-19 on Digital Platforms and Change in E-Commerce Shopping Trends. In Proceedings of the 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Coimbatore, India, 7–9 October 2020; pp. 861–866. [ Google Scholar ]
• Zhang, X.; Liu, H.; Chen, X.; Zhong, J.; Wang, D. A Novel Hybrid Deep Recommendation System to Differentiate User’s Preference and Item’s Attractiveness. Inf. Sci. 2020 , 519 , 306–316. [ Google Scholar ] [ CrossRef ]
• Lu, Y.; Zhao, L.; Wang, B. From Virtual Community Members to C2C E-Commerce Buyers: Trust in Virtual Communities and Its Effect on Consumers’ Purchase Intention. Electron. Commer. Res. Appl. 2010 , 9 , 346–360. [ Google Scholar ] [ CrossRef ]
• Sulikowski, P.; Zdziebko, T.; Hussain, O.; Wilbik, A. Fuzzy Approach to Purchase Intent Modeling Based on User Tracking For E-Commerce Recommenders. In Proceedings of the 2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Luxembourg, 11–14 July 2021; pp. 1–8. [ Google Scholar ]
• Jiang, L.; Cheng, Y.; Yang, L.; Li, J.; Yan, H.; Wang, X. A trust-based collaborative filtering algorithm for E-commerce recommendation system. J. Ambient. Intell. Humaniz. Comput. 2019 , 10 , 3023–3034. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Sarwar, B.; Karypis, G.; Konstan, J.; Reidl, J. Item-Based Collaborative Filtering Recommendation Algorithms. In Proceedings of the Tenth International Conference on World Wide Web—WWW ’01, Hong Kong, China, 1–5 May 2001; pp. 285–295. [ Google Scholar ]
• Hwangbo, H.; Kim, Y.S.; Cha, K.J. Recommendation System Development for Fashion Retail E-Commerce. Electron. Commer. Res. Appl. 2018 , 28 , 94–101. [ Google Scholar ] [ CrossRef ]
• Ho, S.Y.; Bodoff, D.; Tam, K.Y. Timing of Adaptive Web Personalization and Its Effects on Online Consumer Behavior. Inf. Syst. Res. 2011 , 22 , 660–679. [ Google Scholar ] [ CrossRef ]
• Guo, Y.; Yin, C.; Li, M.; Ren, X.; Liu, P. Mobile E-Commerce Recommendation System Based on Multi-Source Information Fusion for Sustainable e-Business. Sustainability 2018 , 10 , 147. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Sulikowski, P.; Zdziebko, T. Deep Learning-Enhanced Framework for Performance Evaluation of a Recommending Interface with Varied Recommendation Position and Intensity Based on Eye-Tracking Equipment Data Processing. Electronics 2020 , 9 , 266. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Castro-Schez, J.J.; Miguel, R.; Vallejo, D.; López-López, L.M. A Highly Adaptive Recommender System Based on Fuzzy Logic for B2C E-Commerce Portals. Expert Syst. Appl. 2011 , 38 , 2441–2454. [ Google Scholar ] [ CrossRef ]
• Tröster, G. The Agenda of Wearable Healthcare. Yearb. Med. Inf. 2005 , 14 , 125–138. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Apple. Use Emergency SOS on Your Apple Watch . 2021. Available online: https://support.apple.com/en-us/HT206983 (accessed on 27 April 2021).
• Mishra, T.; Wang, M.; Metwally, A.A.; Bogu, G.K.; Brooks, A.W.; Bahmani, A.; Alavi, A.; Celli, A.; Higgs, E.; Dagan-Rosenfeld, O.; et al. Pre-Symptomatic Detection of COVID-19 from Smartwatch Data. Nat. Biomed. Eng. 2020 , 4 , 1208–1220. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zheng, J.W.; Zhang, Z.B.; Wu, T.H.; Zhang, Y. A Wearable Mobihealth Care System Supporting Real-Time Diagnosis and Alarm. Med. Bio. Eng. Comput. 2007 , 45 , 877–885. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Raghupathi, W.; Raghupathi, V. Big Data Analytics in Healthcare: Promise and Potential. Health Inf. Sci. Syst. 2014 , 2 , 3. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Rehman, A.; Naz, S.; Razzak, I. Leveraging Big Data Analytics in Healthcare Enhancement: Trends, Challenges and Opportunities. Multimed. Syst. 2021 , 1 , 1–33. [ Google Scholar ] [ CrossRef ]
• Duan, L.; Street, W.N.; Xu, E. Healthcare Information Systems: Data Mining Methods in the Creation of a Clinical Recommender System. Enterp. Inf. Syst. 2011 , 5 , 169–181. [ Google Scholar ] [ CrossRef ]
• Chen, J.; Li, K.; Rong, H.; Bilal, K.; Yang, N.; Li, K. A Disease Diagnosis and Treatment Recommendation System Based on Big Data Mining and Cloud Computing. Inf. Sci. 2018 , 435 , 124–149. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Thong, N.T.; Son, L.H. HIFCF: An Effective Hybrid Model between Picture Fuzzy Clustering and Intuitionistic Fuzzy Recommender Systems for Medical Diagnosis. Expert Syst. Appl. 2015 , 42 , 3682–3701. [ Google Scholar ] [ CrossRef ]
• Ko, H.; Huh, J.-H. Electronic Solutions for Artificial Intelligence Healthcare. Electronics 2021 , 10 , 2421. [ Google Scholar ] [ CrossRef ]
• Iwendi, C.; Khan, S.; Anajemba, J.H.; Bashir, A.K.; Noor, F. Realizing an Efficient IoMT-Assisted Patient Diet Recommendation System Through Machine Learning Model. IEEE Access 2020 , 8 , 28462–28474. [ Google Scholar ] [ CrossRef ]
• Yang, L.; Hsieh, C.-K.; Yang, H.; Pollak, J.P.; Dell, N.; Belongie, S.; Cole, C.; Estrin, D. Yum-Me: A Personalized Nutrient-Based Meal Recommender System. ACM Trans. Inf. Syst. 2017 , 36 , 1–31. [ Google Scholar ] [ CrossRef ]
• Sezgin, E.; Ozkan, S. A systematic literature review on Health Recommender Systems. In Proceedings of the 2013 E-Health and Bioengineering Conference (EHB), Iasi, Romania, 21–23 November 2013. [ Google Scholar ] [ CrossRef ]
• Sanchez Bocanegra, C.L.; Sevillano Ramos, J.L.; Rizo, C.; Civit, A.; Fernandez-Luque, L. HealthRecSys: A Semantic Content-Based Recommender System to Complement Health Videos. BMC Med. Inf. Decis. Mak. 2017 , 17 , 63. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Wiesner, M.; Pfeifer, D. Health Recommender Systems: Concepts, Requirements, Technical Basics and Challenges. IJERPH 2014 , 11 , 2580–2607. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ Green Version ]
• Abbas, K.; Afaq, M.; Ahmed Khan, T.; Song, W.-C. A Blockchain and Machine Learning-Based Drug Supply Chain Management and Recommendation System for Smart Pharmaceutical Industry. Electronics 2020 , 9 , 852. [ Google Scholar ] [ CrossRef ]
• Chen, R.-C.; Huang, Y.-H.; Bau, C.-T.; Chen, S.-M. A Recommendation System Based on Domain Ontology and SWRL for Anti-Diabetic Drugs Selection. Expert Syst. Appl. 2012 , 39 , 3995–4006. [ Google Scholar ] [ CrossRef ]
• Tikhomirov, V.; Dneprovskaya, N.; Yankovskaya, E. Three Dimensions of Smart Education. In Smart Education and Smart e-Learning ; Uskov, V.L., Howlett, R.J., Jain, L.C., Eds.; Smart Innovation, Systems and Technologies; Springer International Publishing: Cham, Switzerland, 2015; Volume 41, pp. 47–56. ISBN 978-3-319-19874-3. [ Google Scholar ]
• Lin, J.; Pu, H.; Li, Y.; Lian, J. Intelligent Recommendation System for Course Selection in Smart Education. Procedia Comput. Sci. 2018 , 129 , 449–453. [ Google Scholar ] [ CrossRef ]
• Zhu, Z.-T.; Yu, M.-H.; Riezebos, P. A Research Framework of Smart Education. Smart Learn. Environ. 2016 , 3 , 4. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Wan, S.; Niu, Z. An E-Learning Recommendation Approach Based on the Self-Organization of Learning Resource. Knowl.-Based Syst. 2018 , 160 , 71–87. [ Google Scholar ] [ CrossRef ]
• Shu, J.; Shen, X.; Liu, H.; Yi, B.; Zhang, Z. A Content-Based Recommendation Algorithm for Learning Resources. Multimed. Syst. 2018 , 24 , 163–173. [ Google Scholar ] [ CrossRef ]
• Chen, Y.; Li, X.; Liu, J.; Ying, Z. Recommendation System for Adaptive Learning. Appl. Psychol. Meas. 2018 , 42 , 24–41. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Klašnja-Milićević, A.; Vesin, B.; Ivanović, M.; Budimac, Z. E-Learning Personalization Based on Hybrid Recommendation Strategy and Learning Style Identification. Comput. Educ. 2011 , 56 , 885–899. [ Google Scholar ] [ CrossRef ]
• Dwivedi, P.; Bharadwaj, K.K. E-Learning Recommender System for a Group of Learners Based on the Unified Learner Profile Approach. Expert Syst. 2015 , 32 , 264–276. [ Google Scholar ] [ CrossRef ]
• Wu, D.; Lu, J.; Zhang, G. A Fuzzy Tree Matching-Based Personalized E-Learning Recommender System. IEEE Trans. Fuzzy Syst. 2015 , 23 , 2412–2426. [ Google Scholar ] [ CrossRef ]
• Obeid, C.; Lahoud, I.; El Khoury, H.; Champin, P.-A. Ontology-Based Recommender System in Higher Education. In Proceedings of the Companion Proceedings of The Web Conference 2018, Lyon, France, 23–27 April 2018; pp. 1031–1034. [ Google Scholar ]
• Gulzar, Z.; Leema, A.A.; Deepak, G. PCRS: Personalized Course Recommender System Based on Hybrid Approach. Procedia Comput. Sci. 2018 , 125 , 518–524. [ Google Scholar ] [ CrossRef ]
• Esteban, A.; Zafra, A.; Romero, C. Helping University Students to Choose Elective Courses by Using a Hybrid Multi-Criteria Recommendation System with Genetic Optimization. Knowl.-Based Syst. 2020 , 194 , 105385. [ Google Scholar ] [ CrossRef ]
• Serrano-Guerrero, J.; Herrera-Viedma, E.; Olivas, J.A.; Cerezo, A.; Romero, F.P. A Google Wave-Based Fuzzy Recommender System to Disseminate Information in University Digital Libraries 2.0. Inf. Sci. 2011 , 181 , 1503–1516. [ Google Scholar ] [ CrossRef ]
• Tejeda-Lorente, Á.; Porcel, C.; Peis, E.; Sanz, R.; Herrera-Viedma, E. A Quality Based Recommender System to Disseminate Information in a University Digital Library. Inf. Sci. 2014 , 261 , 52–69. [ Google Scholar ] [ CrossRef ]
• He, Q.; Pei, J.; Kifer, D.; Mitra, P.; Giles, L. Context-aware citation recommendation. In Proceedings of the 19th International Conference on World Wide Web—WWW ’10, Raleigh, NC, USA, 26–30 April 2010. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Wang, D.; Liang, Y.; Xu, D.; Feng, X.; Guan, R. A Content-Based Recommender System for Computer Science Publications. Knowl.-Based Syst. 2018 , 157 , 1–9. [ Google Scholar ] [ CrossRef ]
• Park, N.; Roman, R.; Lee, S.; Chung, J.E. User Acceptance of a Digital Library System in Developing Countries: An Application of the Technology Acceptance Model. Int. J. Inf. Manag. 2009 , 29 , 196–209. [ Google Scholar ] [ CrossRef ]
• Jeong, H. An Investigation of User Perceptions and Behavioral Intentions towards the E-Library. Libr. Collect. Acquis. Tech. Serv. 2011 , 35 , 45–60. [ Google Scholar ] [ CrossRef ]
• Achakulvisut, T.; Acuna, D.E.; Ruangrong, T.; Kording, K. Science Concierge: A Fast Content-Based Recommendation System for Scientific Publications. PLoS ONE 2016 , 11 , e0158423. [ Google Scholar ] [ CrossRef ]
• Färber, M.; Jatowt, A. Citation Recommendation: Approaches and Datasets. Int. J. Digit. Libr. 2020 , 21 , 375–405. [ Google Scholar ] [ CrossRef ]
• Brian, D. Netflix Subscriber and Growth Statistics: How Many People Watch Netflix in 2021? Available online: https://backlinko.com/netflix-users (accessed on 25 February 2021).
• Statista. Disney+’s Number of Subscribers Worldwide from 1st Quarter 2020 to 2nd Quarter 2021 . Available online: https://www.statista.com/statistics/1095372/disney-plus-number-of-subscribers-us/ (accessed on 19 August 2021).
• BBC News. Netflix Launches UK Film and TV Streaming Service . Available online: https://web.archive.org/web/20120109175608/http://www.bbc.co.uk/news/technology-16467432 (accessed on 9 January 2012).
• PR Newswire. Netflix Launches in Sweden, Denmark, Norway and Finland . Available online: https://web.archive.org/web/20141127211314/http://www.prnewswire.com/news-releases/netflix-launches-in-sweden-denmark-norway-and-finland-174749581.html (accessed on 27 November 2014).
• Variety. Netflix’s ‘House of Cards’ Falls Prey to Piracy . Available online: https://variety.com/2013/digital/news/pirates-swarm-netflixs-house-of-cards-1200333171/ (accessed on 4 April 2013).
• Gupta, G.; Singharia, K. Consumption of OTT Media Streaming in COVID-19 Lockdown: Insights from PLS Analysis. Vision 2021 , 25 , 36–46. [ Google Scholar ] [ CrossRef ]
• Ryan; Meg; Anousha. Shaken Studios. Empty Theaters. What Hollywood Lost during the Pandemic . Available online: https://www.latimes.com/entertainment-arts/business/story/2020-12-09/everything-hollywood-lost-during-the-pandemic (accessed on 9 December 2020).
• Insider Intelligence. Top OTT VOD Streaming Services in 2021 by Viewer Count and Growth . Available online: https://www.insiderintelligence.com/insights/ott-vod-video-streaming-services/ (accessed on 21 July 2021).
• Statista. Facebook Keeps on Growing . Available online: https://www.statista.com/chart/10047/facebooks-monthly-active-users/ (accessed on 4 February 2021).
• Prabhakaran. 50+ Instagram Stats Everyone Must Know in 2020 . Available online: https://www.connectivasystems.com/instagram-stats-2020/ (accessed on 1 April 2020).
• Statista. Number of Monthly Active Twitter Users Worldwide from 1st Quarter 2010 to 1st Quarter 2019 . Available online: https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/ (accessed on 31 December 2021).
• Hughes, D.J.; Rowe, M.; Batey, M.; Lee, A. A Tale of Two Sites: Twitter vs. Facebook and the Personality Predictors of Social Media Usage. Comput. Hum. Behav. 2012 , 28 , 561–569. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Highfield, T.; Leaver, T. A Methodology for Mapping Instagram Hashtags. First Monday 2014 , 20 , 1–11. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Sarah, F. The Inside Story of How Facebook Acquired Instagram . Available online: https://onezero.medium.com/the-inside-story-of-how-facebook-acquired-instagram-318f244f1283 (accessed on 5 August 2020).
• Dave, S. Chart of the Day: Instagram Is Now Bigger Than Twitter . Available online: https://www.businessinsider.com.au/chart-of-the-day-instagram-is-now-bigger-than-twitter-2014-12 (accessed on 11 December 2014).
• Li, S.; Takahashi, S.; Yamada, K.; Takagi, M.; Sasaki, J. Analysis of SNS Photo Data Taken by Foreign Tourists to Japan and a Proposed Adaptive Tourism Recommendation System. In Proceedings of the 2017 International Conference on Progress in Informatics and Computing (PIC), Nanjing, China, 15–17 December 2017; pp. 323–327. [ Google Scholar ]
• Chao, L.; Jian, Y.; Xiang, L.; Hui, C.J. A Social Network System Oriented Hybrid Recommendation Model. In Proceedings of the 2012 2nd International Conference on Computer Science and Network Technology, Changchun, China, 29–31 December 2012; pp. 901–906. [ Google Scholar ]
• Jeong, O.-R. SNS-Based Recommendation Mechanisms for Social Media. Multimed Tools Appl 2015 , 74 , 2433–2447. [ Google Scholar ] [ CrossRef ]
• Statista. New Year’s Peak Illustrates Airbnb’s Growing Stature . Available online: https://lb-aps-frontend.statista.com/chart/20386/guests-staying-at-airbnb-appartments-on-new-years-eve/ (accessed on 2 January 2020).
• Siegler, M.G. Airbnb Cozies Up to Facebook to Help You Feel More at Home When Away from Home . Available online: https://techcrunch.com/2011/05/09/airbnb-social-connections/ (accessed on 10 May 2011).
• Kontogianni, A.; Alepis, E. Smart Tourism: State of the Art and Literature Review for the Last Six Years. Array 2020 , 6 , 100020. [ Google Scholar ] [ CrossRef ]
• Balasaraswathi, M.; Srinivasan, K.; Udayakumar, L.; Sivasakthiselvan, S.; Sumithra, M.G. Big Data Analytic of Contexts and Cascading Tourism for Smart City. Mater. Today Proc. 2020 , S2214785320377129. [ Google Scholar ] [ CrossRef ]
• Yoo, C.W.; Goo, J.; Huang, C.D.; Nam, K.; Woo, M. Improving Travel Decision Support Satisfaction with Smart Tourism Technologies: A Framework of Tourist Elaboration Likelihood and Self-Efficacy. Technol. Forecast. Soc. Chang. 2017 , 123 , 330–341. [ Google Scholar ] [ CrossRef ]
• Tribe, J.; Mkono, M. Not Such Smart Tourism? The Concept of e-Lienation. Ann. Tour. Res. 2017 , 66 , 105–115. [ Google Scholar ] [ CrossRef ]
• Kim, I.-S.; Jeong, C.-S.; Jung, T.-W.; Kang, J.-K.; Jung, K.-D. AR Tourism Recommendation System Based on Character-Based Tourism Preference Using Big Data. Int. J. Internet Broadcasting Commun. 2021 , 13 , 61–68. [ Google Scholar ] [ CrossRef ]
• Statista. Net Revenue of Amazon from 1st Quarter 2007 to 1st Quarter 2021 . Available online: https://www.statista.com/statistics/273963/quarterly-revenue-of-amazoncom/ (accessed on 19 July 2021).
• Statista. Annual Revenue of Alibaba Group from Financial Year 2011 to 2021 . Available online: https://www.statista.com/statistics/225614/net-revenue-of-alibaba/ (accessed on 15 September 2021).
• Linden, G.; Smith, B.; York, J. Amazon.Com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Comput. 2003 , 7 , 76–80. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Smith, B.; Linden, G. Two Decades of Recommender Systems at Amazon.Com. IEEE Internet Comput. 2017 , 21 , 12–18. [ Google Scholar ] [ CrossRef ]
• Wang, J.; Huang, P.; Zhao, H.; Zhang, Z.; Zhao, B.; Lee, D.L. Billion-Scale Commodity Embedding for E-Commerce Recommendation in Alibaba. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK, 19–23 August 2018; pp. 839–848. [ Google Scholar ]
• Chen, Q.; Zhao, H.; Li, W.; Huang, P.; Ou, W. Behavior Sequence Transformer for E-Commerce Recommendation in Alibaba. In Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data, Anchorage, Alaska, 5 August 2019; pp. 1–4. [ Google Scholar ]
• Statista. Apple Watch Installed Base Worldwide from 2015 to 2020 . Available online: https://www.statista.com/statistics/1221051/apple-watch-users-worldwide/ (accessed on 17 March 2021).
• Statista. Fitbit’s Community Grows Amid Stalling Device Sales . Available online: https://www.statista.com/chart/13083/worldwide-fitbit-shipments/ (accessed on 4 August 2020).
• Dino, G. Huffpost. Fitbit Recalls Fitbit Force after Complaints of Severe Rashes . Available online: https://www.huffpost.com/entry/fitbit-force-recall_n_4832771 (accessed on 21 February 2014).
• Mallory, H. Apple Study Finds Watch Can Detect More Types of Irregular Heartbeats . Available online: https://www.mobihealthnews.com/news/apple-study-finds-watch-can-detect-more-types-irregular-heartbeats (accessed on 29 September 2021).
• Statista. Global Apple Watch shipments from 2017 to 2019. Available online: https://www.statista.com/statistics/526005/global-apple-watch-shipments-forecast/ (accessed on 31 December 2021).
• Zoë, B. Apple Topped the Swiss in Holiday Watch Shipments . Available online: https://www.businessinsider.com/apple-watch-sells-more-than-swiss-watches-charts-2018-2 (accessed on 13 February 2018).
• Elearningfeeds. eLearning Statistics and Trends: 2020 . Available online: https://elearningfeeds.com/elearning-statistics-and-trends-2020/ (accessed on 14 December 2020).
• Miran, P.; Sunil, J.; Seungin, K. A Study on Application and Prospect with Prevalence Tablet PC -An Approach of Humanities, Science and Design. J. Korea Des. Knowl. 2011 , 18 , 84–93. Available online: http://www.dbpia.co.kr/journal/articleDetail?nodeId=NODE01772204 (accessed on 30 June 2011).
• Jisu, K.; Ko, H. Comparison of the Convolutional Neural Network models with an art dataset. In Proceedings of the 17th International Conference on Multimedia Information Technology and Applications (MITA 2021), Seogwipo, Korea, 5–7 July 2021; pp. 117–120. [ Google Scholar ]

Click here to enlarge figure

Share and Cite

Ko, H.; Lee, S.; Park, Y.; Choi, A. A Survey of Recommendation Systems: Recommendation Models, Techniques, and Application Fields. Electronics 2022 , 11 , 141. https://doi.org/10.3390/electronics11010141

Ko H, Lee S, Park Y, Choi A. A Survey of Recommendation Systems: Recommendation Models, Techniques, and Application Fields. Electronics . 2022; 11(1):141. https://doi.org/10.3390/electronics11010141

Ko, Hyeyoung, Suyeon Lee, Yoonseo Park, and Anna Choi. 2022. "A Survey of Recommendation Systems: Recommendation Models, Techniques, and Application Fields" Electronics 11, no. 1: 141. https://doi.org/10.3390/electronics11010141

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Help | Advanced Search

Computer Science > Information Retrieval

Title: recent developments in recommender systems: a survey.

Abstract: In this technical survey, we comprehensively summarize the latest advancements in the field of recommender systems. The objective of this study is to provide an overview of the current state-of-the-art in the field and highlight the latest trends in the development of recommender systems. The study starts with a comprehensive summary of the main taxonomy of recommender systems, including personalized and group recommender systems, and then delves into the category of knowledge-based recommender systems. In addition, the survey analyzes the robustness, data bias, and fairness issues in recommender systems, summarizing the evaluation metrics used to assess the performance of these systems. Finally, the study provides insights into the latest trends in the development of recommender systems and highlights the new directions for future research in the field.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Data Mining
• Recommendation Systems

Recommendation systems: Principles, methods and evaluation

• August 2015
• Egyptian Informatics Journal 16(3)
• CC BY-NC-ND 4.0

• Ekiti State University, Ado Ekiti

• Wentworth Institute of Technology

• Federal University of Technology, Akure

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Petrus Jacobus Kok
• Fahad Anwaar
• Adil Mehmood Khan

• Muhammad Awais

• Jonnifer C Mandigma
• Casielyn Capistrano
• Aleta C Fabregas
• Nisha Bhalse
• Ramesh Thakur
• Archana Thakur
• Ankur Ratmele

• Zhongshan Liu

• Anita Kumari Yadav

• Deuk-Hee Park
• Hyea-Kyeong Kim
• Il-Young Choi

• Brent Smith
• Jeremy York
• IEEE T KNOWL DATA EN
• Gediminas Adomavicius

• Yehuda Koren

• Chris Volinsky
• EXPERT SYST APPL

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sensors (Basel)

Movie Recommender Systems: Concepts, Methods, Challenges, and Future Directions

Sambandam jayalakshmi.

1 Department of Computer Science and Engineering, Vel Tech Multi Tech Dr. Rangarajan Dr. Sakunthala Engineering College, Chennai 600 062, India; gro.hcetitlumhcetlev@imhskalayaj (S.J.); gro.hcetitlumhcetlev@nhsenag (N.G.)

Narayanan Ganesh

Robert Čep.

2 Department of Machining, Assembly and Engineering Metrology, Faculty of Mechanical Engineering, VSB-Technical University of Ostrava, 708 00 Ostrava, Czech Republic

Janakiraman Senthil Murugan

3 Department of Computer Science and Engineering, Vel Tech High Tech Dr. Rangarajan Dr. Sakunthala Engineering College, Chennai 600 062, India; [email protected]

Associated Data

Not applicable.

Movie recommender systems are meant to give suggestions to the users based on the features they love the most. A highly performing movie recommendation will suggest movies that match the similarities with the highest degree of performance. This study conducts a systematic literature review on movie recommender systems. It highlights the filtering criteria in the recommender systems, algorithms implemented in movie recommender systems, the performance measurement criteria, the challenges in implementation, and recommendations for future research. Some of the most popular machine learning algorithms used in movie recommender systems such as K -means clustering, principal component analysis, and self-organizing maps with principal component analysis are discussed in detail. Special emphasis is given to research works performed using metaheuristic-based recommendation systems. The research aims to bring to light the advances made in developing the movie recommender systems, and what needs to be performed to reduce the current challenges in implementing the feasible solutions. The article will be helpful to researchers in the broad area of recommender systems as well as practicing data scientists involved in the implementation of such systems.

1. Introduction

Modern technology has revolutionized the volume, variety, and velocity at which data are generated. Digitalization of day-to-day experiences has led to the big data era. However, the enormous data have also led to the problem of information overload. Information overload may be defined as the state of being overwhelmed by the sheer volume of data presented to an average human for processing and decision making. Data mining methods can aid in obtaining and processing the relevant data and deal with the issue of information overload. Perhaps the most widely exploited tool among data mining methods is recommender systems.

Recommender systems work by assessing the available information about the likely patterns of the users and making suggestions from the information available [ 1 ]. The suggestions from the recommender systems help the system users find what is most suitable for them. Recommender systems are designed to ease product or service searches based on the least information available about the features [ 2 ]. A combination of various factors is used to assess the correlations in patterns and user characteristics to determine the best product suggestions for the customers [ 3 ].

The development of recommender systems depends on the field of application. The major application is in e-commerce websites where they suggest to the users the products or services based on the information available such as past search, age, gender, and other preferences [ 4 ]. They are also applied in job search platforms where the website suggests to a candidate the best possible positions fit for the skills. Since various industries have moved from an age of little available data to the era of big data, the junk information available is so much that it can delay the decision-making process. The recommender systems are typically made to ease the information search over the online systems so that the users find a more convenient way to connect to their preferences [ 5 ].

One of the applications of recommender systems is suggesting movies to watch to customers based on their preferences data. Movie recommender systems work by assessing the characteristic features of the users to make endorsements to the customers on what is best suited for them. It works by assessing the age, the previous preferences, gender, the content, context, and other demographic data to propose the movies. It checks the similarity among the users and items in the system to determine what could best fit the new user [ 6 ]. For example, a child will most likely receive recommendations for movies that children watch such as cartoons and animations based on the best similarity index for the children. Apart from that, children of various ages have different types of cartoons/animations to watch, and the recommender systems will propose the best depending on what other children of the same age are watching.

Movie recommender systems have helped the users overcome the chunk of information online to find only what is suited for them [ 7 ]. They use data mining techniques that match the similarities and help users find what is best suited for them [ 8 ]. Various criteria determine how the recommender systems work. The criteria are based on machine learning or deep learning algorithms that are used in matching the similarities before the suggestions are made. The algorithms achieve different levels of accuracy and require different computational times to retrieve the suggestions. Various computational algorithms have been proposed and used to increase the efficiency of recommender systems e [ 9 ]. However, each algorithm has its advantages and disadvantages; these make using the systems meet various needs based on their strengths. To reduce the limitations of each, the algorithms may be combined so that they perform better in making the recommendations [ 10 ].

This review paper aims to assess the challenges of recommender systems and make propositions to increase the accuracy of the systems. It assesses the recommendation approaches, the evaluation criteria of their efficiency, the challenges of these approaches, and possible solutions. A systematic literature review is conducted to determine the findings of the operational characteristics of the various recommendation approaches used and the performance criteria. The author aims to suggest the best solutions to make the approaches work better to achieve the operational expectations of the users.

The rest of the paper is arranged as follows: Section 2 details the methodology followed in this article. Section 3 describes different types of recommendation systems. Section 4 highlights some of the most popular machine learning algorithms used in movie recommender systems. Section 5 details the commonly used metaheuristic algorithms in movie recommendation tasks. Model metrics used for verifying the accuracy of recommendation systems are discussed in Section 6 . Some common problems with recommendation systems are discussed in detail in Section 7 . A critical discussion is presented in Section 8 . Section 9 presents the concluding remarks and the limitations of the study.

2. Review Methodology

This section describes the method used in obtaining information for the literature review. Peer-reviewed sources were used to gather information about movie recommender systems. The databases used were EBSCO Academic Search Premier, ScienceDirect, IEEE Library, ResearchGate, SpringerLink, and the ACM Portal. Google Scholar was also used to find leads to specific aspects of recommender systems for review.

Search Descriptors: Some of the keywords used in finding information about the movie recommender systems were “movie recommender systems”, “movie personalization”, “algorithms used in movie recommender systems”, “filtering techniques in movie recommender systems”, and “machine learning model metrics and measurement criteria”.

Inclusion Criteria: The inclusion criteria were papers that had information about recommender systems, the information had to be from published peer-reviewed sources. The paper abstracts were read to verify the validity of their information for use in this study. The exclusion criteria were papers that had grey literature on recommendation systems. The inclusion criteria for the articles and the methodology steps are summarized in Table 1 and Figure 1 .

Steps in conducting the systematic review.

Selection criteria for including sources in this review.

3. Movie Recommendation Systems

Movie recommendation work by filtering out data that is irrelevant and including only that which have matching characteristics or features [ 11 ]. As highlighted earlier, the world has moved from an era of scarcity of data online to an exponential growth in data. The systems work by manipulating the data to make sure it is efficient to drive data-driven decisions. In the jungle of available information about products, the systems need to evaluate what fits a certain customer and what does not. The systems go further in target and retargeting marketing to increase product viewership and hence increase the chance of the customers purchasing [ 12 ].

It is important for the developers to come up with systems that have higher performance characteristics and efficiency in matching the similarities in customer wants to seal the product sales or movie viewership [ 7 ]. The major types of filtering methods are collaborative filtering, content-based filtering, context-based filtering, and hybrid filtering.

3.1. Collaborative Filtering

Collaborative filtering works by matching the similarities in items and users. It looks at the characteristics of the users and the characteristics of the items the users have watched or searched for before [ 13 ]. In general, latent features obtained from rating matrices are looked at. In movie recommender systems, the recommendations are made based on the user information and what other people with similar user information are watching. For example, collaborative filtering in movie recommender systems picks the user demographic characteristics such as age, gender, and ethnicity [ 14 ]. Through these features, movie recommendations are made that match other people with similar demographic characteristics and previous user search history. Collaborative filtering suffers from a cold start if the user has not input any information, or the information is too little for any accurate clustering. In these cases, it does not know what to suggest [ 15 ]. The accuracy of the suggestion is also limited because people with similar demographic characteristics may not have similar preferences [ 16 ].

3.2. Content-Based Filtering

In contrast to collaborative filtering, content-based techniques employ user and item feature vectors to make recommendations. The fundamental differences between the two approaches are that content-based systems recommend items based on content features (no need for data about other users; recommendations about niche items, etc.) whereas collaborative filtering is based on user behaviour only and recommends items based on users with similar patterns (no domain knowledge; serendipity, etc.). A content-based filtering method works by making movie proposals to the user based on the content in the movies. It recognizes that clustering in the collaborative filtering recommendations may not match the preferences of the users [ 7 ]. The tastes and preferences of people with similar demographic characteristics are very different; what person X likes may not be similar to what person Y likes to watch. To solve this problem, content-based filtering algorithms give recommendations based on the contents of the movies [ 17 ]. In movie recommendations, some of the contents are the key characters and the genre of the movie.

3.3. Context-Based Filtering

This filtering technology is an improvement of the collaborative filtering method. It assumes that if person A and person B hold the same opinion on issue X, it is most likely that the same people will hold the same opinion/preference/thinking on a different issue Z. For example, if both people are attracted to Christmas movies from Netflix, it is most likely that they will still like Christmas movies by Showmax. The context-based filtering method recommends items with similar features or characteristics because the applications have just been extended to a different context [ 12 ]. It makes the same suggestions though the contexts are different. In most cases, web browsers import bookmarks and other settings when one upgrades from one browser to the next. This represents a change in context, since most of the settings and other items are imported into the new context, and the data available are used in making useful suggestions. Similarly, movie recommender systems may make a similar recommendation based on data from the previous context [ 18 ]. It is worth mentioning here about context-aware recommender systems (CARS), where the concept of context is well defined [ 19 , 20 , 21 , 22 ]. CARS acclimatize to the exact condition in which the recommended item will be used [ 23 , 24 , 25 , 26 ]. In this respect, CARS could avoid recommending a very long film to a user after a stressful day at work or suggest a romantic film if he/she is in the company of his/her partner.

3.4. Hybrid Filtering

This is a filtering technique that applies the concepts of all the other algorithms. It combines both collaborative filtering, content-based filtering, and context-based filtering to overcome the challenges of each method [ 10 ]. It is superior because it achieves higher performance in making the suggestions and also a faster computational time [ 11 ]. For instance, collaborative filtering may lack information about domain dependencies while content-based filtering lacks information about the preferences of the people [ 6 , 9 , 27 ]. A combination of these overcomes these challenges since user behaviour data and the content data are used to come up with recommendations.

4. Machine Leaning Algorithms for Movie Recommendation Systems

These are the algorithms that are used in filtering information and data mining so that the desired outcomes can be achieved. It is essential to understand the working of the information filtering methods so that the right algorithm is selected for the specific task in recommender systems [ 28 ].

4.1. K-Means Clustering

This is one of the simplest collaborative filtering approaches that categorizes the users based on their interests [ 29 ]. It is common for someone who wants to purchase an item to ask someone who has already purchased the product for their opinion. There is a higher chance that the influence of the current owner will affect the preferences and the tastes of the potentially new owner. Similarly, the algorithm compares the interesting features that can be associated with individuals that are classified to be within a group [ 30 ].

K -means clustering uses interests that are common among the users such as age, gender, movie time, history of the previous movies watched, etc. K -means clustering aims to group the features into clusters that represent the characteristics of the group [ 31 ]. If the classification is based on age, the probable K -means clustering will use children, teens, youth, and adult clustering methods. If a client falls within any of these age groups, movies are recommended based on what other people within that age group do. If the clustering depends on age, the closer an age is to the centroid age, the better the classification recommendation. The steps in the classification are measuring the similarity between the user and item features, selection of the neighbours, computing the prediction, and suggesting it [ 14 ].

4.1.1. Measurement of the Similarities

The first step is finding the similarity in the user features that the new user has with the previous system users. The algorithm always has the basic classifications for a beginning, where the user can give inputs and the predictions can be made [ 32 ]. Common features in finding the similarities are age, previous history, and geographical locations. Other recommender systems in movie theatres, including the price, the time to watch the movies, etc., are used in coming up with the means (centroids) for clustering. The distance from the centroids can be based on a Pearson correlation, cosine-based similarities, or an adjustment of the cosine-based similarity. The calculation of the similarity may be item-based or user-based. Item-based computation finds the similarities based on the features in the movies that similar people liked. If it is user-based, the calculation of the centroids is based on the demographic features of the user [ 15 ].

The computation of the similarities between items or users is shown in the mathematical equations below:

The equation above computes the correlation between the user and the item; it computes the closeness of the value to the centroid value. It is assumed that the two items i ∩ j are the correlated features (items or users); the value r j ¯ is the centroid feature, while the value r i is the value of the new user or new feature to be compared through correlation [ 33 ].

4.1.2. Selection of the Neighbours

There is always a consideration when developing the algorithm. The key metrics are the accuracy to obtain and the running time of the algorithm. To increase the accuracy of an algorithm, a large number of neighbours, which increases the computational time of the algorithm, is required. If a smaller computational time is needed, accuracy will be compromised [ 34 ]. To strike a balance, the selection may be threshold-based or use the top-N technique. The threshold technique will run only a specific number (sample number that meets the threshold value) of assessments of the neighbours and predict if that threshold is reached. For example, if the population is 1000, the system will run a prediction from 100 samples and predict out of the 100 samples [ 35 ]. In the top-N technique, only the top number of similarities (N) is run rather than the whole population of neighbours. For example, it will select only the top 10 for suggestions based on the nearest neighbours rather than assessing the whole population [ 36 ].

4.1.3. Prediction Computation

The computation of the subsequent predictions is based on the closest neighbours found in the system database. The prediction is obtained by the formula below:

The prediction or the nearest neighbour to the centroid ( K -mean) is made. In the equation above, the K -means is represented by r u ¯ while the correlation of the other variable on the right-hand side of the equation gives the nearest neighbour, both used in making the suggestion prediction [ 27 ].

4.1.4. Limitations of K -Means Clustering

• Cold-Start Problem : This is a prediction problem that happens with a new user to the system. There is very little information about the user; hence it is difficult for the system to make any predictions until the user starts feeding some information that can be correlated to and suggestions made based on the user or previous item characteristics. The negative impact is the system accuracy is greatly reduced [ 6 ]. It is because of this problem that new and excellent movies are not recommended to users, or new users do not find what is best for them.
• Sparsity in the dataset : The recommendation system involves assessing a large amount of data in the movie database. The users only look for a few items in the database; they are not able to use and assess a significant portion of the database to effectively evaluate the features. Apart from that, the users do not rate the movies they watched in the system. It becomes hard for the system to determine if the user liked the movie they watched, or they never liked it because they never left any rating. The negative impact is leaving some of the best movies not recommended in the large dataset since they have not been rated by the user. Moreover, the threshold/top-N techniques leave out the best matching suggestions [ 37 ].
• Scalability : One of the challenges cited in the selection of the neighbours was balancing the computational time and the accuracy of the system. The K -means filtering technique is accurate when the database has a small number of movies to recommend or few users [ 38 ]. However, with an increase in the number of users and the number of movies, the computational time or the threshold number of items increases; therefore, the computational time increases [ 39 ]. To overcome this disadvantage, computation and training of the algorithm are performed offline so that when the systems are back online, recommendations are made easily [ 40 ].

The K -means filtering algorithm is the most basic collaborative filtering technique. It is from this technique that other filtering concepts are developed. The computation technique to arrive at the predictions may not be the same; the mode of working mimics the K -means nearest neighbour [ 41 ]. The other algorithms are developed to overcome the K -means clustering limitations.

4.2. Principal Component Analysis K-Means

This is a content-based movie filtering technique that improves on the K -means clustering technique. The major components in the movie are used to classify the movies before recommendations can be made to the customers. The K -means algorithm calculates the closeness of a feature to the centroid using the distance from the mean point. However, the principal component analysis creates a covariance matrix to calculate the eigenvectors and eigenvalues [ 42 ]. Therefore, it widens the scalability to find better comparisons to make the movie suggestions [ 43 ]. To illustrate this, assume a K -means algorithm computes the similarity of a single feature at a time. This implies the computational time and accuracy are compromised. However, using PCA, a covariance matrix of various features is created; hence the scalability is increased and computed faster. If there are similarities that fall within the matrix, they can be found easily, and its eigenvector is computed. Suggestions close to such an eigenvector are then made to recommend the movies [ 44 ].

Steps in Conducting Principal Component Analysis

Structure of the tuples.

Where r i is the movie rating, u i is the user characteristics, and i i is the item characteristics (movie characteristics). A and B will give a 2D matrix of dimension m × n matrix, while C will give a 3D matrix of dimension m × n × o .

• Calculation of the covariance matrix: A covariance matrix of the dimension of the data formulated in the previous step is computed.
• Calculation of the eigenvectors and eigenvalues : The covariance matrix calculated will be a square matrix of the dimension of the data. It is used to compute the eigenvalues and eigenvectors which characterize the data. The computed eigenvectors are sorted in decreasing order according to the eigenvalues; a future vector is constructed [ 45 ].

4.3. Principal Component Analysis Self-Organizing Maps (PCA-SOM)

Self-organizing maps (SOMs) is a technique based on neural networks; it is an unsupervised learning technique, and there is no need for intervention of humans during the learning phase. It is vital in clustering data without knowing the class memberships in the input data [ 46 ]. The self-organizing feature map (SOFM) is known for detecting the features inherent in particular items which is important for the features in the movie recommender systems. SOM also uses topology-preserving mapping, which implies that the algorithm preserves the relative distance between all the points in the initial dataset [ 47 ]. Therefore, it effectively achieves the objective of transforming the arbitrary dimensions into a 1D or 2D discrete map. PCA is integrated with SOM because it is easier for the PCA to convert the matrices generated by SOM to eigenvectors and eigenvalues for ranking in the order of significance [ 48 ]. The steps in working out PCA-SOM are listed below:

• Obtain data without any rankings or classifications;
• Data modelling;
• SOM classified the data using unsupervised learning to bring together that which has similarities in features;
• PCA takes over from the classification achieved by SOM, checks the principal components, and comes up with further classifications of the dataset;
• The decision to make the suggestion.

Initialization: Once the data are obtained, random values are chosen for the weight of the initial vectors. The weights of the vectors represent the neurons in the data, and their values are also computed [ 49 ].

Sampling: A known sample x is drawn from the input space with a known probability. This is the activation pattern that is applied to the lattice. This pattern maps the x dimension to be proportional to the m-pattern in the new lattice [ 49 ].

Similarity Matching: The best matching is found at time step-n using the minimum Euclidean distance between the neuron centroid.

Updating: The synaptic weight of the neurons is adjusted using the formula below:

where η n is the learning rate, h j i x n is the neighbourhood function of i x winner neuron. These two are dynamic to obtain the optimum results.

Apply PCA: After the synaptic weights are derived from the minimum Euclidian distance from the formula above, the PCA process in creating the eigenvalues and eigenvectors is used further in processing the data to obtain a more accurate estimation [ 50 ].

Decision: After similarities are matched, the suggestions are made.

The great features of SOM that make it a good tool in recommender systems are:

Insights into the input space: The method uses unsupervised learning to classify the data by weight vectors and give output in a feature map. The cold starting is significantly reduced [ 51 ]. The user can then input data in light of the initial output features shown.

Topological arrangement: The feature map of SOM works by mapping the field of the input pattern to a spatial location in the output grid [ 52 ].

Density Matching: Once the input is fed into the system, any alterations in input distribution are equally represented in the output grid so that there will be a good representation of the highest density areas with the most matches and lower density areas with fewer matches [ 49 ].

Feature Selection: The SOM algorithm selects the best attributes for the non-linear distribution in the input data so that it can effectively match the similarities to the grids [ 50 ].

4.3.1. Advantages

Since it is based on unsupervised learning, it automatically updates the features and functions [ 53 ]. It is flexible to new input because it learns by itself. It is suitable for unidentified new inputs, for example, new movies that have no ratings or new users where there is no data about them. The new movies may be recommended when the system extracts their features, and the new users will not experience a cold start because they have somewhere to begin on the output feature map [ 54 ]. It is also faster in computation since it easily organizes complex data and makes a good representation of the mapping for easy interpretation.

4.3.2. Disadvantages

The major drawback is that feature classification may not be according to the expected output; therefore, the unsupervised learning classification algorithms have to be initialized often to maintain the relevance of the clustering [ 55 ].

5. Metaheuristic Algorithms for Movie Recommendation Systems

Metaheuristic algorithms are high-level methods or heuristics which have been developed to search, create, or select a heuristic that may produce a satisfactory solution for optimization problems. Metaheuristics find wide usage in almost all aspects of optimization problems. For example, metaheuristics have been used in design optimization [ 56 , 57 , 58 , 59 ], process optimization [ 60 , 61 , 62 , 63 ], structural optimization [ 64 , 65 ], knapsack problems [ 66 , 67 ], workflow scheduling [ 68 ], image segmentation [ 69 , 70 , 71 ], etc.

5.1. Genetic Algorithm

This is a hybrid filtering algorithm that uses the improved K -means clustering and is combined with the genetic algorithm (GA). It uses the PCA technique to partition the high dimensional space into clusters hence reducing the complexity of computations when making intelligent recommendations. The method has higher performance characteristics; hence it makes better recommendations. The steps of the recommendation system are outlined below:

5.1.1. Data Preprocessing Using PCA

The first step is processing the data, extracting it from the original high dimensional space into a linear relatively low space with denser features that carry the information. The PCA feature extraction technique has been very effective. It combines the data represented by the principal component with the highest eigenvalue with the significant information after ranking them. The components with lower significance are ignored but components with higher significance are given prominence. After the linear reduction, only a selected number of components from the rank is fed to the GA-KM algorithm for classification.

5.1.2. Enhanced K -Means Clustering Optimization by Generic Algorithms (GA-KM)

The objective is to make sure that the users/neighbours with like-minded interests or features are grouped. Therefore, it performs it in two stages which are K -means clustering and GA algorithms.

5.1.3. K -Means Clustering

The technique, as discussed, centres its clusters around centroids based on the linear distance from the central feature. The correlation of distance from the central point determines the similarity index. If it is too similar, there is convergence; if there is a high dissimilarity, then the dataset is sparse. As discussed, it suffers a cold start, and its first centroid may be based on the local optimum rather than the global optimum. The steps in K -means clustering are selecting the centroids, assigning objects to the closest clusters, computing the sum of squared distances from the members in the cluster, and checking for convergence in the computed objects. The procedure for computation is similar to that discussed.

5.1.4. Genetic Algorithm

This mimics biological evolution as explained by Darwin’s theory of evolution. The algorithm uses the population of individuals as chromosomes; the chromosomes represent possible solutions to the evolution problem [ 29 ]. Each of the chromosomes contains the genes with the survival ability. Therefore, through natural selection, the chromosomes with the highest quality genes have the highest chance of survival and are fit for reproduction for the next generation. The iterations are based on selection, crossover, and mutation. Selection picks just a proportion of the genes to breed for the next generation. Crossover swaps two parent chromosomes to be recombined into the offspring. Mutation randomly alters the value of a gene to produce offspring. The processes extend the diversity of the offspring. The processes end when the fitness conditions in the environment/context are met.

The GA algorithm is used to prevent premature convergence in the K -means algorithms. The centroids in K -means are considered the chromosomes; the fitness function to evaluate the quality of the solution is:

The fitness value is the sum of the distances of the inner points to the cluster centres. The values are minimized to find the optimal partitions. To find the optimal partitions, the three generic operators precede the construction of the offspring based on the survival fitness principles; convergence occurs when the fitness criterion is satisfied [ 72 ]. The pseudocode of the algorithm is summarized below:

• Initialization

Parameter initialization: Set the maximum iterations, population size, cluster numbers, probability crossover, probability mutation, and fitness function to minimize the total distance of every sample to its nearest centre;

Population initialization: Randomly generate the initial population for each of the k -centers.

Selection operation;

Cross-over operation;

Mutation operations;

Obtain the initial k -centres with optimal fitness values;

K -means optimization: generate new clusters with k -centres.

When tested with the MovieLens dataset, the algorithm has better performance features especially in reducing the cold-start problem [ 29 ].

5.2. Firefly Algorithm

The algorithm is also bio-inspired from the fireflies and combines it with a fuzzy C -means clustering technique. In the natural world, the fireflies are pulled to the brightest firefly using the light signal. Each firefly pulls the other, but the brightest has the highest attractiveness, and other fireflies are clustered around it. Similarly, the algorithm centres its suggestion on features of the users with the highest attractiveness (highest user ratings) [ 55 ]. If a movie has the highest rating from many users, the movie recommender system will make subsequent recommendations based on movies rated highest by users with similar characteristics [ 73 ]. The algorithm for the recommender system is highlighted below:

• All the fireflies are unisexual, and every firefly pulls to another firefly.
• The attractiveness of a firefly depends on its brightness, and the other fireflies will be pulled closer to the brighter one (feature reduction using the firefly algorithm).
• This brightness is related to a primary function in the FCM.
• FCM allocates memberships and utilizes them to show data elements from one cluster to another.
• The FCM separates a finite set of elements X = X 1 … .   X n from the memberships into a set of c fuzzy clusters; hence it comes up with a list of cluster centres C = C 1 …   C 2   . The partition matrix W = W i   0 , 1 ;   i = 1 …   n ;   j = 1 … c expresses the degree to which each element X i is placed into a cluster C j . The aim is to reduce the objective function to optimal. C a r g   ∑ i = 1 C ∑ j = 1 C W i j M   X i − C j 2   (6)
• Then the fuzzy C -means clustering f l m m = 1 ∑ K = 1 C   x i − c m x i − c k   2 n − 1   (7)

The recommender system efficiency and performance are generally higher than the traditional K -means clustering.

5.3. Artificial Bee Colony

This is a bioinspired algorithm that makes recommendations based on the workings of the bees in finding flowers for the best nectar [ 74 ]. The bees are mainly divided into two groups. Scouting bees go out to scout flowers with the best nectar, and the employee bees follow after the best flowers have been found [ 53 ]. It is worth noting that several scouting bees are sent out and come back with information to the hive regarding the quality of the nectar found. The employee bees will filter out the low-quality nectar from the information and follow the scout bee to the source of the best nectar. Similarly, the artificial bee colony in recommender systems works as an improvement of the K -means clustering algorithm [ 75 ]. In the K -means algorithm, an assumption is made that the data is based on a centroid where the closeness of the feature to the centroid feature determines the recommendation. In an artificial bee colony, there are many centroids (just as there are many flowers), and information from or to these centroids will bring a variety. From this variety, the user may choose what is best suited for them; henceforth, the recommendation system will bring recommendations close to the centroid chosen [ 76 ]. It is a good method to solve the sparsity, scalability and cold-starting problem. The user will choose the best suited feature from the first random set of options available. Subsequent recommendations will depend on the K -means around that particular choice. The steps are summarized below:

• Initialize the system users and movies in a matrix;
• Use the K -means clustering to find several centroids of various product features. This finds several centroids for clustering;
• Selection of the nearest clusters;
• Calculation of the estimated rating values from the user history;
• Use the artificial bee colony to select the closest to user likes based on ratings and features;
• Reclassification of the users for further iterations;
• Coming up with the recommendations.

ABC determines the community of vectors that explore the similarities in the neighbours. The objective function is then continually reduced when narrowing down to the nearest possibilities [ 77 ]. The aim is to minimize the objective function below:

The objective function is controlled by succeeding iterations determined by the detecting vector z → as below:

From the succeeding iterations, the points with the most similar features are selected, and the system recommends the movies to the user [ 78 ]. For instance, an initial allocation of the centroids may be classified as horror movies, thrillers, comedy, or thrillers. If the client selects thriller movies, a further classification may be Hollywood thriller, Bollywood thriller, etc. If the client selects any from these, the subsequent recommendations will be based on this particular centroid [ 74 ]. As seen, it has an advantage because there are always initial centroids that the user can select that further narrow down the selection. Optimization is reduced by the detecting vector to optimize the suggestion to the most viable suggestions [ 79 ].

5.4. Cuckoo Search

The cuckoo search algorithm is a combination of K -means clustering and the use of Levy’s flight function. In this process, the K -means algorithm divides the MovieLens Dataset into different clusters. This is performed using randomly selected centroids [ 14 ]. Measures such as the Euclidian distance and cosines are used to find the distance between centroids, and the features and/or users are reassigned to the closest cluster with similar characteristics.

The cuckoo search algorithm gets its inspiration from the cuckoo bird. The cuckoo bird does not sit on eggs to hatch; rather, it searches for the best nest with optimal conditions and lays eggs for the host bird to sit on to hatch [ 80 ]. If the host bird identifies the egg, it may throw it away or abandon the whole nest. Similarly, in the recommender algorithm, if the centroid does not present the optimal solution, the centroid is abandoned for a new iteration until no re-assignment happens. The pseudocode for the recommender system is outlined in the procedure below:

5.4.1. K -Means Clustering

• Initialize the number of k clusters;
• Random selection of centroids using K -means clustering;
• While no centroid is changed, assign each data point to the closest centroid and calculate the new centroids;
• Assign data points to the closest cluster mean.

5.4.2. Cuckoo Search Algorithm

• Begin the fitness function f x ,   X i =   x 1 ,   x 2 … ;
• Initialize the random population of n host nests (centroids from K -means);
• Calculate the fitness function value for each nest;
• Find the ith cuckoo randomly by Levy flights, and calculate its fitness, Fi ;
• Select a nest (centroid);
• If ( F j > F i ) ; replace j with the new solution;
• The unqualified nests (centroids) are abandoned and new ones built by Levy flights function;
• The best solutions found are ranked and suggested to the client.

The cuckoo search algorithm has higher precision, recall, and a lower MAE than the PCA- K -means, hence higher performance characteristics. The computations can also be performed offline so that recommendations are made when the system is online, making it faster to make suggestions to the user.

5.5. Grey Wolf Optimizer

This is a recommendation system that is based on mimicking the leadership and hunting tactics of the grey wolfs [ 81 ]. The algorithm first conducts feature selection using the grey wolf optimizer (GWO) method, before clustering using the FCM method [ 82 ]. The algorithm pseudocode is listed below:

• Load the GWO culture;
• Initialize the coefficient points r, Q and R;
• The appropriateness of each explorer is estimated X α   X β   X δ ;
• Carry out iterations to determine the appropriateness of entire explorer negotiators;
• Return X α representing the position of centroids by GWO;
• Randomly select the cluster centres based on fuzzy means;
• Load the fuzzy clustering formula matrix and estimate f l m m as in the formula below: f l m m = 1 ∑ K = 1 C   x i − c m x i − c k   2 n − 1 (10)
• Determine midpoints B (k) = [ c m ] with F (k) ; continue with iterations until ||F( k + 1) − F( k ))) < ε;
• Return to the newly formed cluster centres and make recommendations based on the cluster centres.

This recommender system has a relatively better performance.

5.6. Other Metaheuristic Algorithms

Some researchers have used other metaheuristic algorithms to develop movie recommender systems. For example, Papneja et al. [ 83 ] developed a movie recommendation using a whale optimization algorithm. Tripathi et al. [ 84 ] hybridized a map-reduce-based tournament along with a WOA to achieve a superior recommendation experience.

6. Model Metrics

Various aspects have to be measured apart from the accuracy to make sure that the algorism makes the right predictions. For example, the algorithms may be highly accurate but have too much logarithmic loss. Accuracy is not the only metric to determine the performance efficiency of a model. The metrics are discussed below:

6.1. Mean Absolute Error (MAE)

This is the average difference between the predicted values and the original values. In our case, it is the average difference between the choice of the movie by the customer (user) from the suggestion made (prediction). It gives the variation between the suggestion and what the customer chose. The only disadvantage is it does not give the direction of the error [ 85 ]. Generally, a low mean absolute error is desirable. The mathematical formula for MAE is shown below:

where n s is the number of samples, y ¯ i is the predicted suggestion, and y i is the true feature that the user picks/wants.

6.2. Mean Squared Error (MSE)

This gives the square of the MAE (the square of the average difference between the original values and the predicted values). The advantage is that it makes the large errors more pronounced so that the model focuses on the large errors and their causes [ 86 ]. In addition, it is easier to model the linear programming models in the computation of the slope using the mean absolute error since the differences will be clearer. The formula for the MSE is shown below:

6.3. Log Loss

This is a cross-entropy loss given by probability estimates. It is used in neural networks and recommender system optimizations. It calculates the probability of the suggestions rather than giving only discrete predictions, especially during the ranking of the suggestions [ 35 , 87 ].

6.4. Confusion Matrix

This is one of the most used metrics in determining the accuracy of a model. It is mainly used for classification problems, especially when the outputs expected should have more classifications [ 34 ]. The various characteristics of the confusion matrix are shown in Table 3 .

Characteristics of a confusion matrix.

As noted earlier, the movies are clustered based on the features or the users. In clustering, the true represents the actual classification of the movie, while the predicted gives the predicted classification of the movie before recommendation [ 88 ]. For example, a movie may be classified as a comedy when it is a thriller movie. The user may choose to think it is a comedy because of the characters only to find it is a thriller movie. The movie may be classified as a thriller, and it is a thriller; hence the users get what they want. Such variations happen in the movie classification; hence there is a need for accurate predictions.

True Positive: This points to a case in the recommender system where the actual suggestion was positive, and the client selection of the movie was positive, i.e., the system suggests what the client needed. An example is when the movie is classified as a comedy when the client needed comedy and selects it [ 89 ].

True Negative: This happens when the actual classification is negative, and the prediction is also negative. In our movie recommendation example, if the movie is not a comedy and our algorithm does not classify it as comedy, this output is termed as true negative [ 90 ].

False Positive: This happens when the actual classification is negative, but the system predicts it as positive [ 91 ]. In the movie recommendation system example, the actual case may be the movie is not a comedy; yet the prediction algorithm classifies it as comedy. The actual movie is not a comedy, hence negative, but the prediction is comedy, hence the term positive.

False Negative: This happens when the actual data is true (positive), but the prediction is negative (false) [ 87 ]. The actual classification is true; yet the system predicts it as negative. In the movie recommender systems, the actual movie is a comedy, but the system predicts not comedy.

6.5. Precision

This assesses how many of the true positives are true positives. It gives a fraction of the true positive predictions to the total positive predictions [ 92 ]. The mathematical format is shown below:

6.6. Recall/Sensitivity

This is used as the fraction of true values from the total true values [ 35 ]. In our recommender systems, it gives a fraction of what is classified as comedy out of a total of what is comedy. The fraction of the true values out of the total true values is mathematically modelled as below. Note that false negative is true, but the algorithm classified it as else:

Precision focuses on capturing the classifications correctly, while recall focuses on whether the system is able to capture the features we want though they may not be correctly captured [ 34 ].

6.7. Accuracy

This is the fraction of correct predictions with the total predictions. The accuracy is mathematically modelled as below:

Accuracy as a metric should be used only when the data are balanced and have various classes [ 93 ]. It should not be used as a metric when the data are skewed (have a majority of only one class). For example, if the data are made of 100 movies, and only 5 movies are comedies, the rest are different genres such as thrillers. If the algorithm wrongly predicts all the movies as thrillers, it will be 95% accurate because 95% are thrillers, and only five are comedies. However, from a rational standpoint, the algorithm failed to classify comedy movies. In a recommendation system, the client would think that none of the movies in the datasets is comedy and not watch; yet there are five top-rated comedy movies. Therefore, accuracy should be used when the dataset is well-balanced.

This is simply the harmonic mean of precision and recall. It shows how precise the system was and how it never missed significant instances. If the F 1 score is high, the model performance is high [ 94 ]. The mathematical formula for precision is shown below:

6.9. Computational Time

This is the time that the algorithm takes to come up with the final solution in the prediction. If the systems take long, they may be unreliable if the users want an immediate response before they select the movies to watch. The algorithms should ensure that the best results are found within the shorted period possible [ 53 , 95 ]. A high-performing system returns the most efficient results within a relatively short period. It increases its reliability and dependency. Sometimes the data to be analyzed may be too large to give immediate results. To overcome this limitation, the computations are performed when the system is offline so that the output is shown when it is back online for effective prediction suggestions [ 96 ].

7. Problems Associated with Movie Recommender Systems

7.1. cold start.

The best target audience for a recommendation always depends on the previous user characteristics and the features of the products they watched. A comparison is always made on the characteristics of the user was and the features of the movie and the rating given to the movie. However, in some instances, there are no user characteristics that can be used for a recommendation if the user is new, and nothing is known about them [ 97 ]. Sometimes the user may not be new but has used a different device when accessing the movies’ websites hence there are no stored cookies that can trace the user history.

A cold-start problem occurs when the recommender system is not able to make any suggestions to the user because the user is new or there is no information available about the user [ 98 ]. The problem is common in collaborative filtering which uses only user details to make recommendations of the best movie. The problem is overcome by using content-based filtering, context-based filtering and hybrid filtering. In content-based filtering, the movies are classified by the features such as the main characters, the genre etc. The new user will select any genre based on the content. A context-based filter is based on some of the user information derived from the device such as location, and the operating system, and correlates them with what other users from similar contexts are using. In hybrid filtering, the content, context, and user characteristics are used, therefore, if the recommender system does not have any information about the user, it will use the content and context to make the first recommendations [ 99 ]. The subsequent recommendations will depend on the hints of information available.

7.2. Accuracy

If the database for the recommender systems has few movies, the system will have higher accuracy. If the database is large, there tends to be a lower accuracy because the pool of information searched is too large. To counter the problem, the K -means algorithm reduces the computational time by restricting the computation to a certain number of iterations or selecting only the top-N number of movies for recommendations [ 100 ]. However, if some of the movies have never been rated, they are likely to be biased in the searches [ 101 ].

To increase the system accuracy in making the recommendations, some of the algorithms have employed sophisticated search criteria that will conduct a thorough search and match the product features to user and item characteristics [ 102 ]. In addition, two or more algorithms are combined to allow the perfect user and feature analysis and come up with the desired output. Some of the classification processes such as the PCA-SOM conducts the logarithmic computations offline so that they give recommendations easily when they are online. It reduces the computational time and increases the recommender system accuracy. In modern recommender systems, cold-start problems and the accuracy is solved by having a dialog box where the users can type in the features they need, and recommendations will be given according to what matches the search words [ 103 ].

7.3. Diversity

New movies in the recommender systems rarely appear among those that are suggested to the users. The new excellent movies may end up not being watched because of the lack of being rated by the users. Some of the excellent movies also may not be rated, leaving the recommendation system blank about whether the movies are great for a specific class of watchers or not. To overcome some of these challenges for new movies and movies that are not rated by the users, the diversity aspect is introduced by the recommender systems. In diversification, the new movies or unrated movies are given priority so that they can be noticed by the users [ 104 ]. If they are pleasing, they will be rated and watched more. From this information, the recommender systems will make subsequent decisions on whether to recommend the movie or archive it. From the number of watches, it will also classify the movie according to the features or user characteristics [ 102 ]. Diversity is often used to recommend new debut movies to increase their marketability and presence. It increases the diversity of the user to try out new features or new products.

7.4. Scalability

While sparsity and diversity aim to increase the chances of movies with new features appearing in top searches, scalability aims to solve the problem of increased computational time and increase the performance of the recommender system. Scalability ensures that there is a balance obtained between accuracy and computational time. If it is necessary, some of the classification computations are performed beforehand so that by the time the user comes to select an item to watch, the system makes an almost immediate recommendation with high levels of efficiency [ 105 ].

7.5. Sparsity

Sparsity in the movie data relates to the large volume of movie data in the system, but the users only utilize a few of the features or resources. It is common in K -means clustering where the data is interpolated linearly and gives fewer perspectives to the non-linear data [ 106 ]. Recommendation systems may sometimes be biased by only suggesting the most rated or the most liked movies based on a limited assessment of all the possible cluster features. By using the top-N theory to make the recommendation, those that do not meet the threshold of these can find better algorithms that consider the sparsity of information available [ 107 ]. Some of the methods such as PCA-SOM map all the features on a lattice; hence the user can find most of the features. WOA also widens the scope of the search by using both linear and orthogonal systems to find the desirable features and make recommendations that are sparser and more diverse. Generally, implementing systems that consider non-linearly related data is efficient [ 108 , 109 ].

8. Discussions

The current movie recommendation systems have to work in contexts where there is so much data to be considered before making recommendations. Both user and context information are so varied that the accuracy and precision of the systems are brought to real tests. For example, most of the user information is shared through social media platforms to generate interest in the movies. The MovieLens dataset was created approximately 20 years ago when there was little or no developments in the use of social media where users share movie information to create interest. However, current technologies need to analyze the content, context. and user characteristics in social media platforms to recommend the right movies to the customers. Some companies have taken steps to integrate analytics in their recommender system algorithms. They ask the customer to connect to their social media accounts such as Twitter, YouTube, and Meta not only for advertising but also to analyze the activity of the user on these social media accounts to recommend the best movies for them. Through connecting to these platforms, they analyze the previous history of the user and recommend appropriate movies. This significantly reduces the cold-start problem since new user information can be obtained.

Context-based filtering is gaining traction in the movie recommender systems. It has been adequately used in product recommendations on e-commerce platforms., for example, the most discounted products during black Fridays, the holiday products during Christmas seasons, etc. Movie recommender systems that integrate time stamps to recommend the best movies in various contexts should be studied and developed. For example, it will help recommend movies for children learning during the day and children lullaby movies when it is time to sleep at night.

There are various advances in the use of blockchain technology, and some of these applications may affect the efficacy of algorithms in movie recommender systems. Blockchain technology enhances user privacy through user data encryption; yet collaborative filtering depends on the availability of user information so that it can match the features and characteristics before making recommendations. If user information is concealed by the blockchain systems, the algorithms have to use advanced methods to prevent a decline in the accuracies such as the use of context and content-based filtering.

9. Conclusions

In this article, movie recommender systems have been described and classified. The various types of recommender systems are introduced and discussed. Special emphasis is given to explain in detail the various machine learning and metaheuristic algorithms commonly deployed in movie recommendation research. The various model metrics that summarize the quality of the model are discussed at length. The problems associated with movie recommender systems are also summarized in a structured way and discussed. A total of 77 articles strictly on the area of movie recommender systems are included in the study, and their major conclusions are presented. In addition, 32 other related articles on metaheuristics and recommender systems (not for movies) are also introduced in various sections to present a coherent and meaningful review. One of the limitations of the study is that the Scopus and Web of Science databases were not directly used for selecting the articles for review. In contrast, EBSCO Academic Search Premier, ScienceDirect, IEEE Library, ResearchGate, SpringerLink and the ACM Portal were used for the literature search. Nevertheless, more than 80% of the reviewed papers were found to be indexed in Scopus while more than 60% were available in the Web of Science database.

Funding Statement

This research received no external funding.

Author Contributions

Conceptualization, S.J., N.G. and R.Č.; data curation, S.J., N.G. and J.S.M.; formal analysis, S.J., N.G., R.Č. and J.S.M.; investigation, J.S.M.; methodology, S.J., N.G. and R.Č.; supervision, R.Č.; writing—original draft, S.J., N.G. and J.S.M.; writing—review and editing, R.Č. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Informed consent statement, data availability statement, conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

ORIGINAL RESEARCH article

Developing and evaluating a university recommender system.

• 1 Behavioral Data Analytics & Recommender Systems Research Group (DARS), Department of Information Science & Media Studies, University of Bergen, Bergen, Norway
• 2 Marketing and Consumer Behaviour Group, Wageningen University & Research, Wageningen, Netherlands
• 3 Software and Systems Engineering Research Group, Free University of Bolzano & Research, Bolzano, Italy

A challenge for many young adults is to find the right institution to follow higher education. Global university rankings are a commonly used, but inefficient tool, for they do not consider a person's preferences and needs. For example, some persons pursue prestige in their higher education, while others prefer proximity. This paper develops and evaluates a university recommender system, eliciting user preferences as ratings to build predictive models and to generate personalized university ranking lists. In Study 1, we performed offline evaluation on a rating dataset to determine which recommender approaches had the highest predictive value. In Study 2, we selected three algorithms to produce different university recommendation lists in our online tool, asking our users to compare and evaluate them in terms of different metrics (Accuracy, Diversity, Perceived Personalization, Satisfaction, and Novelty). We show that a SVD algorithm scores high on accuracy and perceived personalization, while a KNN algorithm scores better on novelty. We also report findings on preferred university features.

1. Introduction

Recommender Systems have been used in various domains to retrieve and suggest personalized content to users. Most commonly, they are used to support low-stake decisions in consumerism domains that have leisure-related goals, such as when deciding what movie to watch ( Gomez-Uribe and Hunt, 2015 ), or what product to promote on an e-commerce website ( Jannach et al., 2010 ). In domains where the “decision stakes” are higher, because the user decision is important or costly, the recommender approaches or algorithms should also be aimed at optimizing diversity and longer-term preferences, instead of maximizing short-term engagement only. This applies to, for instance, recommender systems that promote healthy foods or suggestions for real estate ( Yuan et al., 2013 ; Starke and Trattner, 2021 ; Starke et al., 2021b ).

A domain with high-stake decisions and a large potential choice set is university education. This applies to choices one can make while attending higher education, such as what college major to take and what electives to follow ( Dwivedi and Roshni, 2017 ; Khoja and Shetty, 2017 ; Obeid et al., 2018 ), as well as to the decision of attending a university or another higher education institution. Whereas the former has been the topic of various recommender system and learning analytics approaches [cf., Hasan et al. (2016) ], universities are rarely featured in personalized approaches ( Rivera et al., 2018 ). This is arguably surprising, because a significant proportion of students attending higher education in G20 countries is not native to those countries ( OECD, 2013 ) – even though most prospective students opt for institutions that are close to home, thus based on proximity ( Simões and Soares, 2010 ; White and Lee, 2020 ). Those who would like venture further in terms of proximity, would benefit from a personalized information-filtering system, such as a recommender system, since there are over ten thousand of higher education institutions worldwide to choose from 1 .

One's choice for higher education not only determines where one needs to move to, but also affects one's future ( Kanoje et al., 2016 ). The recommended content should not only reflect one's current preferences, but also future prospects ( Ekstrand and Willemsen, 2016 ), such as factors whether one wishes to focus on proximity for short-term benefits or if one seeks out prestige for long-term gains. Hence, choosing a university and enrolling into one is a vastly different experience from, for instance, watching a movie. People may watch a lot of movies during their lifetime, while experiencing universities is costly and often not feasible. One underlying reason is that people do not always have a clear idea on where to obtain helpful information, beyond the website of a specific institution and popular university rankings ( Hemsley-Brown, 2012 ). Therefore, accessing a system that could effectively support such decision and personally help in this choice process could be as beneficial as a personal adviser would.

Current online services that assist users by ranking universities typically use explicit criteria, for example by applying decision filters ( Rivera et al., 2018 ). Most tools are built by comparing a large number of universities, renowned colleges and schools ( Hemsley-Brown, 2012 ). Among the most well-known resources are international university rankings, such as the World University Rankings compiled by Times Higher Education. 2 For such ranked lists, different comparison dimensions are typically considered and weighted, and a final score is computed for each institution, for example based on a university's reputation and produced patents. Such a score could indicate how desirable a higher education institution could potentially be for a user. However, such rankings are compiled generically and are computed equally for everybody, ignoring the specific set of preferences that each person may have, which may not be reflected by such a one-size-fits-all ranking.

1.1. Approach

This paper addresses this problem by proposing a system that provides personalized ranking lists of universities. We go beyond current applications in the field that, in most cases, rely on search functionalities and a limited set of criteria filters, such as a university's geographical location (see CollegeBoard, 2019 for an example). We compare different algorithms that are well-known in the Machine Learning community, and can be used to produce a list of university recommendations. Although there have been a few studies that have attempted to build customized university rankings ( Hasan et al., 2016 ; Bodily and Verbert, 2017 ; Rathore, 2017 ; Rivera et al., 2018 ; CollegeRaptor, 2019 ), to the best of our knowledge, none of the current world-class university rankings offer a customized ranking list that is tailored to the particular preferences and needs of the users.

We explore the effectiveness of personalized university recommendations and the users' decision-making process in more detail. This paper extends findings from our previous short paper ( Elahi et al., 2020 ), that explored the possibilities of different Collaborative Filtering (CF) recommender algorithms, as one of the most popular types of recommender approaches. First, we compare the effectiveness of different CF-based algorithms, reporting the findings from an offline simulation study. Specifically, we measure the rating prediction accuracy of different approaches. Second, we present novel findings from our online user study, in which we have examined different aspects of university recommendation. Not only do we inquire on specific features that people (or users, for that matter) pay attention to when considering to choose a university, but we also assess which recommender approach is most suitable to model user preferences for specific universities, based on collected ratings and algorithmic evaluation. Finally, we validate the use of our recommender interface by asking users to assess the system's usability.

This work significantly extends initial findings from an offline simulation study, presenting the results of an online comprehensive evaluation methodology. To do so, we employ a wide range of validated of metrics to measuring the quality of recommendation perceived by the real users participating in the evaluation of the up-and-running university recommender system. This has been conducted using a set of beyond-accuracy metrics, including Diversity, User Satisfaction , and Novelty . To our knowledge, none of the prior works have conducted such a comprehensive experiment with similar setup in this application domain.

We examine the following research questions:

• RQ1: Which recommender approach has the highest predictive value when generating personalized university rankings?

• RQ2: How do users perceive and evaluate different university recommender approaches?

• RQ3: What are the most important features that users consider when choosing a university to attend?

1.2. Related Work

An increasing amount of data is being collected in the context of education. To make sense of this and to employ it effectively, data analytics in the context of education has become more common in the past decade. The use of so-called “learning analytics” often aims to predict a student's course performance based on interaction data ( Conijn et al., 2016 ; De Medio et al., 2020 ). However, the extent to which data-driven predictions to date are robust seems to vary. For example, in the context of Learning Management Systems, using student data (e.g., interaction times, clicks) to predict course performance shows strong differences across different courses ( Conijn et al., 2016 ). Moreover, although such techniques provide insight to the system owners and managers, they often do not help students with education-related problems, such as deciding what course to follow next.

The task of predicting a student's performance overlaps with more traditional retrieval or recommendation tasks surrounding course content. However, personal education is among the lesser-explored recommender domains ( Dascalu et al., 2016 ; Bodily and Verbert, 2017 ). For one, recommender systems have been used to predict student performance as a means for intelligent tutoring systems, by assessing the difficulty of different course components ( Thai-Nghe et al., 2010 ). The scope of educational recommender systems can vary strongly ( Rivera et al., 2018 ), both in terms of what algorithmic approaches are used and what areas of education are covered. With regard to the former, it seems that collaborative filtering (CF) and hybrid approaches that involve a CF component are most popular ( Rivera et al., 2018 ), arguably because Learning Management Systems (LMSs) generate a lot of interaction data from which student-related parameters can be distilled ( Conijn et al., 2016 ; Hasan et al., 2016 ).

The types of areas of education covered can vary in terms of level of education, scope, and level of detail ( Rivera et al., 2018 ). Older recommender studies in an educational context examine how individual learning tasks can be recommended. For instance, personalized recommendations to learn English and measurement models for writing ability could help systems to determine which task or assignment is suitable for which student ( Engelhard, 1992 ; Hsu, 2008 ). More contemporary methods, such as through fuzzy linguistic web methods, have also been employed in the past decade to move toward personalized education approaches ( Tejeda-Lorente et al., 2015 ). Among other approaches, recommender systems are used to suggest personalized content on e-learning platforms, as well as to generate personalized curricula for a given major or university education ( Meryem et al., 2016 ).

Recommenders are also used to predict student performance before they enter higher education. For example, college admission recommender systems can guide higher education staff on decisions on whom to admit to their program ( Ragab et al., 2012 , 2014 ). Similar approaches have also been employed for university admission, typically using hybrid approaches ( Wakil et al., 2014 ).

More contemporary systems aim to recommend courses or college majors ( Dwivedi and Roshni, 2017 ; Khoja and Shetty, 2017 ). A recent work-in-progress proposed a method to introduce an ontology-based recommender system to help high school students to navigate college majors and to select one, along with a university ( Obeid et al., 2018 ). However, an empirical recommender study (e.g., with crowdsourcing data) has yet to be performed, as Obeid et al. (2018) only identified the student requirements, interests, and capabilities. Nonetheless, identifying such a set of relevant features may be a good starting point to effectively perform preference elicitation.

Dwivedi and Roshni (2017) present a collaborative filtering approach to recommend elective courses to university students. This is based on a student's performance across different courses and computing the inter-item similarity between courses, which is also found in a few other proposed approaches ( O'Mahony and Smyth, 2007 ). Other recommender approaches for courses in higher education are content- or knowledge-based, or hybrid ( O'Mahony and Smyth, 2007 ; Khoja and Shetty, 2017 ). In a similar vein, such approaches also employ similar-item retrieval to generate course recommendations that are close to a course that a student is currently following. Moreover, a study that employed a knowledge-based recommender system also explored more detailed aspects of higher education, such as how to match a student to a supervisor ( Samin and Azim, 2019 ).

Related work that suggests content at the institution level (i.e., which university to attend), particularly in an international context, is much rarer. While some recommender studies have examined university recommendation as a topic because of an interest in college major advice ( Obeid et al., 2018 ), there is little work that examines specific characteristics of the university beyond its majors ( Bodily and Verbert, 2017 ). Studies to date have suggested different approaches that are examined using offline evaluation. For example, Bokde et al. (2015) perform dimensionality reduction techniques (i.e., Singular Value Decomposition) in a collaborative filtering approach, based on student ratings for different criteria. The study is, however, unclear about how data was collected, which makes it hard to generalize the approach.

1.3. Contribution

What stands out from the corpus of related work is that few studies on university recommender systems have been performed. Moreover, those that have been reported typically rely on offline evaluation to predict user ratings and, at times, to generate a personalized list of university recommendations ( Bodily and Verbert, 2017 ; Rivera et al., 2018 ). A rigorous two-step approach, in which the predictive accuracy of different recommender approaches is compared (cf. [RQ1]), after which also an online evaluation takes place (cf. [RQ2]), is much rarer.

On top of that, and in line with an earlier study ( Bokde et al., 2015 ), we also inquire on the most important aspects for selecting a university (RQ3). This can provide insight into future university recommender interfaces, which may also involve filtering criteria, such as knowledge-based recommenders Jannach et al. (2010) . Finally, to validate whether our university recommender interface is acceptable and understandable, we ask users to assess the usability of our university recommender system. Previous works on university recommender systems have mostly been evaluated offline ( Rivera et al., 2018 ). As a result, there has been little attention for interface design and whether the presented recommendation list aligns with a user's preferences and needs. In addition to addressing this omission through a user's perceived evaluation, we will examine this through the system's perceived usability ( Brooke, 1996 ).

2. Materials and Methods

This section covers how our university recommender system is set up, in terms of data collection for ratings, features, and algorithms. Moreover, we explain how we evaluated our algorithms: both for our offline and online studies.

2.1. Recommender System

2.1.1. system development and procedure.

To address our research questions, we developed a system prototype capable of interacting with users and learning their preferences for different universities. The system architecture is illustrated in Figure 1 , which included several components, interconnected and operational to generate real-time recommendations of universities. When a new user entered the system (on the left of Figure 1 ), it initiated a registration process in which the system requested a user to disclose personal information. This was followed by eliciting user preferences in two different ways. First, by inquiring on what a user believed to be the most important features when choosing a university to study. Second, by asking users to rate known universities in terms of favorability. Additional information was collected in the form of personality traits, preferred university features and favorite countries. All data were passed on to a recommender algorithm in order to generate lists of personalized recommendations. Finally, users were requested to interact with the evaluation component and to assess the quality of a recommendation list.

Figure 1 . System Architecture for our University Recommender System. It depicts the flow of information in our system, as well as the different steps and features that users (depicted on the left-hand side) take when using our system.

For the development of the system, a LAMP stack was employed. This entailed the use of a APACHE server for hosting the web project's PHP files, several JS/HTML/CSS files for the front-end, and developing custom PHP scripts for the server-side processing of user data. We adopted MySQL database to store the user demographics, item descriptions, ratings, and experimental data. In order to train the recommendation engine, a separate server was utilized, based on a set of RESTful API endpoints. Finally, a customized version of WordPress content management system was developed with a specific front-end theme.

2.1.2. Algorithms

We utilized multiple algorithms for our university recommendation. Most of them followed a popular recommendation approach called Collaborative Filtering (CF) ( Jannach et al., 2010 ). Algorithms based on CF exploit ratings provided by a community of users in order to predict the unknown ratings of the items. The items with the largest predicted ratings are recommended to users ( Elahi, 2014 ). We adopted three categories of CF algorithms in this work: Neighborhood-based, Matrix Factorization, and co-clustering.

Neighborhood-based approaches are a category of algorithms that calculate rating prediction using two sets of preference data: the ratings of the user for other items and the ratings of other similar users. A prominent example of such an algorithm is K-Nearest Neighbors (KNN), which would identify k number of neighbors that are relevant for a particular user ( Jannach et al., 2010 ). The item's rating prediction is calculated based on how the item was rated by the users similar to the target user. The rating r ^ u , i for a user u and an item i was predicted as follows:

where r ̄ u denotes the mean rating of user u , sim ( u, u ′) is a similarity metric between two users u and u ′, and N i ( u ) is a set of users that have similar preferences as user u (i.e., are part of the neighborhood set) who also rated item i . Similarity was computed based on a Cosine metric.

Matrix Factorization algorithms typically learn both of the users' and items' (latent) factors of the same size. The vectors are then computed from the user's rating. Each value of the factor vector, assigned to an item, represents how well the item describes a specific latent aspect.

User factor vectors are indicative of the specific preference of the user for each factor. A well-known example, also used in this study, is SVD ( Jannach et al., 2010 ). The task of the factorization is to break down the matrix of ratings R into two smaller matrices S and M .

where S denotes the | U | × F matrix, and M denoted | I | × F matrix. F reflects the number of latent factors we would like to utilize. Then, predictions for the ratings are made in the following way ( Funk, 2006 ).

where s uf is the degree in which the user u likes the latent factor f and the value m if denotes how strong the factor f is in the item i .

Co-clustering is a different type of algorithmic approach that exploits groups of similar users and similar items within calculated clusters ( George and Merugu, 2005 ; Reshef, 2015 ). The prediction of r ^ u i is computed by assigning the users and items to some clusters C u , C i and co-cluster C ui :

where Ĉ ui is the average rating of co-cluster C ui , Ĉ u is the average rating of u's cluster, and Ĉ i is the average ratings of i's cluster, and clusters are assigned using a straightforward optimization method.

Based on these three types of recommender approaches, we evaluated seven recommendation algorithms in order to identify the best algorithm in terms of the prediction accuracy. This included two commonly used baseline algorithms: Random and SlopeOne (cf. for more details: Jannach et al., 2010 , p. 41–43). Our algorithms included two types of neighborhood-based recommenders, two matrix factorization recommenders and a co-clustering approach (see above). Each of them evaluated a small dataset that contained 1,515 ratings for 551 universities (this is described in more detail in section 2.2).

For our neighborhood-based recommenders, we employed KNN Basic (i.e., “KNN1”) and KNN with Baseline (i.e., “KNN2”). The former was a simple version, while KNN with Baseline also considered the baseline rating. This was a factor that was estimated through a learning process. We set the number of neighbors for both KNN algorithms at 40. In addition, we adopted two matrix factorization (MF) recommenders: SVD and SVD++ ( Koren, 2008 ; Koren et al., 2009 ). The latter was an extension of SVD as it was capable of taking into account implicit ratings ( Elahi et al., 2019 ). The number of factors in both SVD algorithms was set to 20.

2.2. Dataset

To be able to recommend universities, we crawled the web to obtain data about 12,003 universities from across the world. The data included, among others, their names, country of location, and their official website URL. We used this data in the initial version of the system to collect a (small) preference rating dataset. We used a convenience sample by distributing the link of the study on the social media platforms of some of the authors. Participants were asked to provide a number of preference ratings to the universities (i.e., “Tell us what you think of these universities”) that might be familiar to them. In doing so, we obtained 1,515 ratings from 80 users, which were provided to 551 different universities. The ratings were provided in the range of [0–100], and were further utilized in our experiments to generate university recommendation lists.

In addition, we compiled a set of features that could be important to a student when choosing a university. This was based on findings from a survey administered among students from across the world in 2017 ( Quacquarelli Symonds, 2017 ). We used these features to better understand the preferences of the users and to obtain more information about their particular interests. The list of features was as follows:

• High-quality teaching

• Low or free tuition

• Research or internship opportunities

• High employment rate among graduates

• International diversity

• Cost of food and rent in the area

• Prestigious university brand

• Party environment or extracurricular activities

• Size of the university

• Access to sports facilities and sport clubs

• Family members have attended the university.

2.3. Offline Evaluation Setup

We evaluated the performance of seven different recommender algorithms to generate personalized ranking lists of universities in an offline experiment (RQ1). We performed the common k-fold cross validation methodology, where k was set to 5. This randomly split the rating dataset into 5 disjoint subsets. For each fold, 1 subset would be used as a test set and the 4 other subsets as training sets, eventually averaging the predictions across the five folds.

As mentioned earlier, we evaluated five different algorithms (i.e., KNN1, KNN2, SVD, SVD++, Co-Clustering) and two baselines. The main metric used was Root Mean Square Error (RMSE) , which measured the prediction accuracy in terms of the deviation of predicted ratings from the actual values in the test set.

2.4. Online Evaluation Setup

Based on the results of the offline experiment (cf. section 3.1), we selected the best performing algorithms for further examination in our online user study. We compared Singular Value Decomposition (SVD), Basic K-Nearest Neighbor (referred to as “KNN1”) and K-Nearest Neighbor with baselines (referred to as KNN2). To this end, a demo recommender system was developed to evaluate the quality of these algorithms.

2.4.1. Users

Participants were recruited through social media and at an Italian university, as a means of convenience sampling. A total of 56 participants accepted the invitation and started the experiment. Among them, only 52 selected the most important features and submitted their ratings. Eventually, 41 participants filled out the user evaluation questionnaire, while 37 participants completed the whole study. Participants provided on average 19 ratings to different universities. While all of our participants rated a minimum of 3 universities, this was somewhat skewed by one participant providing ratings to 150 different universities.

Among those that submitted the most important features, 38 identified as male, 11 as female, and 3 participants did not wish to disclose their gender. Most participants were either 18–24 (36.7%), 25–34 (38.78%), or 35–44 (22.5%) years old. Since submitting one's level of education was not compulsory, only 34 participants did so, among which the majority had obtained at least a bachelor's degree (79.4%).

2.4.2. Procedure

Participants were first informed about the overall goal of the experiment. After providing basic demographic information (e.g., education, age, and gender) 3 , we presented users a set of features that could possibly play an important role when making a decision on which university to choose for one's study (cf. Figure 2A ).

Figure 2 . Snapshots of the system, in different stages of user interaction. In (A) , users had to select at least three features they found important when selecting a university. In (B) ,they were asked rate at least three universities, while in (C) they were presented three personalized recommendation lists with universities. Panel (D) depicts the System Usability Questionnaire. Not depicted are the demographics and user evaluation screens.

In the main application, participants were asked to provide their preferences (through ratings) for a set of universities that might be familiar to them (see Figure 2B ). They needed to focus on universities (a minimum of three) that they had either attended for education or had sufficient experience with to make a judgment. This information was used to build a user profile, which was exploited by our recommender algorithms (see Figure 2C ). Participants received three lists of recommended universities, composed of five universities each, generated by the three different algorithms that had the highest accuracy in our offline evaluation. Participants were asked to carefully inspect each list and to compare them in terms of different evaluation criteria. To investigate how a user evaluated each recommendation list (RQ2), we presented users different questions in relation to the contents of the different recommendation lists and, thus, their underlying algorithms, based on different metrics (i.e., Accuracy, Diversity, Perceived Personalization, Satisfaction, and Novelty). Finally, to validate the usability of our recommender interface (see Figure 2D ), we asked users to indicate to what extent they agreed with propositions from the System Usability Survey (SUS) .

2.4.3. User Evaluation Metrics

The evaluation questionnaire consisted of fourteen questions. It was based on previous work in the movie recommender domain from Ekstrand et al. (2014) , and was adapted to the university domain. Per question, users needed to select one recommendation list that would contain either the best (e.g., having the most attractive suggestions) or the worst recommendations (e.g., having the least appealing suggestions), in relation to different evaluation metrics. This setup allowed for asymmetrical user preferences, in the sense that the least chosen “best option” may not be the worst.

Different subsets of questions addressed different evaluations metrics. To address a user's evaluation of our three algorithms (i.e., SVD, KNN1, KNN2; [RQ2]), we measured the perceived Accuracy of a recommendation list, the perceived Diversity within a list, whether a user perceived that a list was personalized toward her preferences (i.e., Understands Me ), the experienced level of Satisfaction , and the perceived level of Novelty . The list of questions was as follows, noting that some questions were formulated positively , while others were formulated negatively :

• Accuracy: Q1. Which list has more selections that you find appealing? [ positive ]

• Accuracy: Q2. Which list has more obviously bad suggestions for you? [ negative ]

• Diversity: Q3. Which list has more universities that are similar to each other? [ negative ]

• Diversity: Q4. Which list has a more varied selection of universities? [ positive ]

• Diversity: Q5. Which list has universities that match a wider variety of preferences? [ positive ]

• Understands Me: Q6. Which list better reflects your preferences in universities? [ positive ]

• Understands Me: Q7. Which list seems more personalized to your university ratings? [ positive ]

• Understands Me: Q8. Which list represents mainstream ratings instead of your own? [ negative ]

• Satisfaction: Q9. Which list would better help you find universities to consider? [ positive ]

• Satisfaction: Q10. Which list would you likely to recommend to your friends? [ positive ]

• Novelty: Q11. Which list has more universities you did not expect? [ positive ]

• Novelty: Q12. Which list has more universities that are familiar to you? [ negative ]

• Novelty: Q13. Which list has more pleasantly surprising universities? [ positive ]

• Novelty: Q14. Which list provides fewer new suggestions? [ negative ].

2.4.4. System Usability

Finally, we explored the usability of our recommender interface through the System Usability Scale (SUS) ( Brooke, 1996 ). The SUS was composed of a ten-item questionnaire with different propositions on the system's usability. The full list of items was as follows, where even-numbered propositions evaluated system aspects positively and odd-numbered propositions did so negatively:

• P1: I think that I would like to use this recommender system for finding the right university. [ positive ]

• P2: I found the recommender system unnecessarily complex. [ negative ]

• P3: I thought the recommender system was easy to use. [ positive ]

• P4: I think that I would need the support of a technical person to be able to use this recommender system. [ negative ]

• P5: I found the various functions in this recommender system were well integrated. [ positive ]

• P6: I thought there was too much inconsistency in this recommender system. [ negative ]

• P7: I would imagine that most people would learn to use this recommender system very quickly. [ positive ]

• P8: I found the recommender system very cumbersome to use. [ negative ]

• P9: I felt very confident using the recommender system to find my preferred university. [ positive ]

• P10: I needed to learn a lot of things before I can get going with the recommender system. [ negative ].

In line with Brooke (1996) , we used 5-point Likert scales to measure user responses. For positive items, these responses amounted to points, ranging from -1 (Strongly Disagree) to 3 (Strongly Agree). For negative items, points yielded from responses ranged from 5 (Strongly Disagree) to 1 (Strongly Agree). After adding up the points from all items, the total score was computed by multiplying the sum with 2.5, yielding a score between 0 and 100. In this context, the average SUS score computed in a benchmark of 500 studies was 68 ( Sauro, 2011 ).

We discuss the results obtained from our studies. First, we performed an offline evaluation to investigate which algorithm had the highest predictive power in university recommendation (RQ1). Second, we performed online evaluation, by letting users in a crowdsourcing study evaluate different aspects of three recommender algorithms (RQ2). Moreover, we inquire on desirable university features (RQ3).

3.1. Experiment A: Offline Study

We examined which of our collaborative filtering recommender approaches had the highest predictive value when generating personalized university rankings. We evaluated each algorithm offline, by performing and computing the Root Mean Squared Error (RMSE) for each algorithm ( Schedl et al., 2018 ).

The results of our offline evaluation, performed using five-fold cross-validation, are presented in Table 1 . As indicated by the lowest RMSE value, the best results were obtained for the SVD algorithm, with a mean value of 23.7. In terms of predictive accuracy, the second-best performing algorithm was SVD++ (Mean RMSE = 24.1), while the third-best algorithm in terms of RMSE was KNN2 algorithm, with a mean RMSE value of 24.9. Although the random baseline produced, as expected, much worse results (RMSE = 36.5) than the other algorithms, the relatively simple SlopeOne approach performed relatively well, for it had a lower mean RMSE (26.8) than both KNN1 (RMSE = 27.7) and Co-Clustering (RMSE = 27.5).

Table 1 . Results of the offline experiment, performed using five-fold cross-validation.

To proceed, we also considered how computationally demanding our algorithms were. In terms of runtime, the SVD recommender algorithm had an excellent performance, while SVD++ was the slowest among all algorithms. The latter might be due to the setup of the algorithm, which was originally proposed to work with implicit feedbacks (e.g., clicks) rather than explicit feedback (e.g., ratings). Hence, we did not consider SVD++ for our online evaluation. Instead, for the next phase, we opted for both KNN1 and KNN2. KNN1 had a short runtime combined with a somewhat worse accuracy, while KNN2 had both a comparatively good accuracy level and a decent runtime.

3.2. Experiment B: Online Study

3.2.1. evaluation of recommendation lists.

We compared how users evaluated different university recommendation lists, which were generated by different algorithms. Table 2 outlines per question the percentage of instances in a which recommendation list was chosen, designated by the algorithm generating it. Some questions contributed positively to a specific metric (e.g., Q1 to Accuracy), while those denoted in italics contributed negatively to that metric (e.g., Q2).

Table 2 . Results of paired t -tests on different evaluation metrics (based on Ekstrand et al., 2014 ), in which users were asked to choose a recommendation list in relation to specific metrics.

To examine which algorithm had the best performance per metric, we performed pairwise t -tests per questionnaire item. Reported on the right-hand side of Table 2 are the t -statistics, while the p -values are indicated by asterisks in superscript. The tests were performed by creating dummy variables for each algorithm, assigning the value 1 to an algorithm if its recommendation list was chosen by a user for a specific item.

Table 2 shows that the recommender algorithms are evaluated differently across different metrics. In terms of Accuracy (i.e., Q1, Q2), the best results were achieved by the SVD algorithm in terms of % chosen. Although there were no significant differences between SVD (Q1: 51%) and KNN1 (37%), they both significantly outperformed the KNN2 (12%) algorithm. The difference in perceived accuracy was largest for SVD, outforming KNN2 both on question 1: t (40) = 3.56, as well as on question 2: t (40) = −2.65, p = 0.012.

In terms of Diversity (i.e., Q3-Q5), both KNN1 and KNN2 seemed to be chosen more frequently than SVD, and thus might be favored. For Q3, SVD (42%) was perceived as generating more similar recommendations than KNN1 and KNN2 (both 29%). However, pairwise t -tests indicated that these differences were not significantly different (both: p > 0.05). For the positively formulated questions (i.e, Q4, Q5), both KNN1 and KNN2 were selected most often, once for each item. However, similar to Q3, pairwise t -tests did not reveal significant differences between SVD and the KNN algorithms.

Table 2 further suggests that the SVD algorithm was evaluated as generating the most favorable recommendation lists for our Understands Me and Satisfaction metrics. SVD was selected significantly more often (59%) than KNN1 (29%) for Q6: t (40) = 2.08, p < 0.05, as well as than KNN2 (12%): t (40) = 4.18, p < 0.001. SVD also performed significantly better than KNN2 on Q7 ( p < 0.001), as did KNN1 ( p < 0.01), both suggesting that users felt better understood by SVD than the KNN algorithms, particularly KNN2. Furthermore, we found that the recommendation lists generated by the SVD algorithm (51%) led to higher levels of satisfaction than those produced by KNN2 (12–15%); both for Q9: t (40) = 3.56, p = 0.001, as well as for Q10: t (40) = 3.19, p = 0.003. In contrast, SVD did not significantly outperform KNN1 on these questions (i.e., 34–37%), while a pairwise t -test indicated that KNN1 was selected more often KNN2 for Q9 ( p = 0.023). In contrast with the favorable findings for SVD, we also observed that it was selected most often for our negatively formulated item Q8 (44%), suggesting that it produced more mainstream items than the KNN2 algorithm (20%): t (40) = 2.03, p = 0.049. It was suggested that our users did not seem to interpret “mainstream items” necessarily as a negative aspect, while similar validation problems were also observed by Ekstrand et al. (2014) .

Finally, for our Novelty metric, KNN2 was the best performing algorithm for the majority of the items (Q11, Q12, Q14). As shown by pairwise t -tests (cf. Table 2 ), most users (i.e., 68%) indicated that the KNN2 algorithm recommended more unexpected universities (Q11) than SVD and KNN1: t (40) < −4.2, p < 0.001. Similar effects were observed for Q12 and Q14, as KNN2 generated significantly fewer recommendation lists that consisted of familiar universities than SVD and KNN1 [ t (40) > 3.30, p < 0.01], as well as was selected the least often for our item on “provides fewer new suggestions” (20%; significantly less than SVD). Finally, in contrast, Table 2 did not outline any significant differences between all three algorithms for Q13, as user choices were distributed almost equally between them. This could be attributed to the combination of “pleasantly” and “surprising” compared to use of the word “surprise” only in other questionnaire items.

3.2.2. University Features

We further examined which university features were found by users to be the most important when choosing a university. As indicated in the method section, the features were obtained from a marketing agency that administered a questionnaire among students worldwide ( Quacquarelli Symonds, 2017 ), while users could select multiple features to be important. On average, female users selected 4.9 features, while male users selected 3.9 features.

Table 3 presents an overview of the selected features, in descending order of how often they were selected, as well as divided across self-identified genders. Both for males and females, the most important features were High-quality teaching (80.8% across all users) and Low or free tuition (55.8%). Overall, Research or internship opportunities (51.9%) was found to the third most important feature, but was relatively speaking more important to male users than females, as High graduate employment rate was selected by 72.7% of female users. Hence, we observed varying priorities, also for lower-ranked features. For example, Table 3 describes that 36.4% of female users valued Access to sports facilities and sports clubs , while only 15.8% of male users did so. Interestingly, almost no participants considered the feature Family members have gone to that university in their university decision-making, deeming it to be least important feature.

Table 3 . Self-reported features that are most important to users when choosing a university to attend for study, distributed across females and males.

3.2.3. Usability

Finally, we validated the use of our recommender interface by administering a questionnaire on the System Usability Scale (SUS) ( Brooke, 1996 ). The results of our ten-item questionnaire with 5-point Likert scale are outlined in Table 4 , noting that the average SUS score computed in a benchmark of 500 studies was 68 ( Sauro, 2011 ).

Table 4 . Frequencies of user responses to questionnaire items (i.e., propositions, such as “P1”) from the System Usability Scale (SUS) ( Brooke, 1996 ).

Table 4 describes that most participants evaluated the usability of the demo university recommender system higher than the benchmark score. While the actual scores given by users ranged from 42.5 (lowest) to 100 (highest), the mean score was 74.5, which indicated that our university recommender system had a good usability ( Brooke, 1996 ). This can be observed in Table 4 , for most users responded with “Agree” or “Strongly Agree” to positive items, while “Disagree” was most common for negative items. This all indicated that most of the participants evaluated the system's usability positively.

Users could also fill out an open-ended text box at the end of the study to leave comments. We reviewed the comments of users who evaluated the system with a score below 51, which entailed 3 out of 37 users. Only a single user provided a comment in which she expressed concerns regarding the lack of clarity on which criteria she was supposed to rate the universities.

4. Conclusion

Overall, the presented results in the offline and online studies seemed promising. They reflected the potential effectiveness of the proposed university recommender system, opening up the possibility to generate personalized ranking lists of universities in the near future. It seemed that the SVD algorithm was most appropriate to do so, based on accuracy, perceived personalization, and satisfaction. These results illustrated the potential of such a system and its importance in supporting individuals who are searching for the best universities for their future studies, which is a high-stake decision and, therefore, unlike most other recommender systems.

5. Discussion

This paper has addressed a challenging recommendation problem in the domain of education, specifically about where one should follow their university education. The existence of a large number of educational institutions has exacerbated the task of choosing where to study. We have described the design and development of a demo system that can provide a personalized ranking list of universities. To that end, we have compared different recommender approaches and algorithms, both in an offline and online evaluation context.

In particular, this work has attempted to address the following research questions:

In relation to RQ1, we have found that the SVD and “k-Nearest Neighbor with baselines” approaches performed best in terms of predictive accuracy. The main metric we have used is RMSE, which indicates that we have focused on predicting the ratings in a dataset using training and test split. Although the list of approaches considered is by no means exhaustive (i.e., it only comprises collaborative filtering recommender approaches), we have shown which algorithms among a set of common approaches performed best. In addition, we have also considered runtime as a pragmatic factor on deciding which algorithms to consider for our online evaluation. This has led us to select SVD, KNN basic (i.e., “KNN1”), and KNN with baselines (i.e., “KNN2”) for the online evaluation.

With regard to RQ2 and RQ3, our findings illustrate that recommending a university to a prospective user (e.g., a student) out of the existing large number of universities is a complex problem. This is arguably due to the various dimensions and features that are involved when a user makes this choice, as well as the complexity and multi-facetedness of the recommendation items in question. Furthermore, we have noticed that preferences for a particular recommendation algorithm may vary for different users considering different features when choosing the university to study, as well as that the performance of different recommender algorithm depends on the evaluation metric in question.

Overall, we have found our SVD approach to outperform the KNN approaches on accuracy and “fit-related” metrics. Although not all differences have been found to be statistically significant, SVD outperformed the KNN approaches on perceived accuracy, perceived personalization, and satisfaction. In contrast, it is suggested that KNN2 (i.e., the KNN approach with baselines) outperforms SVD in terms of more exploratory aspects, such as diversity and novelty, although the differences for diversity have not been found to be statistically significant. Based on the findings in, among others, Table 2 , it is suggested that SVD may be more suitable for users who already have a better understanding of what they are looking for and who wish to reduce the set size of potential universities. In contrast, the KNN approaches seem to be more suitable to users who are still exploring the space of possible universities.

A few of the results in the evaluation questionnaire are found to be somewhat inconsistent within a single metric. For one, it has not become clear whether KNN1 or KNN2 performs better on diversity, although the differences are rather small in the first place. Another peculiar outcome is that the perceived personalization (i.e., “Understands Me”) for SVD is found to be highest for the two positively formulated items, but also highest for the negatively formulated item. However, it seems that the wording of Q8 in Table 2 , specifically mentioning “mainstream items,” made it also relate to other aspects, arguably a lack of diversity. Such an explanation, that the item does not correctly measure the “Understands me” metric, is consistent with the findings from Ekstrand et al. (2014) , where the item is also found to be less related to perceived personalization than the other items used. We argue that a similar problem is also observed for Q13, which seems to measure serendipity (“pleasant” and “surprising,” Ge et al., 2010 ) instead of novelty; this item also seemed to not fully measure novelty in a previous study ( Ekstrand et al., 2014 ).

We have further obtained evidence that some features seem to play a more important role in university selection than others (RQ3). In the overall top-4, we have found study and career-related features, such as high-quality teaching, research of internship opportunities , and high graduate employment rate , but also a feature that is more related to feasibility ( low or free tuition ). Features that are more related to contextual factors (i.e., cost in the area, facilities, party environment) are found to be less important, nor have we found that prestige and familiarity are particularly important. With regard to the latter, almost none of our participants have indicated that it is important that family members have attended a particular university, even though this has been part of the widespread questionnaire among university students ( Quacquarelli Symonds, 2017 ).

5.1. Limitations

We would like to point out a few limitations to our online study. First, the use of a convenience sample might have reduced the quality of the collected data. We have sent out a web link to our potential participants, through the social network and personal channels of part of the research team. As a result, this has reduced the control we could exert over the type of participants that enrolled in the study and might have skewed the distribution of demographics in the sample toward men and people who have attended a university education. Moreover, this might have affected the extent to which users were actually interested in selecting a university and whether anything was “at stake” for them. However, we argue that the large proportion of participants that has attended a university education in our Study 2 sample, increases the likelihood that the task has been relevant to them. It could be argued that this makes their judgments more valid than participants who do not have such experience or “vested interests.” Nonetheless, we wholeheartedly recommend a user study to be conducted among a sample of participants that still needs to select a university education, such as high school students (16–18 year olds), who have a clear incentive to take this task seriously. We would envision a longitudinal study design that assesses whether students that followed the recommendations of a personalized ranking system have a lower drop-out rate than students who have obtained their information in different ways.

A more general limitation of online evaluation is that one cannot account for possible failures of a participant's internet connection. However, this does not seem to have played a large role in the collected data, as the dropout during the study was acceptable.

A potential lack of recognition among the recommended universities has made it possibly harder for users to assess the presented items. We have attempted to mitigate this by providing “More Info” buttons alongside each university, which would take the user to the website of that institution. However, we have not monitored whether these have been used extensively. Nonetheless, we argue that a lack of recognition among users will be a generic challenge in developing any type of recommender system in the university domain. We feel that this can be mitigated by, for instance, effective interface design that also focuses on helping users to explore new universities and obtain new information. With respect to that, we have found that the overall usability of our system is good, but this questionnaire has not inquired on specific interface aspects and user goals (e.g., exploration vs. exploitation).

5.2. Future Work

Since our work concerns a new application domain, that of personalized ranking and recommendation of universities, there is still more research that needs to conducted. First and foremost, it is still unclear to what extent receiving a personalized university ranking and auxiliary advice has benefits over offering a non-personalized ranking that can be found on various websites, such as The Times Higher Education ranking. Although our novel system serves as a proof-of-concept, in the sense that it can both elicit user preferences for universities through ratings and construct a personalized university ranking list, we have only compared different algorithmic baselines. Nonetheless, we have observed that recommender approaches outperform a random baseline, but it is likely that a popularity baseline will perform somewhat better.

As future work, we plan to conduct more experiments with larger datasets in terms of the number of ratings provided to universities. We will also conduct user studies to consolidate algorithms that can learn from other sources of information, such as the social media profiles of users. We will also redesign the user interface and improve the interaction model by taking advantages of novel design elements ( Cremonesi et al., 2017 ).

Another future direction can be the development of recommender approaches that consider future goals, which has been explored in some recommender domains ( Ekstrand and Willemsen, 2016 ; Starke et al., 2021a ). The algorithms used in the current studies are not much different from those used in traditional domains and are therefore likely to optimize for short-term engagement ( Jannach et al., 2010 ). This contrasts with the conception that one's university education benefits one in the longer-term and has a big impact one's future career and life ( Rivera et al., 2018 ). This is, however, a broader problem in recommender system research ( Ekstrand and Willemsen, 2016 ), also in other high-stake domains, such as healthy eating ( Elsweiler, 2019 ).

Finally, we are planning to incorporate personality information, provided by the users, in the prediction model. This may enable the algorithms to generate ranking lists that suit the personality characteristics of the users. In the context of educational recommender systems, the use of personality traits is—to the best of our knowledge—a new approach. The types of approaches used in previous studies are generally memory-based or use a combination of collaborative filtering and content-based recommendation ( Bodily and Verbert, 2017 ; Rivera et al., 2018 ). Nonetheless, one's personality has been considered as an important trait in studies in the general education domain and is, therefore, possibly a feasible predictor in determining future preferences. For example, although the existence of learning styles is currently being questioned in many studies ( Riener and Willingham, 2010 ), it has been used in the past to tailor educational content to a person's learning style ( Felder and Silverman, 1988 ; Rovai, 2003 ). The same principle, exploiting the relation between personality traits and learning styles, has also been exploited in a learning analytics and Learning Management Systems ( Halawa et al., 2015 ).

Based on the feature importance that we have collected as part of RQ3 and existing metadata of universities, it might also be possible to pursue knowledge-based approaches, for these are more uncommon in past recommender systems in the education domain ( Bodily and Verbert, 2017 ; Rivera et al., 2018 ). We argue that this and the aforementioned personality-based approaches, could be appropriate for the university domain and can overcome possible cold-start problems. In our view, one-size-fits-all rankings can easily be replaced by recommender systems, for they more effective and are more efficient in the long-run.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

ME was responsible for generating the initial idea, the write-up overall, particularly to the method and the discussion sections, and also helped to collect data. AS contributed to the Introduction, Related Work, and Results sections and performed the statistical evaluation of the online study. NE and CT supervised the process and helped to finalize the manuscript, while NE also generated the idea. AL helped to collect the data and performed the offline evaluation. All authors contributed to the article and approved the submitted version.

This work was supported by industry partners and the Research Council of Norway with funding to MediaFutures: Research Centre for Responsible Media Technology and Innovation, through the Centres for Research-based Innovation scheme, project number 309339.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. ^ Please check Webmetrics which already indexed tens of thousands of universities: http://www.webometrics.info

2. ^ https://www.timeshighereducation.com/world-university-rankings

3. ^ We also inquired on a user's personality characteristics using scales from the Big Five Inventory, but we did not consider these for analysis in the current study.

Bodily, R., and Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Trans. Learn. Technol. 10, 405–418. doi: 10.1109/TLT.2017.2740172

PubMed Abstract | CrossRef Full Text | Google Scholar

Bokde, D. K., Girase, S., and Mukhopadhyay, D. (2015). “An approach to a university recommendation by multi-criteria collaborative filtering and dimensionality reduction techniques,” in 2015 IEEE International Symposium on Nanoelectronic and Information Systems (Indore: IEEE), 231–236.

Google Scholar

Brooke, J.. (1996). SUS-a quick and dirty usability scale. Usabil. Eval. Indus. 189, 4–7.

CollegeBoard (2019). College Search . Available online at: https://bigfuture.collegeboard.org/college-search (accessed February 17, 2019).

CollegeRaptor (2019). College Search . Available online at: https://www.collegeraptor.com/college-search/ (accessed February 17, 2019).

Conijn, R., Snijders, C., Kleingeld, A., and Matzat, U. (2016). Predicting student performance from lms data: a comparison of 17 blended courses using moodle LMS. IEEE Trans. Learn. Technol. 10, 17–29. doi: 10.1109/TLT.2016.2616312

Cremonesi, P., Elahi, M., and Garzotto, F. (2017). User interface patterns in recommendation-empowered content intensive multimedia applications. Multimedia Tools Appl. 76, 5275–5309. doi: 10.1007/s11042-016-3946-5

CrossRef Full Text | Google Scholar

Dascalu, M.-I., Bodea, C.-N., Mihailescu, M. N., Tanase, E. A., and Ordoñez de Pablos, P. (2016). Educational recommender systems and their application in lifelong learning. Behav. Inform. Technol. 35, 290–297. doi: 10.1080/0144929X.2015.1128977

De Medio, C., Limongelli, C., Sciarrone, F., and Temperini, M. (2020). Moodlerec: a recommendation system for creating courses using the moodle e-learning platform. Comput. Hum. Behav. 104:106168. doi: 10.1016/j.chb.2019.106168

Dwivedi, S., and Roshni, V. K. (2017). “Recommender system for big data in education,” in 2017 5th National Conference on E-Learning & E-Learning Technologies (ELELTECH) (Hyderabad: IEEE), 1–4.

Ekstrand, M. D., Harper, F. M., Willemsen, M. C., and Konstan, J. A. (2014). “User perception of differences in recommender algorithms,” in Proceedings of the 8th ACM Conference on Recommender Systems (Foster City, CA: ACM), 161–168.

Ekstrand, M. D., and Willemsen, M. C. (2016). “Behaviorism is not enough: better recommendations through listening to users,” in Proceedings of the 10th ACM Conference on Recommender Systems (Boston, MA), 221–224.

Elahi, M.. (2014). Empirical evaluation of active learning strategies in collaborative filtering (Ph. D. thesis) . Free University of Bozen-Bolzano, Bolzano, Italy.

Elahi, M., Braunhofer, M., Gurbanov, T., and Ricci, F. (2019). “User preference elicitation, rating sparsity and cold start,” in Collaborative Recommendations: Algorithms, Practical Challenges and Applications (World Scientific).

Elahi, M., El Ioini, N., Alexander Lambrix, A., and Ge, M. (2020). “Exploring personalized university ranking and recommendation,” in Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization , eds S. Berkovsky, I. Cantador, and D. Tikk (Singapore), 253–294.

Elsweiler, D., Ludwig, B., Said, A., Schäfer, H., Torkamaan, H., and Trattner, C. (2019). HealthRecSys'19: Proceedings of the 4th International Workshop on Health Recommender Systems, co-located with 13th ACM Conference on Recommender Systems (HealthRecSys'19) (Copenhagen).

Engelhard, G. Jr.. (1992). The measurement of writing ability with a many-faceted rasch model. Appl. Measure. Educ. 5, 171–191.

Felder, R. M., and Silverman, L. K. (1988). Learning and teaching styles in engineering education. Eng. Educ. 78, 674–681.

Funk, S.. (2006). Netflix Update: Try This at Home . Available online at: http://sifter.org/~simon/journal/20061211.html

Ge, M., Delgado-Battenfeld, C., and Jannach, D. (2010). “Beyond accuracy: evaluating recommender systems by coverage and serendipity,” in Proceedings of the Fourth ACM Conference on Recommender Systems (Barcelona), 257–260.

George, T., and Merugu, S. (2005). “A scalable collaborative filtering framework based on co-clustering,” in Fifth IEEE International Conference on Data Mining (ICDM'05) (Houston, TX: IEEE), 4.

Gomez-Uribe, C. A., and Hunt, N. (2015). The netflix recommender system: algorithms, business value, and innovation. ACM Trans. Manage. Inform. Syst. 6, 1–19. doi: 10.1145/2843948

Halawa, M. S., Shehab, M. E., and Hamed, E. M. R. (2015). “Predicting student personality based on a data-driven model from student behavior on LMS and social networks,” in 2015 Fifth International Conference on Digital Information Processing and Communications (ICDIPC) (Sierre: IEEE), 294–299.

Hasan, M., Ahmed, S., Abdullah, D. M., and Rahman, M. S. (2016). “Graduate school recommender system: Assisting admission seekers to apply for graduate studies in appropriate graduate schools,” in 2016 5th International Conference on Informatics, Electronics and Vision (ICIEV) (Dhaka: IEEE), 502–507.

Hemsley-Brown, J.. (2012). “The best education in the world”: reality, repetition or cliché? international students' reasons for choosing an English University. Stud. High. Educ. 37, 1005–1022. doi: 10.1080/03075079.2011.562286

Hsu, M.-H.. (2008). A personalized English learning recommender system for ESL students. Expert Syst. Appl. 34, 683–688. doi: 10.1016/j.eswa.2006.10.004

Jannach, D., Zanker, M., Felfernig, A., and Friedrich, G. (2010). Recommender Systems: An Introduction . Cambridge, UK: Cambridge University Press.

Kanoje, S., Mukhopadhyay, D., and Girase, S. (2016). User profiling for university recommender system using automatic information retrieval. Proc. Comput. Sci. 78, 5–12. doi: 10.1016/j.procs.2016.02.002

Khoja, Z., and Shetty, S. (2017). “Hybrid recommender system for college courses,” in 2017 International Conference on Computational Science and Computational Intelligence (CSCI) (: Las Vegas, NV: IEEE), 1167–1171.

Koren, Y.. (2008). “Factorization meets the neighborhood: a multifaceted collaborative filtering model,” in KDD '08: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY: ACM), 426–434.

Koren, Y., Bell, R., and Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer 42, 30–37. doi: 10.1109/MC.2009.263

Meryem, G., Douzi, K., and Chantit, S. (2016). “Toward an e-orientation platform: Using hybrid recommendation systems,” in 2016 11th International Conference on Intelligent Systems: Theories and Applications (SITA) (Mohammedia: IEEE), 1–6.

Obeid, C., Lahoud, I., El Khoury, H., and Champin, P.-A. (2018). “Ontology-based recommender system in higher education,” in Companion Proceedings of the The Web Conference 2018 (Lyon), 1031–1034.

OECD (ed.), . (2013). “How many students study abroad and where do they go?” in Education at a Glance (Paris: OECD Publishing), 32–35.

O'Mahony, M. P., and Smyth, B. (2007). “A recommender system for on-line course enrolment: an initial study,” in Proceedings of the 2007 ACM Conference on Recommender Systems (Minneapolis, MN), 133–136.

Quacquarelli Symonds (2017). International Student Survey . Available online at: https://www.internationalstudentsurvey.com/ (accessed February 17, 2019).

Ragab, A. H. M., Mashat, A. F. S., and Khedra, A. M. (2012). “HRSPCA: Hybrid recommender system for predicting college admission,” in 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA) (Kochi: IEEE), 107–113.

Ragab, A. H. M., Mashat, A. F. S., and Khedra, A. M. (2014). Design and implementation of a hybrid recommender system for predicting college admission. Int. J. Comput. Inform. Syst. Indus. Manage. Appl. 6, 35–44.

Rathore, R. K., Jayanthi, J., and Kaur, G. (2017). Student prediction system using data mining algorithm – survey. Int. J. Res. Appl. Sci. Eng. Technol. 5.

Reshef, R.. (2015). How do you build a “people who bought this also bought that”-style recommendation engine . Available online at: https://datasciencemadesimpler.wordpress.com/2015/12/16/understanding-collaborative-filtering-approach-to-recommendations/ (accessed May 01, 2020).

Riener, C., and Willingham, D. (2010). The myth of learning styles. Change Mag. Higher Learn. 42, 32–35. doi: 10.1080/00091383.2010.503139

Rivera, A. C., Tapia-Leon, M., and Lujan-Mora, S. (2018). “Recommendation systems in education: a systematic mapping study,” in International Conference on Information Technology & Systems (Peninsula de Santa Elena: Springer), 937–947.

Rovai, A. P.. (2003). The relationships of communicator style, personality-based learning style, and classroom community among online graduate students. Internet Higher Educ. 6, 347–363. doi: 10.1016/j.iheduc.2003.07.004

Samin, H., and Azim, T. (2019). Knowledge based recommender system for academia using machine learning: a case study on higher education landscape of Pakistan. IEEE Access 7, 67081–67093. doi: 10.1109/ACCESS.2019.2912012

Sauro, J.. (2011). Measuring usability with the system usability scale (SUS) . Available online at: https://measuringu.com/sus (accessed August 5, 2021).

PubMed Abstract | Google Scholar

Schedl, M., Zamani, H., Chen, C.-W., Deldjoo, Y., and Elahi, M. (2018). Current challenges and visions in music recommender systems research. Int. J. Multimedia Inform. Retrieval 7, 95–116. doi: 10.1007/s13735-018-0154-2

Simões, C., and Soares, A. M. (2010). Applying to higher education: information sources and choice factors. Stud. Higher Educ. 35, 371–389. doi: 10.1080/03075070903096490

Starke, A., Asotic, E., and Trattner, C. (2021a). “Serving each user”: supporting different eating goals through a multi-list recommender interface,” in Fifteenth ACM Conference on Recommender Systems (Amsterdam), 124–132.

Starke, A. D., and Trattner, C. (2021). “Promoting healthy food choices online: a case for multi-list recommender systems,” in HEALTHI'21: Joint Proceedings of ACM IUI 2021 Workshops (College Station, TX: ACM).

Starke, A. D., Willemsen, M. C., and Trattner, C. (2021b). Nudging healthy choices in food search through visual attractiveness. Front. Artif. Intell. 4:621743. doi: 10.3389/frai.2021.621743

Tejeda-Lorente, Á., Bernabé-Moreno, J., Porcel, C., Galindo-Moreno, P., and Herrera-Viedma, E. (2015). A dynamic recommender system as reinforcement for personalized education by a fuzzly linguistic web system. Proc. Comput. Sci. 55, 1143–1150. doi: 10.1016/j.procs.2015.07.084

Thai-Nghe, N., Drumond, L., Krohn-Grimberghe, A., and Schmidt-Thieme, L. (2010). Recommender system for predicting student performance. Proc. Comput. Sci. 1, 2811–2819. doi: 10.1016/j.procs.2010.08.006

Wakil, K., Akram, B., Kamal, N., and Safi, A. (2014). Web recommender system for private universities' admission in Iraq: Uhd case study. Int. J. e-Educ. e-Bus. e-Manage. e-Learn. 4:329. doi: 10.7763/IJEEEE.2014.V4.348

White, P. M., and Lee, D. M. (2020). Geographic inequalities and access to higher education: is the proximity to higher education institution associated with the probability of attendance in England? Res. Higher Educ. 61, 825–848. doi: 10.1007/s11162-019-09563-x

Yuan, X., Lee, J.-H., Kim, S.-J., and Kim, Y.-H. (2013). Toward a user-oriented recommendation system for real estate websites. Inform. Syst. 38, 231–243. doi: 10.1016/j.is.2012.08.004

Keywords: recommender systems, education, offline evaluation, user study, usability, university

Citation: Elahi M, Starke A, El Ioini N, Lambrix AA and Trattner C (2022) Developing and Evaluating a University Recommender System. Front. Artif. Intell. 4:796268. doi: 10.3389/frai.2021.796268

Received: 16 October 2021; Accepted: 27 December 2021; Published: 02 February 2022.

Reviewed by:

Copyright © 2022 Elahi, Starke, El Ioini, Lambrix and Trattner. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Mehdi Elahi, mehdi.elahi@uib.no

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

SRRS: Design and Development of a Scholarly Reciprocal Recommendation System

• Published: 21 September 2024

Cite this article

• Shilpa Verma   ORCID: orcid.org/0000-0002-6273-3774 1 ,
• Sandeep Harit 1 &
• Kundan Munjal 2  

20 Accesses

Explore all metrics

The aim of this work is to propose a hybrid reciprocal recommendation algorithm for cold-start authors in a network based on text information and network-based features. The proposed algorithm is a novel collaborative filtering algorithm that combines text information with network features for more accurate and personalized recommendations. The feature importance values are used to understand the impact of each feature on the prediction and to identify the most important features for a given task. In the proposed algorithm, a community detection algorithm is used in addition to the baseline method, which uses a first-order neighborhood approach. Furthermore, varying T on edge weights in the co-author graph with optimal T is used to obtain hybrid recommendations in the same community. The results demonstrate that the proposed method is effective in predicting collaborators for cold-start authors in the network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

A Collaborators Recommendation Method Based on Multi-feature Fusion

Co-author Recommender System

Scientific Matchmaker: Collaborator Recommender System

Explore related subjects.

• Artificial Intelligence

Afsar, M. M., Crump, T., & Far, B. H. (2023). Reinforcement learning based recommender systems: A survey. ACM Computing Surveys , 55 (7), 145:1–145:38.

Agarwal, K., Uniyal, P., Virendrasingh, S., Krishna, S., & Dutt, V. (2021). Spam mail classification using ensemble and non-ensemble machine learning algorithms. In Machine learning for predictive analysis: Proceedings of ICTIS 2020 (pp. 179–189). Springer.

Alhoori, H., & Furuta, R. (2017). Recommendation of scholarly venues based on dynamic user interests. Journal of Informetrics, 11 (2), 553–563.

Article   Google Scholar  

Amami, M., Pasi, G., Stella, F., & Faiz, R. (2016). An LDA-based approach to scientific paper recommendation. In E. Métais, F. Meziane, M. Saraee, V. Sugumaran, & S. Vadera (Eds.), Natural language processing and information systems (pp. 200–210). Springer.

Bai, X., Wang, M., Lee, I., Yang, Z., Kong, X., & Xia, F. (2019). Scientific paper recommendation: A survey. IEEE Access, 7 , 9324–9339.

Chandrasekaran, K., Gauch, S., Lakkaraju, P., & Luong, H. P. (2008). Concept-based document recommendations for citeseer authors. In W. Nejdl, J. Kay, P. Pu, & E. Herder (Eds.), Proceedings of 5th international conference on Adaptive hypermedia and adaptive web-based systems, AH 2008 , Hannover, Germany, 29 July 29–1 August 2008. Lecture notes in computer science (Vol. 5149, pp. 83–92). Springer.

Choudhury, A., Kaushik, S., & Dutt, V. (2018). Social-network analysis in healthcare: Analysing the effect of weighted influence in physician networks. Network Modeling Analysis in Health Informatics and Bioinformatics 7 (1), 17.

de Sousa, G. A., Diniz, M. A., Brandão, M. A., & Moro, M. M. (2015). CNARE: Co-authorship networks analysis and recommendations. In H. Werthner, M. Zanker, J. Golbeck, & G. Semeraro (Eds.), Proceedings of the 9th ACM conference on recommender systems, RecSys 2015 , Vienna, Austria, 16–20 September 2015 (pp. 329–330). ACM.

Dewang, R. K., & Singh, A. K. (2018). State-of-art approaches for review spammer detection: A survey. Journal of Intelligent Information Systems, 50 (2), 231–264.

Falconnet, A., Coursaris, C. K., Beringer, J., Van Osch, W., Sénécal, S., & Léger, P.-M. (2023). Improving user experience with recommender systems by informing the design of recommendation messages. Applied Sciences, 13 (4), 2706.

Gregory, S. (2009). Finding overlapping communities in networks by label propagation. New Journal of Physics, 12 , 103018.

Hristakeva, M., Kershaw, D., Rossetti, M., Knoth, P., Pettit, B., Vargas, S., & Jack, K. (2017). Building recommender systems for scholarly information. In Proceedings of the 1st workshop on scholarly web mining, SWM@WSDM 2017 , Cambridge, UK, 10 February 2017 (pp. 25–32). ACM.

Huang, L., Chen, X., Zhang, Y., Zhu, Y., Li, S., & Ni, X. (2021). Dynamic network analytics for recommending scientific collaborators. Scientometrics, 126 (11), 8789–8814.

Jena, K. K., Bhoi, S. K., Malik, T. K., Sahoo, K. S., Jhanjhi, N. Z., Bhatia, S., & Amsaad, F. (2022). E-learning course recommender system using collaborative filtering models. Electronics, 12 (1), 157.

Jie, L., Dianshuang, W., Mao, M., Wang, W., & Zhang, G. (2015). Recommender system application developments: A survey. Decision Support Systems, 74 , 12–32.

Jin, T., Qiong, W., Xuan, O., & Yu, J. (2021). Community detection and co-author recommendation in co-author networks. International Journal of Machine Learning and Cybernetics 12 (2), 597–609.

Khan, S., Liu, X., Shakil, K. A., & Alam, M. (2017). A survey on scholarly data: From big data perspective. Information Processing and Management, 53 (4), 923–944.

Kleinerman, A., Rosenfeld, A., Ricci, F., & Kraus, S. (2021). Supporting users in finding successful matches in reciprocal recommender systems. User Modeling and User-Adapted Interaction 31 (3), 541–589.

Kong, X., Jiang, H., Wang, W., Bekele, T. M., Xu, Z., & Wang, M. (2017). Exploring dynamic research interest and academic influence for scientific collaborator recommendation. Scientometrics, 113 (1), 369–385.

Kong, X., Mao, M., Wang, W., Liu, J., & Bo, X. (2021). VOPRec: Vector representation learning of papers with text information and structural identity for recommendation. IEEE Transactions on Emerging Topics in Computing 9 (1), 226–237.

Li, J., Xia, F., Wang, W., Chen, Z., Asabere, N. Y., & Jiang, H. (2014). ACRec: a co-authorship based random walk model for academic collaboration recommendation. In C.-W. Chung, A. Z. Broder, K. Shim, & T. Suel (Eds.), 23rd International world wide web conference, WWW ’14 , Seoul, Republic of Korea, 7–11 April 2014, Companion Volume, (pp. 1209–1214). ACM.

Li, L., & Li, T. (2012). MEET: a generalized framework for reciprocal recommender systems. In X. Chen, G. Lebanon, H. Wang, & M. J. Zaki (Eds.), 21st ACM international conference on information and knowledge management , CIKM’12, Maui, HI, USA, 29 October–02 November 2012 (pp. 35–44). ACM.

Liang, W., Zhou, X., Huang, S., Chunhua, H., Xuesong, X., & Jin, Q. (2018). Modeling of cross-disciplinary collaboration for potential field discovery and recommendation based on scholarly big data. Future Generation Computer Systems, 87 , 591–600.

Liu, Z., Xie, X., & Chen, L. (2018). Context-aware academic collaborator recommendation. In Y. Guo & F. Farooq (Eds.), Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2018 , London, UK, 19–23 August 2018 (pp. 1870–1879). ACM.

Makarov, I., & Gerasimova, O. (2019). Predicting collaborations in co-authorship network. In 14th International workshop on semantic and social media adaptation and personalization , SMAP 2019, Larnaca, Cyprus, 9–10 June 2019 (pp. 1–6). IEEE.

Naik, N. N., Chandrasekaran, K., Venkatesan, M., & Prabhavathy, P. (2022). Deep learning-based prediction, classification, clustering models for time series analysis: A systematic review. In Advances in information communication technology and computing: proceedings of AICTC 2021 (pp. 377–390).

Neve, J., & Palomares, I. (2019). Aggregation strategies in user-to-user reciprocal recommender systems. In 2019 IEEE International conference on systems, man and cybernetics , SMC 2019, Bari, Italy, 6–9 October 2019 (pp. 4031–4036). IEEE.

Neve, J., & Palomares, I. (2020). Hybrid reciprocal recommender systems: Integrating item-to-user principles in reciprocal recommendation. In A. El Fallah Seghrouchni, G. Sukthankar, T.-Y. Liu, & M. van Steen (Eds.), Companion of the 2020 web conference 2020 , Taipei, Taiwan, 20–24 April 2020 (pp. 848–854). ACM/IW3C2.

Parimi, R., & Caragea, D. (2014). Community detection on large graph datasets for recommender systems. In Z.-H. Zhou, W. Wang, R. Kumar, H. Toivonen, J. Pei, J. Z. Huang, & X. Wu (Eds.), 2014 IEEE international conference on data mining workshops, ICDM workshops 2014 , Shenzhen, China, 14 December 2014 (pp. 589–596). IEEE Computer Society.

Potts, B. A., & Khosravi, H. (2018). Competency, compatibility and preferences in reciprocal peer recommendation.

Potts, B. A., Khosravi, H., Reidsema, C., Bakharia, A., Belonogoff, M., & Fleming, M. (2018). Reciprocal peer recommendation for learning purposes. In A. Pardo, K. Bartimote-Aufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Proceedings of the 8th international conference on learning analytics and knowledge, LAK 2018 , Sydney, NSW, Australia, 07–09 March 2018 (pp. 226–235). ACM.

Prabhakar, S., Spanakis, G., & Zaïane, O. R. (2017). Reciprocal recommender system for learners in massive open online courses (MOOCS). In H. Xie, E. Popescu, G. P. Hancke, & B. Fernández-Manjón (Eds.), Proceedings of 16th international conference on advances in web-based learning—ICWL 2017 , Cape Town, South Africa, September 20-22, 2017. Lecture Notes in Computer Science (Vol. 10473, pp. 157–167). Springer.

Pradhan, T., & Pal, S. (2020). A multi-level fusion based decision support system for academic collaborator recommendation. Knowledge-Based Systems, 197 , 105784.

Pradhan, T., Sahoo, S., Singh, U., & Pal, S. (2021). A proactive decision support system for reviewer recommendation in academia. Expert Systems with Applications, 169 , 114331.

Que, X., Checconi, F., Petrini, F., & Gunnels, J. A. (2015). Scalable community detection with the Louvain algorithm. In 2015 IEEE International parallel and distributed processing symposium, IPDPS 2015 , Hyderabad, India, 25–29 May 2015 (pp. 28–37). IEEE Computer Society.

Rao, A. K. G., Janikow, C., Bhatia, S., & Climer, S. (2018). Efficient reduced-bias genetic algorithm (ERBGA) for generic community detection objectives.***

Rodrigues, M. W., Brandão, W. C., & Zárate, L. E. (2018). Recommending scientific collaboration from researchgate. In 7th Brazilian conference on intelligent systems, BRACIS 2018 , São Paulo, Brazil, 22–25 October 2018 (pp. 336–341). IEEE Computer Society.

Son, J., & Kim, S. B. (2018). Academic paper recommender system using multilevel simultaneous citation networks. Decision Support Systems, 105 , 24–33.

Tang, J., Zhang, J., Yao, L., Li, J., Zhang, L., & Su, Z. (2008). Arnetminer: extraction and mining of academic social networks. In Y. Li, B. Liu, & S. Sarawagi (Eds.), Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining , Las Vegas, Nevada, USA, 24–27 August 2008 (pp. 990–998). ACM.

Verma, P., Sharma, K., & Walia, G. S. (2021). Depression detection among social media users using machine learning. In D. Gupta, A. Khanna, S. Bhattacharyya, A. Ella Hassanien, S. Anand, & A. Jaiswal (Eds.), International conference on innovative computing and communications (pp. 865–874). Springer.

Verma, S., Bhatia, R., Harit, S., & Batish, S. (2023). Scholarly knowledge graphs through structuring scholarly communication: a review. Compl. Int. Syst., 9 (1), 1059–1095.

Vukotic, Aleksa, Watt, Nicki, Abedrabbo, Tareq, Fox, Dominic, & Partner, Jonas. (2015). Neo4j in action , volume 22. Manning Shelter Island.

Xia, F., Chen, Z., Wang, W., Li, J., & Yang, L. T. (2014). MVCWalker: Random walk-based most valuable collaborators recommendation exploiting academic factors. IEEE Transactions on Emerging Topics in Computing, 2 (3), 364–375.

Xia, F., Wang, W., Bekele, T. M., & Liu, H. (2017). Big scholarly data: A survey. IEEE Transactions on Big Data, 3 (1), 18–35.

Yang, C., Sun, J., Ma, J., Zhang, S., Wang, G., & Hua, Z. (2015). Scientific collaborator recommendation in heterogeneous bibliographic networks. In T. X. Bui & R. H. Sprague, Jr. (Eds.), 48th Hawaii international conference on system sciences, HICSS 2015 , Kauai, Hawaii, USA, 5–8 January 2015 (pp. 552–561). IEEE Computer Society.

Zhou, X., Ding, L., Li, Z., & Wan, R. (2017). Collaborator recommendation in heterogeneous bibliographic networks using random walks. Inf. Retr. J., 20 (4), 317–337.

Download references

Author information

Authors and affiliations.

Punjab Engineering College, Chandigarh, India

Shilpa Verma & Sandeep Harit

Punjabi University, Patiala, India

Kundan Munjal

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Shilpa Verma .

Ethics declarations

Conflict of interest.

On behalf of all authors, the corresponding author states that there is no Conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Verma, S., Harit, S. & Munjal, K. SRRS: Design and Development of a Scholarly Reciprocal Recommendation System. Scientometrics (2024). https://doi.org/10.1007/s11192-024-05143-8

Download citation

Received : 11 February 2024

Accepted : 20 August 2024

Published : 21 September 2024

DOI : https://doi.org/10.1007/s11192-024-05143-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Reciprocal recommendation
• Content filtering
• Collaborative filtering
• Find a journal
• Publish with us
• Track your research

A Federated Learning Method for Secure Recommendation Systems with Blockchain's Help

15 Pages Posted: 21 Sep 2024

Nanjing University

Wanchun Dou

Due to the privacy advantages of federated learning (FL), federated recommendation systems (FedRSs) are gaining popularity for improving recommendation performance through training on local data.However, FedRSs frequently face the significant challenge of high communication costs between the server and clients. Most FedRSs utilize a client-server communication architecture, leading to heavy communication loads and single points of failure due to dependence on a central server. Clients may also encounter problems due to limited communication resources.In view of this challenge, in this paper, we propose a blockchain-assisted federated learning method at edge for communication-efficient recommendation systems, named BFedRec. Specifically, BFedRec reduces reliance on the central server by utilizing blockchain systems to aggregate and distribute the recommendation model. To mitigate the high communication costs in each iteration, a communication-efficient training algorithm is used that trains the recommendation model directly on low-rank compressed parameters. Meanwhile, we apply a fake model detection mechanism to prevent malicious poisoning attacks from clients, ensuring the security and reliability of the federated recommendation system.Finally, we conduct extensive experiments on real-world datasets to verify the communication efficiency of BFedRec compared to existing methods. The experimental results show that BFedRec effectively improves communication efficiency without compromising recommendation performance.

Keywords: Federated Learning, Recommendation System, Communication Efficiency, Security, Blockchain, Low-rank Training, Fake Model Detection

Suggested Citation: Suggested Citation

Hang Zhang (Contact Author)

Nanjing university ( email ).

Nanjing China

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, computing technologies ejournal.

Subscribe to this fee journal for more curated articles on this topic

Artificial Intelligence eJournal

Innovation law & policy ejournal, ebusiness & ecommerce ejournal, data science, data analytics & informatics ejournal.