AI Powered Recommendations in Education & Apache Prediction IO

Posted by:
Muhammad Mazher

October 11, 2019

Research shows that people who consult online product recommendations tend to choose those products twice as often as people who don’t consult similar recommendations. In today’s fast-paced world, people want to save time wherever and whenever possible. However, when you have a plethora of options to choose from in your favourite online store, narrowing down what you want tends to take time. This is where personalised recommendations come in handy.

Online shopping is not the only place where you experience and benefit from personalisation. Recommendations based on personalisation have propagated into various aspects of our lives including travel, news, stock market prediction, sales predictions, law enforcement, education, medicine and much more. It is human nature to appreciate something that is targeted for you and you only. However, it should be noted that recommendations can both enhance as well as deteriorate user experience. For example, promotional emails that you never signed up for are a rather poor form of unsolicited recommendations.

Personalisation in Teaching & Learning

Over the last decade providing deeply personalised teaching and learning has been a key objective for higher education providers. In practice however, this can be difficult to achieve. Delivering the same educational content using the same methods for different learner profiles is not the ideal approach anymore. Personalisation in education is particularly important because it helps students as well as academics in several different ways.

Targeted (personalised) recommendations can:

1. Help students stay on top of their assignments and quizzes,

2. Inform students about important learning materials,

3. Save students time in finding the right learning material,

4. Help students find the right course of study as opposed to conventional student-teacher counselling where a counsellor gives course options to choose based on grades,

5. Improve the learning experience of students by allowing them to purely focus on their studies and

6. Help academics determine focus areas of teaching.

Studies have shown that students tend to learn better when taught in smaller groups as they feel a part of the classroom and have more opportunities to interact with the teacher.

This effect can be compounded with Cloud (off-campus) students as they may experience a sense of disconnection from the physical classroom. In an ideal world, the best way to deliver a premium education experience to a learner would be to have a one-to-one setting between teacher and student.

However, this is not realistic given the limited academic resources. Therefore, in order to help bridge the gap between what is ideal and what is reality in teaching delivery, a personalised recommendation assistant fits in quite nicely.

In order to fully reap the manifold benefits of modern education, we should embrace personalisation in all forms of education delivery. Universities use personal assistants in the form of mobile apps to provide students with general and personalised information. Surveys1 have shown that the majority of students prefer having a personal assistant available to triage first-level support requests in real-time as they need help. Deakin Genie is such an assistant developed by Deakin University.

AI (Artificial Intelligence) Approach to Personalisation in Teaching & Learning

Traditional approaches of demonstrating valuable learning content such as displaying a page view count do not prove to be very effective. Just because a resource has a higher page view count does not necessarily make it the best resource out there. There are several reasons why page view count is the wrong metric for deciding recommendations:

1. Application itself might refresh the page every few seconds to have an updated view, hence adding to the page view count,

2. The user may refresh the page unintentionally,

3. Bad implementation of tracking code might count page visits falsely and

4. Let’s not forget the robots and spiders (both good and bad) crawling all over the web adding to your page view count.

As big data platforms become mature, more data is now available to join together and determine how to personalise a resource so that it is the most effective for the student. Imagine a platform that houses thousands of data points which then helps build an image of a user’s journey rather than just their page hits. For example, Amazon uses data points such as a user’s purchase history, current items in their cart, item ranking, likes received by an item and much more. This is why 35% of Amazon’s sales originate as hyper-personalised recommendations to its users. Some traditional and effective approaches include item-based filtering, collaborative filtering and rule-based filtering.

An AI-based approach to provide personalised recommendations to its users is generally multi-pronged. Some of the key areas of an AI-based recommendation system can include:

1. User profiling,

2. Content profiling and

3. Pattern matching.

In order to generate accurate and meaningful personalised recommendations it is imperative to have lots of accurate data that is very carefully modelled by a data expert. The goal of processing thousands of inter-related data points is to build a clean, accurate, and truthful model of a user. It should be kept in mind that artificial intelligence is a child of data science. Therefore, AI is not effective without any data. Particularly, when it comes to recommendations, the formula is: the more the better.

Apache Prediction IO

There are several open-source recommender systems available that may be able to provide basic personalised recommendations. Apache Prediction IO is a good example of such a system with templates available for e-commerce, product and music recommendations as well as for frequent pattern mining. It is an easy-to-use machine learning server that is built on top of a state-of-the-art open-source stack for developers and data scientists that allows you to quickly build predictive engines. The key component that makes Prediction IO so appealing is its wide coverage of RESTful APIs, which basically makes integration with websites and mobile apps totally effortless.


You can install Prediction IO either from its source code or use it as a Docker container. Installation instructions can be found here:

It is quite easy to set it up as a Docker container as it packages all the dependencies well and needs only one command to initiate. Once initiated, you only need one command to build: “pio build”, one command to train “pio train” and one command to deploy the engine: “pio deploy”. This makes the engine available to be consumed over a RESTful API running locally on the server on a dedicated port. Configuring the pre-sets is as easy as modifying a handful of attributes in a JSON file.

Apache Prediction IO


Prediction IO for Recommendations uses an architecture called DASE, which stands for Data, Algorithms, Serving and Evaluator. The open-source stacks that combine to make Prediction IO recommender engines include Apache Spark, Apache HBase, and Spray. Prediction IO is made up of two primary building blocks:

1. The Event Server which is responsible for holding all data in the form of time-based events and

2. The Engine which is responsible for building predictive models based on pre-set algorithms and hyper-parameters using the data in event server.

Apache Prediction IO Architecture


Apache Prediction IO is just one of many open-source recommender systems available, each with its pros and cons. A few things to keep in mind while choosing a recommender system for your organisation are:

1. Level of difficulty to setup, use, and integrate with apps,

2. Scalability,

3. Skills required to use the engine,

4. Activity on GitHub (or any other hosting for version control software) and

5. Support available in the community.

Last but not least, if you wish to check out Apache Prediction IO predictive engines, their Classification, Regression and NLP engines are definitely worth a visit.

Other open-source (academic-focused) recommender systems include:

1. LibRec (Java)

2. RankSys (Java)

3. LIBMF (Matrix-factorisation library)

4. proNet-core – A general-purpose network embedding framework which provides several factorisation-based models for recommender systems.

Some more recommender systems:

1. Raccoon Recommendation Engine (Node.js)

2. HapiGER (Node.js)

3. Seldon (Java)

4. LensKit (Java, Apache Spark)

5. Oryx v2

6. Crab (Python)

7. Predictor (Ruby)

8. LightFM

9. Rexy (Python, NoSQL)

Research / Surveys:

Subsection: 7 AI-Based Digital Assistants as Tutors and BISE Researchers as Integrators of Interdisciplinary Insights and Creators of Design Knowledge




Post navigation

Join the conversation

Your email address will not be published. Required fields are marked *

back to top