Beginners Guide: Apache Spark Machine Learning Scenario With A Large Input Dataset
What if you want to create a machine learning model but realized that your input dataset doesn’t fit your computer memory? Usual you would use distributed computing tools like Hadoop and Apache Spark for that computation in a cluster with many machines. However, Apache Spark is able to process your data in local machine standalone mode and even build models when the input data set is larger than the amount of memory your computer has. In this blog post, I’ll show you an end-to-end scenario with Apache Spark where we will be creating a binary classification model using a 34.6 gigabytes of input dataset. Run this scenario in your laptop (yes, yours with its 4–8 gigabytes of memory and 50+ gigabytes of disk space) to test this.
1. Input data and expected results
In the previous post we discussed “How To Find Simple And Interesting Multi-Gigabytes Data Set”. The Posts.xml file from this dataset will be used in the current post. The file size is 34.6 gigabytes. This xml file contains the stackoverflow.com posts data as xml attributes:
- Title — post title
- Body — post text
- Tags — list of tags for post
- 10+ more xml-attributes that we won’t use.
The full dataset with stackoverflow.com Posts.xml file is available here at https://archive.org/details/stackexchange. Additionally I created a smaller version of this file with only 10 items\posts in it. This file contains a small size of original dataset. This data is licensed under the Creative Commons license (cc-by-sa).
As you might expect, this small file is not the best choice for model training. This file is only good for experimenting with your data preparation code. However, the end-to-end Spark scenario from this article works with this small file as well. Please download the file from here.
Our goal is to create a predictive model which predicts post Tags based on Body and Title. To simplify the task and reduce the amount of code, we are going to concatenate Title and Body and use that as a single text column.
It might be easy to imagine how this model should work in the stackoverflow.com web site — the user types a question and the web size automatically gives tags suggestion.
Assume that we need as many correct tags as possible and that the user would remove the unnecessary tags. Because of this assumption we are choosing recall as a high priority target for our model.
2. Binary and multi-label classification
The problem of stackoverflow tag prediction is a multi-label classification one because the model should predict many classes, which are not exclusive. The same text might be classified as “Java” and “Multithreading”. Note that multi-label classification is a generalization of different problems — multi-class classification problem which predict only one class from a set of classes.
To simplify our the first Apache Spark problem and reduce the amount of code, let’s simplify our problem. Instead of training a multi-label classifier, let’s train a simple binary classifier for a given tag. For instance, for the tag “Java” one classifier will be created which can predict a post that is about the Java language.
By using this simple approach, many classifiers might be created for almost all frequent labels (Java, C++, Python, multi-threading etc…). This approach is simple and good for studying. However, it is not perfect in practice because by splitting predictive models by separate classifiers, you are ignoring the correlations between classes. Another reason — training many classifiers might be computationally expensive.
3. Setup and Run Apache Spark in a standalone mode
If you don’t have Apache Spark in your machine you can simply download it from the Spark web page http://spark.apache.org/. Please use version 1.5.1. Direct link to a pre-built version — http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz
You are ready to run Spark in Standalone mode if Java is installed in your computer. If not — install Java.
For Unix systems and Macs, uncompress the file and copy to any directory. This is a Spark directory now.
Run spark master, slave and shell:
Spark shell can run your Scala command in interactive mode.
Windows users can find the instruction here: http://nishutayaltech.blogspot.in/2015/04/how-to-run-apache-spark-on-windows7-in.html
If you are working in cluster mode in a Hadoop environment, I’m assuming you already know how to run the Spark shell.
4. Importing libraries
For this end-to-end scenario we are going to use Scala, the primary language for Apache Spark.
5. Parsing XML
We need to extract Body, Text and Tags from the input xml file and create a single data-frame with these columns. First, let's remove the xml header and footer. I assume that the input file is located in the same directory where you run the spark shell command.
Spark has good functions for parsing json and csv formats. For Xml we need to write several additional lines of code to create a data frame by specifying the schema programmatically.
Note, Scala language automatically converts all xml codes like “<a>” to actual tags “<a>”. Also we are going to concatenate title and body and remove all unnecessary tags and new line characters from the body and all space duplications.
To create a data-frame, schema should be applied to RDD.
6. Preparing training and testing datasets
The next step – creating binary labels for a binary classifier. For this code examples, we are using “java” as a label that we would like to predict by a binary classifier. All rows with the “java” label should be marked as a “1” and rows with no “java” as a “0”. Let's identify our target tag "java" and create binary labels based on this tag. Then dataset can be split into negative and positive subsets by using the new label.
We are going to use 90% of our data for the model training and 10% as a testing dataset. Let's create a training dataset by sampling the positive and negative datasets separately.
The testing dataset should include all rows which are not included in the training datasets. And again – positive and negative examples separately.
7. Training a model
Let's identify training parameters:
- Number of features
- Regression parameters
- Number of epoch for gradient decent
Spark API creates a model based on columns from the data-frame and the training parameters:
8. Testing a model
This is our final code for the binary “Java” classifier which returns a prediction (0.0 or 1.0):
Let's evaluate the quality of the model based on training dataset.
If you use the small dataset then the quality of your model is probably not the best. Area under the ROC value will be very low (close to 50%) which indicates a poor quality of the model. With an entire Posts.xml dataset, the quality is no so bad. Area under the ROC is 0.64. Probably you can improve this result by playing with different transformations such as TF-IDF and normalization. Not in this blog post.
Conclusion
Apache Spark could be a great option for data processing and for machine learning scenarios if your dataset is larger than your computer memory can hold. It might not be easy to use Spark in a cluster mode within the Hadoop Yarn environment. However, in a local (or standalone) mode, Spark is as simple as any other analytical tool.
Please let me know if you encountered any problem or had future questions. I would really like to head your feedback.
The complete source code of this program could be found here.
