NIPS 2014 Highlights

If you didn't go to NIPS 2014, read this great summary by John Platt at Microsoft Research.

Challenges in Machine Learning Practice - Part 1

In this series, I'd like to highlight a few challenges in developing and using a machine learning (ML) system in practice. This first post will focus on model serialization and how to consume a trained model.

The disconnect between training models and putting them into production

What’s the best practice to train and deploy a model?

Most of the major machine learning toolsets and platforms (e.g. SAS, R, Python libraries, Weka, RapidMiner, Hadoop, etc.) focus on providing algorithms and tools to help you analyze and visualize data, extract features, and train models. Once the data scientists are done playing with the data and having trained a good model, how to operationalize that trained model or workflow [1] remains a tricky question to most companies and groups.

What usually happens is one of the following.

1. The whole data mining [2] pipeline is offline. 
1. In some cases, companies just want to get insights from their landfill of data, to help them make more informed decisions. That’s it. They don’t need to or perhaps don’t know how to train/apply a predictive model.
2. Online prediction is not too critical. Some data scientists are willing to run prediction offline on new batches of data. Examples include customer retention analysis, risk analysis, fraud detection (many firms already do fraud detection online but there are still lots of firms running this offline). 
In both cases, machine learning is only for internal use. I’m guessing that the machine learning software market is dominated by these needs but won’t be for long. The future will rely more on predictive analytics.


2. Or after training and validating a model offline, the data scientists will ask other software engineers to implement the model decoder, typically in another language that is more efficient (C++/C#/Java), which can then be deployed as a live service.

(2) is what I want to talk about. There are many problems with this approach.

1. The educational bridge between data scientists (or statisticians) and production engineers. Basically, the data scientists need to explain the model to the engineers. Depending on the complexity of the model and other factors, this process could be long and error prone.
2. Retraining and updating the model is also a pain. After a few months, the data scientists will likely come up with better models, either through retraining with more data or through more fundamental changes. Now the engineers need to update the model. If the change is fundamental, such as switching from SVM to Neural Network, the model decoder would change significantly.


The process of improving the models is and should be endless, at least due to periodic retraining.

If you multiply the cost factors above with the number of projects that use machine learning, the waste of resources is far reaching. It typically takes months and multiple engineers to test and deploy an already trained machine learning model.

Undoubtedly, there are a few solutions out there and they are far from being ideal.

1. Large IT companies (Microsoft, Google, Facebook, etc.) may have the resources to build their own infrastructure to support serializing and consuming models in an automatic way. However, their solutions are (i) internal, (ii) even fragmented within the companies, and (iii) customized for their own data sources/format and their current learning algorithms.
2. Predictive Model Markup Language (PMML) is the most established and common format for serializing machine learning models. Many popular software (such as KNIME, RapidMiner, R, SAS, etc.) [3] support producing models in this format. However, PMML has a few disadvantages:
1. As an old standard, its evolution is very slow. For example, it doesn't support modern methods such as convolutional neural networks.
2. Software that can consume PMML models (e.g. SAS or IBM SPSS Modeler) tend to be expensive and not optimized for online usage.
3. Train a model in a data-friendly environment (let's say Python) and consume the model through that environment [3]. This approach, while fast to make something work, also induces major technical debts that you have to pay later. Some major debts are speed and scalability (both in terms of computation as well as software maintenance).

In summary, producing and consuming machine learning models is still painful for a large number of machine learning practitioners. The current market is very fragmented and still at an early stage. Software that pay attention to producing and consuming models tend to lack other important components to make them widely used. I'll highlight a few of those important components in the next posts.

Notes

[1] It should be noted that almost every predictive model is a workflow, not just a ML-trained sub-model.
[2] Sometimes I use “data mining” to emphasize the practice of mining data just for the sake of understanding but not for making predictions on new examples.

[3] This site provides a comprehensive list of software that can produce and/or consume PMML models.

The importance of cache efficiency in SGD optimization

Recently, some members in our team have experimented with variants of SGD (Stochastic Gradient Descent) and SDCA (Stochastic Dual Coordinate Ascent) on large and sparse datasets (such as the KDD Cup 2010 dataset [*]). Note that we've focused only on linear models (e.g. logistic regression and SVM) which lead to convex optimization.

What we have found during the process is very interesting from the engineering standpoint, yet not covered in any academic papers. That is: in SGD and SDCA, each weights update is typically so fast [**] that cache efficiency can become the limiting factor. Put it another way: we experienced that if the data is randomly shuffled in memory, the algorithm suddenly becomes ~3x slower (see the table below).

The difference between L1 and L2 regularization

Repost of my answer to a Quora question

The difference between \(L_1\) and \(L_2\) regularization is well studied in ML (and you can find lots of reference on the internet). However, the most common explanation is perhaps via this figure.

While it's true (and seemingly intuitive at first), this figure may not be as easy to digest. The figure corresponds to an equivalent constrained optimization problem, but why we need to deal with constrained optimization problem is not clear [1]. In fact, someone asked this question on Quora: What's a good way to provide intuition as to why the lasso (L1 regularization) results in sparse weight vectors?, which motivated me to provide a simpler explanation [2].

ICML 2014 Highlights 2: On Deep Learning and Language Modeling

Previously: Highlights #1 - On ML Fundamentals

Deep Learning and Language Modeling

Images classification seems to be the past. The current wave of DL research is all about language modeling. Here’re some interesting works on this front.

ICML 2014 Highlights 1: On Machine Learning Fundamentals

Abstract

At a high level, Deep Learning (DL) is still hot and DL keeps eating Machine Learning. The conference's attendance distribution was like: half was there for Deep Learning and the other half was there for *Shallow* Learning :). Interestingly, the conference took place in Beijing, for the first time, and more than 50% of the attendants either study or work there (and most of that local population are students). So the attendance distribution could be biased.

In the following, I'll highlight what I've learned and observed from the conference. Here's the outline:

• ML Fundamentals (this post, see below)
• Deep Learning and Language Modeling
• Optimization, Distributed Optimization, and Distributed ML
• Kernel Methods
• Auto ML
• Other Topics

On the imminent decline of MapReduce

Google recently announced at Google IO 2014 that they are retiring MapReduce (MR) in favor of a new system called Cloud Dataflow. Well, the article author perhaps dramatized it when quoting Urs Hölzle's words

We don’t really use MapReduce anymore. 

You can watch the keynote here for a better context. My guess is that no one is writing new MapReduce jobs anymore, but Google would keep running legacy MR jobs for years until they are all replaced or obsolete.

Regardless of what has happened at Google, I'd like the point out that MR should have been ditched long ago.

Someone at Cloudera (the company that used to make money on the hype of Hadoop MapReduce) already partially explained why in this blog post: The Elephant was a Trojan Horse: On the Death of Map-Reduce at Google. Some quotes to remember are:

• Indeed, it’s a bit of a surprise to me that it lasted this long.
• and the real contribution from Google in this area was arguably GFS, not Map-Reduce.

Every real distributed machine learning (ML) researcher/engineer knows that MR is bad [*]. ML algorithms are iterative and MR is not suited for iterative algorithms, which is due to unnecessary frequent I/O and scheduling plus other factors (see the illustration below). For more details on the weaknesses of MR, one can read any intro slides about Spark [**].

Also note that Mahout, the ML library for Hadoop, recently said goodbye to MapReduce.

25 April 2014 - Goodbye MapReduce

The Mahout community decided to move its codebase onto modern data processing systems that offer a richer programming model and more efficient execution than Hadoop MapReduce. Mahout will therefore reject new MapReduce algorithm implementations from now on. We will however keep our widely used MapReduce algorithms in the codebase and maintain them. 

We are building our future implementations on top of a DSL for linear algebraic operations which has been developed over the last months. Programs written in this DSL are automatically optimized and executed in parallel on Apache Spark.

Notes
[*] Unfortunately, lots of companies, including my employer, are still chasing the Hadoop game. Microsoft just less than a year ago announced HDInsight, aka. Hadoop on Azure.
[**] For virtually everything that MR can do, Spark can do equally well and in most cases better. Also note that while Spark is generally fantastic, it is not necessarily the right distributed framework for every ML problem.