< Back to previous page

Project

Flexible and weak supervision techniques for learning

Anomaly detection methods aim to identify examples that do not follow the expected behavior. For various reasons, anomaly detection is typically tackled by using unsupervised approaches that assign real-valued anomaly scores based on various heuristics. For instance, one can assume that anomalies fall in low-density regions and compute the negative log-likelihood as anomaly score.

Because anomaly scores are often hard to interpret, practitioners need class labels (i.e., anomaly yes/no) for decision-making. That is, one needs to set a proper decision threshold to flag high-score examples as anomalies. However, finding a threshold requires having access to labeled examples for evaluating the quality of the predicted class labels, which is unfeasible in unsupervised anomaly detection. Moreover, existing literature has focused mainly on measuring the quality of the anomaly scores through ranking-based metrics (e.g., AUROC), which largely ignores the problem of how to derive class predictions. Here, we fill this gap by proposing three novel approaches to transform scores into class predictions.

Given a detector's class predictions, a natural question is: how likely does a prediction change when learning a detector on training data that is subject to slight perturbation? Because unsupervised detectors cannot refine the decision boundary by leveraging labeled examples, they tend to have high uncertainty in predictions. That is, slight changes in the training set often would yield a different decision boundary which, in turn, would flip some test examples' class prediction. This uncertainty makes it hard to deploy a detector in real-world applications as it deteriorates the practitioner's trust in its crucial predictions. Because existing literature largely ignores this problem, we fill this gap by proposing an unsupervised approach to quantify a detector's uncertainty in predictions.

While quantifying uncertainty is essential, practitioners also need a reliable way to assess whether they can trust a detector's prediction. That is, one needs to answer the question: is the detector's uncertainty low enough to rely on its prediction? This falls into the field of Learning with Rejection, where the model is allowed to abstain (i.e., defer the decision, or "reject" it) when its uncertainty is too high, such that practitioners can trust its output whenever it makes a prediction. Traditionally, learning with rejection approaches rely on evaluating the risk (or, equivalently, the cost) of making mispredictions to design the rejection mechanism, which requires labeled examples. Because no unsupervised method for rejection exists, we fill this gap and propose the first unsupervised anomaly detection algorithm with rejection.

Date:18 Sep 2019 →  Today
Keywords:PU Learning, Anomaly detection, Active learning, Semi-supervised learning, Transfer Learning, Uncertainty Quantification, Learning with Rejection
Disciplines:Computer science
Project type:PhD project