In the current times, the field of Data Science is flourishing rapidly in America. In addition, machine learning is becoming the main component of many decision-making pipelines in diverse industries ranging from fintech to healthcare. Although contemporary artificial intelligence usually highlights deep learning models, classical machine learning techniques continue to be the most widely utilized algorithms in enterprise data stacks.

As a data scientist, whether aspiring or practicing, you should develop a full understanding of the top ten algorithms described in this article. This article provides essential insights and key characteristics of these algorithms, explaining why they must be a part of the professional data scientist toolkit. In addition, we explain how they work under the hood and how you can use them to solve particular business problems.

1. Linear Regression (Supervised – Regression)

Linear regression is probably the most fundamental algorithm in statistical modeling. It predicts a continuous numerical target given one or several independent input features. Mathematically, Linear Regression fits a straight line or a plane to the given data points in order to minimize the sum of squares of the differences between the real data and the regression function.

y = \beta_0 + \beta_1x_1 + \beta_2x_2 + … + \beta_nx_n + \epsilon

Here, y is the target, x’s are input features, β stands for the learned coefficients (or weights) and ϵ is the residual term. The popularity of linear regression stems from the fact that it trains very quickly and produces interpretable coefficients that can be interpreted in the business context – i.e., show the change in real estate prices depending on the area size.

2. Logistic Regression (Supervised – Classification)

Logistic regression might be misleading in its name since this algorithm belongs to the family of classification tools, not regression ones. It is used in binary classification scenarios – cases when the target is divided into two classes, such as “Fraud” or “Normal”. Contrary to linear regression, logistic regression transforms its linear outputs through the Sigmoid Function:

sigma(z) = \frac{1}{1 + e^{-z}}

As a result, Logistic Regression squeezes the output into a narrow [0,1] range to obtain a probability. Then, depending on whether this probability exceeds the defined threshold (usually equal to 0.5), the data point gets assigned to either positive or negative class. Since Logistic Regression is quick to train and simple mathematically, it enjoys popularity in medicine and compliance.

3. Decision Tree (Supervised – Classification/Regression)

Decision Trees represent simple yet powerful algorithms that imitate the human thinking process. The essence of their working lies in repetitive division of the input dataset into increasingly similar subgroups based on a sequence of nested ‘if-then’ statements.

On each step, the algorithm goes through all the input features to identify the best possible dividing criteria maximizing data purity. For regression trees, this criterion is minimized variance, for classification – it can be calculated via Gini Impurity or Information Gain (entropy). Although a single decision tree is very clear and interpretable, the algorithm is extremely prone to Overfitting.

4. Random Forest (Supervised – Classification/Regression)

To address the problem of overfitting in decision trees, data scientists often use ensembles of several decision trees called Random Forest. This technique belongs to a wider class of algorithms – Ensemble Methods, which combine predictions of independent models to make a more reliable conclusion.

In particular, the Random Forest algorithm constructs several hundred independent trees simultaneously. First, the model randomly subsamples its training dataset in order to introduce randomness (Bagging), secondly – it uses only some features at each split point. After the training, the Random Forest passes the test input data through the forest and counts votes for each class – the majority wins.

5. Support Vector Machine (Supervised – Classification/Regression)

Support Vector Machine is a very powerful and geometrically driven algorithm used in challenging multi-dimensional classification problems. The idea behind it is finding the optimal separating plane or hyperplane maximizing the separation margins between different classes.

The data points sitting closest to the boundary are known as support vectors, the algorithm takes their help to place this boundary line correctly. Sometimes, there can be no possibility to separate points by a hyperplane in its native space – then the SVM employs Kernel Method to project the data into higher dimensions.

6. K-Nearest Neighbors (Supervised – Classification/Regression)

K-Nearest Neighbors is probably the most natural data scientist’s tool. The algorithm believes that data points close to one another belong to the same class – birds of a feather flock together. KNN doesn’t train any models at the training step; instead, it only memorizes the training data.

Then, when a new unclassified point arrives, KNN measures distances to all the historical examples and picks K nearest neighbors. K is an integer parameter, chosen by the user (for instance, 3 or 5); next, KNN counts the votes among these neighbors and assigns the data point to the majority class.

7. Naive Bayes (Supervised – Classification)

Naive Bayes belongs to a family of classification algorithms based on classical probability theory and Bayes’ Theorem in particular. Given certain evidence, it finds the probability of the event.

P(A|B) = \frac{P(B|A) * P(A)}{P(B)}

Naive Bayes assumes that all input features are statistically independent of each other (that’s why it is called “naive”), which means that calculating joint probability of all the features becomes technically difficult. Surprisingly, despite this assumption hardly ever holding in practice, Naive Bayes performs impressively in text classification problems.

8. Gradient Boosting Machines (Supervised – Classification/Regression)

Finally, the algorithm standing at the heart of most data science contests is Gradient Boosting. This algorithm, alongside its efficient implementations – XGBoost and LightGBM, are actively employed for solving both binary and multiclass classification and regression tasks.

Contrary to Random Forest, the Gradient Boosting Machine constructs decision trees sequentially rather than in parallel. At each iteration, a new tree is built to fix the mistakes made by its predecessor. To accomplish that, the next model learns the residuals of its preceding model.

9. K-Means (Unsupervised – Clustering)

When dealing with unlabeled data in the domain of unsupervised learning, K-Means algorithm dominates the cluster discovery area. It groups an unstructured collection of data into several ($K$) non-overlapping and visually similar clusters based on similarity between its input features.

10. Principal Component Analysis (Unsupervised – Dimensionality Reduction)

The modern industry generates large amounts of structured or non-structured data, and sometimes, data scientists deal with the “Curse of Dimensionality”: huge number of features causes slowness and instability of predictive models. One of the possible solutions for this problem is Principal Component Analysis.

PC Analysis transforms the input columns to reduce their number (to avoid redundancy) and thus speeds up training. In addition, this method tries to keep all the useful information inside those few selected variables. As a result, training time decreases and the chances of overfitting are substantially lowered.

Conclusion

This list is intended to help you in building your arsenal of algorithms and data science tools. There are many other techniques worth knowing (such as LDA or Autoencoders), and there are also numerous other types of algorithms apart from Supervised and Unsupervised Learning (such as Reinforcement Learning).

Frequently Asked Questions (FAQ)

What is the difference between supervised and unsupervised learning algorithms?

Supervised learning algorithms work with fully labeled datasets, where every input record has the associated correct output, while Unsupervised Algorithms discover correlations in the unlabeled datasets independently.

How do I choose between using Random Forest and Gradient Boosting (XGBoost)?

Although both are excellent algorithms based on decision trees, it is usually easier to configure and train the Random Forest Model. GBM often provides better performance but needs a careful tuning of several hyperparameters in order not to overfit the training data.

Why does the Naive Bayes algorithm assume features are independent?

It makes the mathematics much simpler, although it almost never holds in practice. Nevertheless, it makes the Naive Bayes Algorithm suitable to solve complex text classification problems.

What is the purpose of feature scaling in K-Nearest Neighbors (KNN)?

Feature Scaling is mandatory for the KNN Algorithm because it is used to measure Euclidian distances between points in order to find their neighbors. If some input variables have significantly bigger scales than others (such as Annual income vs Age), the bigger features may dominate the distances and ruin the model’s predictions.

When should a data scientist apply Principal Component Analysis (PCA)?

PCA is needed when a dataset suffers from the “Curse of Dimensionality” – excessive number of input variables slowing down the training process and making it vulnerable to overfitting.