If you’ve ever wondered what’s inside the “black box” of machine learning, the answer is less mysterious than it seems. Machine learning is not a single spell, but a workshop full of tools, each one designed for a different kind of problem. In fact, the No Free Lunch theorem (Wolpert, 1996) proves mathematically that no single algorithm outperforms all others in every possible scenario. The key, then, isn’t memorizing a magic formula — it’s knowing the menu of options and understanding which one to use in each situation.
Below we take a tour through seven fundamental algorithms, organized into two broad categories: supervised learning, where the data comes labeled, and unsupervised learning, where the model must find patterns on its own.
Supervised: when the data already has answers
Linear regression
This is the starting point for almost everything. Linear regression models the relationship between a dependent variable (what we want to predict) and one or more independent variables (the features) as a linear combination of parameters. In simple terms: it draws the straight line that best fits the points, minimizing the sum of squared errors (ordinary least squares).
When to use it? When the target is a continuous numerical value — predicting the price of a house, tomorrow’s temperature, or next quarter’s sales — and the relationship between variables is roughly linear. Its regularized variants (ridge, lasso) help when there are many features or risk of overfitting.
Limitation: it assumes linearity and is sensitive to outliers.
Logistic regression
Despite its name, it isn’t used for regression but for binary classification. Is an email spam or not? Does a patient have a certain disease or not? Logistic regression models the probability that an event will occur using the sigmoid function, which squeezes any value into a number between 0 and 1. The decision boundary is usually set at 0.5, but can be adjusted based on the cost of false positives and false negatives.
When to use it? When you need to classify into two categories and the boundary between them is approximately linear. It’s fast, interpretable, and works well even with relatively small datasets.
Limitation: its linear decision boundary cannot capture complex relationships without additional feature engineering.
Decision trees
A decision tree is like a game of “20 questions”: the model learns a sequence of if-else rules from the data. Each internal node asks about a feature (“does income exceed $50,000?”), each branch is a possible answer, and each leaf is a prediction.
Its great advantage is interpretability: you can visualize the entire tree and explain why each decision was made. It also requires little data preparation and handles both numerical and categorical values.
When to use it? When transparency matters more than raw accuracy — for example, in medicine or finance where you need to justify every prediction.
Limitation: they tend to overfit. A very deep tree can memorize the noise in the data instead of learning the signal. That’s where the next tool comes from.
Random Forest
If a single tree is fragile, a hundred trees together are robust. Random Forest, proposed by Leo Breiman in 2001, builds hundreds of decision trees — each one trained on a slightly different sample of the data (a technique called bagging) and considering only a random subset of features at each split. Then, for classification, it takes the mode of all the trees; for regression, the average.
The result is a model far more stable than a single tree, with lower variance and without significantly increasing bias. The law of large numbers guarantees that the error converges as more trees are added.
When to use it? It’s one of the most versatile models out there. It works well with tabular data, handles high dimensionality, tolerates missing values, and provides a measure of feature importance. If you don’t know where to start, Random Forest is a safe bet.
Limitation: it sacrifices the interpretability of a single tree for accuracy. With hundreds of trees, you can no longer “see” the entire model.
Support Vector Machines (SVM)
SVM searches for the hyperplane that best separates two classes — not just any dividing line, but the one that maximizes the margin between the closest points of each class (the “support vectors”). This makes it especially robust to new data.
Its true power lies in the kernel trick: by transforming the data into a higher-dimensional space, SVM can learn non-linear decision boundaries without explicitly computing that transformation. The most common kernels are linear, polynomial, and RBF (radial basis function).
When to use it? It shines when the number of features exceeds the number of samples — genomics, text classification, facial recognition — and when you need a model with good generalization.
Limitation: choosing the kernel and regularization parameters requires careful tuning. It doesn’t produce direct probabilities (one must resort to computationally expensive cross-validation to obtain them). In the last decade, gradient-boosted trees and neural networks have surpassed it in many tasks, but it remains the queen in certain niches.
Unsupervised: when the data has no labels
K-Means
K-Means is the quintessential clustering algorithm. Its goal is to partition the data into k groups, where each point belongs to the group whose centroid (average) is closest. The process is iterative: k centroids are initialized, each point is assigned to the nearest centroid, centroids are recalculated as the average of the assigned points, and the process repeats until the groups stop changing.
When to use it? For customer segmentation, document organization, image compression, or any task where you suspect the data forms natural groups.
Limitation: you must choose k beforehand (the elbow method and silhouette coefficient help, but they are heuristics). It assumes that clusters are spherical and of similar size. Also, the result depends on initialization — which is why it’s run multiple times with different seeds.
Principal Component Analysis (PCA)
PCA doesn’t predict or cluster: it reduces dimensionality. It transforms a set of possibly correlated variables into a smaller set of orthogonal (uncorrelated) components that retain most of the original variance. It’s a linear transformation — it finds the eigenvectors of the covariance matrix or uses SVD decomposition — and therefore doesn’t capture complex non-linear structures.
When to use it? Before applying another algorithm: PCA reduces noise, speeds up training, avoids the curse of dimensionality, and facilitates the visualization of high-dimensional data. It’s also the foundation of compression and facial recognition systems (eigenfaces).
Limitation: being linear, it fails on data with curved geometry (the classic “Swiss roll”). If the variance doesn’t align with the relevant information, PCA may discard exactly what matters. And it’s sensitive to the scale of features — you must always standardize before applying it.
Why knowing these tools matters
Understanding this menu of algorithms is what separates those who merely run code from those who know how to build robust solutions. Each tool has a distinct profile of interpretability, accuracy, speed, and assumptions. Linear regression gives you clear explanations; Random Forest gives you accuracy at the cost of transparency; SVM gives you clean margins if you know how to choose the kernel; K-Means uncovers groups you didn’t know existed; PCA clears the path before you model.
Real machine learning isn’t about finding the “best” algorithm — it’s about knowing which one is right for the problem in front of you. As George Box said: “all models are wrong, but some are useful.” Knowing your toolbox is the first step toward building the ones that truly are.
Main source: Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning, 2nd ed. Springer. Available at https://hastie.su.domains/ElemStatLearn/. The scikit-learn documentation (v. 1.8.0) and the original papers by Breiman (2001), MacQueen (1967), Pearson (1901), Hotelling (1933), and Wolpert (1996) were also consulted for the verification of each algorithm.