What is ML?
Machine Learning (ML) is a subfield of artificial intelligence (AI) that gives computers the ability to learn without explicitly being programmed.
These algorithms learn patterns within data, and then are able to make accurate inferences about new data.
Training
Note: This could be many articles within itself — I will be keeping this very high level, and focusing on how ML is adapted to the SOC and not how ML models are trained and run at their core.
The journey to a functional ML model begins with the training phase. This is a process of refining algorithms through model training. However, selecting the right learning method is only one part of the equation.
For a model to truly succeed, the underlying dataset must be normalised. Because of how these models perceive patterns, the consistency of the data is critical.
With a foundation of normalised data, the training process becomes significantly less complex, typically resulting in three core efficiencies:
- Computational Cost: Input standardisation minimises the variance within the dataset. This reduced dispersion allows the model to converge more quickly, lowering the total computational cost of the training cycle.
- Computational Time: By providing a consistent data foundation, standardisation eliminates the noise that slows down algorithmic learning. This results in faster model convergence, shortening the gap between data ingestion and actionable security intelligence.
- Minimising Model Hallucinations: By providing normalised data, mathematical relationships between variables remain constant. This results in a model that generalises better to unseen data, significantly reducing the probability of synthetic patterns or hallucinations that arise when an algorithm tries to reconcile un-unified data structures.
I look at it like building a house. Ideally, to build my house, I want stone, or brick that are easy to build with. If I had to build it with a load of end bits and rocks, I wouldn’t be happy.
SOC Benefits
While traditional threat detection remains a cornerstone of the modern SOC, it often leaves significant security gaps that are increasingly difficult to manage. Machine Learning addresses this by automating the monumental task of monitoring anomalies, transforming an otherwise insurmountable manual workload into a very efficient alerting mechanism.
The problem with traditional threat detection is that it is limited in its ability. It does a good job at signature based alerting, but doesn’t do too well with behavioural based alerting.
Now we all know that signature or IOC based alerting is scraping the bottom of the barrel when it comes to threat detection logic, so naturally we want to use behavioural based alerting.
It is so hard to write detection logic that marries perfectly to another set of detection logic, such that you have covered every TTP outlined in your scope. You either have to have a ridiculously small scope of detection logic that you need to create, or you need to constantly baseline, test and update every rule.
Here is where ML helps — a lot.
ML uses User and Entity Behaviour Analytics (UEBA) to learn what normal looks like for every user and device on your network.
The main advantage that UEBA has against traditional threat detection is that it monitors everything. Threat detection will only monitor what the detection engineer has specifically designed.
Unlike traditional threat detection, that gets worse the longer you leave it, UEBA gets better — because it has a wider scope of what is normal in your environment.
UEBA will allow for a different approach to threat detection. It is not a one size fits all approach. It understands that the marketing team’s threat profile and data usage looks a lot different from the finance team’s.
For the SOC analyst, ML can do some really cool things. It answers questions automatically.
SOC Analyst: “Is it normal for John to log in from this IP? What applications does he usually interact with?”
Machine Learning: “John is from a new IP, but his peers in the London office have also used this IP today.”
Because ML provides context around events, it drastically reduces the Mean Time to Respond as analysts are presented with the key information needed to build up the case.
Derwent Labs Solution
By implementing a solution that standardises the logs pre-ingestion time, ML becomes so much easier to build.
One of the primary hurdles for modern security teams is that they are finding it hard to transition to a modern day ML / Autonomous SOC because they will have to re-architect their whole ingestion pipeline.
Derwent Labs augments the architectural foundation of the ingestion layer, allowing for the integration of custom trained models that produce specific, high impact outputs for the modern SecOps team.
Because these models are built on a foundation of high integrity data, the resulting outputs are significantly more accurate, virtually eliminating the false positives that typically cause security teams to lose trust in automated systems.
Final Thoughts
For any SOC that wants to run ML, data normalisation will be imperative.
The SOCs that embrace the detachment of the ingestion layer from the SIEM and adopt a common schema will flourish when adopting ML. The ones who do not, and bolt ML onto data that is not normalised, will fail to keep up.
If there was one argument for the ingestion layer being its own component within a SOC, it would be ML adoption.
The gap between traditional SOCs and next-gen SOCs will grow at a rapid pace and much of the market share will be dominated by those who choose to adopt unification of schemas.
The future of the SOC lies in automated intelligence, but that future is only attainable through comprehensive data normalisation. Build your foundation today, or risk being left behind tomorrow.