The Problem
Two main issues arise when sending data directly from source to SIEM: un-unified schemas and cost. This article focuses on un-unified schemas and their impact on SecOps teams.
What is a Schema?
A schema is a structure that defines how data should be organised and formatted. This ensures clarity and consistency, acting as a strong basis for data usability. A unified schema ensures all schemas within an environment are identical.
Think of it like road terminology. In the UK we say motorway, in the US it’s highway, in Australia it’s freeway. Different names, same infrastructure. Applications do the same thing — each generates logs using its own schema conventions while serving the same underlying security function. With approximately 7,100 languages globally, translators are necessary. Organisations managing thousands of data sources need the equivalent: a universal schema.
Traditional Approach
SIEM vendors prioritise frictionless log onboarding. The problem is that frictionless onboarding removes the only opportunity you have to audit, transform, clean, enrich, and route your data before it enters the estate.
This approach bypasses normalisation at the edge and shifts translation work to analytic runtime — resulting in inefficient queries, high compute costs, and sometimes failed queries entirely. By the time the data is in the bucket, the damage is done.
Threat Detection
Un-unified schemas complicate all threat detection logic, from single IOC rules to complex correlation rules. Detection engineers often must write multiple rule variants because they cannot lift-and-shift a basic detection across different applications when the schemas are inconsistent.
With unified schemas, use cases are written once and work universally. The detection engineer stops spending time on schema translation and starts spending time on actual detection quality.
Incident Response
Incident response teams are measured by MTTR — Mean Time to Respond. In high-severity incidents, responders are working to 5-minute MTTA and 10-minute MTTR windows.
Un-unified data forces responders to maintain a mental translation map for every tool in the estate. Is the IP field src_ip, ip.address, or IPAddress? This cognitive load during active investigations leads to mistakes and missed connections at exactly the wrong moment.
SOAR
Automation effectiveness suffers when schemas are inconsistent. SOAR platforms struggle to create reusable automation playbooks across tools with different data structures. Engineers end up building and maintaining parallel versions of the same playbook — one per vendor — rather than one that works everywhere.
Machine Learning
ML adoption in SOCs is increasing rapidly, with the potential for fully autonomous AI-driven operations on the horizon. But that future requires a strong foundational data layer. ML operates optimally with minimal variables, using mathematical branches to identify patterns and map inputs to outputs.
Un-unified schemas introduce excessive variables. The model wastes compute determining what a field means rather than performing its assigned task. When new applications introduce new schemas, ML must restart iterative training processes from scratch. Industry research on AI hallucinations consistently points to the same root cause: models trained on inconsistent, poorly-structured data generalise badly and produce unreliable outputs.
The fix is the same one that solves every other problem on this list: normalise before the data ever enters the model.
Knowledge Base
Non-unified schemas also complicate knowledge base maintenance — an often-overlooked cost. Investigation guides and work instructions become obsolete whenever a vendor changes a field name. Creation time increases exponentially as teams maintain separate runbooks for each application. Frameworks cannot be standardised across tools with differing schemas, and proper knowledge base maintenance becomes nearly impossible to sustain.
Unified schemas make runbooks universal. Write the investigation guide once — it works regardless of which vendor generated the log.
Derwent Labs Approach
Unlike traditional methods, Derwent Labs provides the opportunity to audit, transform, clean, enrich, and route data before ingestion. We offer a fully managed, yet fully bespoke, security data pipeline that transforms messy, un-unified data into consistent schemas before it ever reaches your SIEM.
Benefits across the stack:
- Threat Detection: Every log maps to standard before SIEM ingestion. Write once, deploy everywhere.
- Incident Response: Analysts consistently find standard field names regardless of vendor source. No more mental translation maps under pressure.
- SOAR: Agnostic automation playbooks with a unified framework. One playbook, any trigger.
- ML Readiness: A solid, normalised foundation for ML adoption from day one.
- Knowledge Base: Minimal revisions needed as the environment evolves.
Final Thoughts
Sustainable systems require solid foundations before you implement tech stacks, processes, and principles on top of them. The modern SOC architecture — agent, SIEM, SOAR, case management — is due to be rewritten with data sanitisation and unification as pivotal layers adopted before data ever reaches the ingestion point.
You don’t just have to manage logs — you can do whatever you want with them. The ingestion layer is the most powerful point of control in your entire security estate. Own it.
We invite you to test our pipeline with your messiest logs at no charge. We’ll show you exactly how it improves your SOC operations — and we can demonstrate it within ten minutes.