Carl D'Halluin, Chief Technology Officer at Datadobi.
In my previous article, I discussed using insights to manage the vast amounts of unstructured data that organizations accumulate. This information is incredibly valuable—it empowers businesses to make informed decisions, optimize data management practices and unlock latent value.
However, extracting meaningful insights is only one part of the equation—businesses must also implement an efficient strategy for managing unstructured data to ensure long-term value.
Given the ubiquitous business love affair with unstructured data, the need for a cohesive approach is no longer up for debate. Depending on the source, unstructured data represents between 80% and 90% of the typical organization’s total data footprint and includes everything from emails and documents to multimedia files and IoT-generated information.
Unlike structured data, which fits neatly into databases, unstructured data lacks a predefined format, making it harder to store, search and secure. The awkward challenge is that it often contains critical business intelligence and sensitive information and can also be subject to regulatory obligations.
By The Book
With that in mind, let’s start by focusing on compliance, which remains one of the biggest stumbling blocks for businesses that want to handle their unstructured data better. Everyone will be familiar with regulations such as GDPR, DORA, APRA, CCPA and HIPAA, among many others, which set out strict rules and best practice guidance on data privacy and security. In each case, organizations must devote time and money to ensure sensitive information is stored, accessed and retained in full compliance with relevant requirements.
Adding to the inherent complexity created by compliance, unstructured data often contains personally identifiable information (PII) and other sensitive content. Without strong access controls and retention policies, businesses face increased exposure to security breaches and the associated regulatory penalties. They must also be ready for legal requests, such as data subject access requests (DSARs) under GDPR, which require organizations to locate, retrieve and delete personal data upon request.
As a result, the way organizations approach data retention and lifecycle management has taken on a new level of importance. For instance, some regulations dictate how long certain types of data must be stored and when they should be deleted—a requirement that becomes particularly important in more highly regulated industries. To be on the safe side, many businesses just default to keeping everything indefinitely, even when there is no specific requirement to do so.
Granted, this mindset can reassure leaders that they won’t be caught out at some point if the need to access a certain dataset emerges out of left field. But in reality, it’s an approach that stores potential problems. Think of it this way: If there are two competing businesses of similar size, but one never archives or deletes legacy data, it will inevitably spend more on storage infrastructure, backup solutions and security measures to protect its ever-growing repository.
Meanwhile, the competitor with a well-structured data archival and data retention policy can optimize costs, limit their exposure to security breaches and put themselves in a better position to ensure compliance without unnecessary overheads.
In practical terms, a properly structured governance framework enables businesses to implement automated lifecycle policies, ensuring data is archived or deleted as required. Regular audits, metadata analysis and real-time visibility into data ownership and access patterns can further strengthen compliance efforts while also helping businesses achieve efficiency and cost optimization goals.
The Quest For Efficiency
The wider issues associated with poor cost optimization are a challenge many businesses are just kicking down the road because they lack the visibility, strategy or tools to tackle it effectively.
But as they accumulate more data, the traditional approach of adding more storage has become unsustainable—it’s expensive, highly inefficient and adds significant security risks. Yet, it remains very common. There are many companies out there that continue to expand their storage infrastructure without assessing whether the data they’re retaining holds any real business value. Instead of implementing proactive data management strategies, they accumulate vast amounts of redundant, obsolete or trivial (ROT) data.
High-value, frequently accessed data should be stored in performance-optimized systems, while ROT data can be moved to lower-cost archival storage—or even outright deleted from the environment. Advanced data management solutions allow organizations to automate these processes, providing real-time insights into data usage patterns that help distinguish between mission-critical information and content that can be safely archived or deleted.
This not only reduces storage costs but also improves efficiency by streamlining access to essential data while eliminating unnecessary overhead. A bonus benefit is that the footprint of data exposed to the risk of a breach or a ransomware event is reduced.
That’s why a proactive data management strategy isn’t just about cost savings—it’s also about ensuring data is readily accessible, accurate and fit for purpose. This means that beyond the financial and compliance implications, businesses must also address how fragmented data environments hinder productivity and innovation.
Addressing these requirements has been given added impetus by the huge growth of GenAI applications. Training these models to deliver reliable outputs depends on providing access to relevant and curated data. If an organization’s unstructured data is fragmented, outdated or poorly governed, there’s a very real risk that AI outputs will be inaccurate "garbage in, garbage out."
By integrating data management with AI workflows, however, businesses can improve data quality, drive better decision-making and put themselves in a much stronger position to fully realize the potential of AI-driven technologies.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
