The year 2023 has been a watershed for data generation, with a deluge of 120 zettabytes—a figure that’s hard to fathom, like the vastness of oceans themselves. Each day, our digital universe expands by more than 300 million terabytes, setting us on a course to generate a staggering 180 zettabytes (180 billion terabytes) by 2025, with 80% of this data being unstructured. Concurrently, the Internet of Things (IoT) is set to surpass 21 billion devices, up from today’s 15 billion. The true challenge lies not in the sheer volume of data collected but in distilling this expansive sea of information into meaningful insights and actionable foresights, especially in sectors like oil and gas, where the untapped potential to revolutionize processes remains vast and largely unexploited.
A typical offshore drilling platform is equipped with approximately 40,000 sensors, collectively generating 2 to 4 terabytes of data each day. Yet, about 40% of this data is not stored, and 80% remains unused. Despite technological advancements in seismic acquisition devices, logging while drilling (LWD), and measurement while drilling (MWD) tools, which have significantly increased the volume of data available for processing, only a small fraction is actively utilized. This inefficiency is exacerbated by the fact that merely 1% of the data is transmitted to onshore facilities for daily analysis, largely because most of it is unstructured and thus not readily leveraged.
In the vast ocean of data, the daily output from a single well can exceed 10 terabytes, as it monitors a multitude of parameters, including downhole pressure, temperature, fluid density, fluid conductivity, and mass flow rate. Even remote onshore operations, such as an isolated oil rig in Northern Canada, contribute significantly, generating over 1 terabyte of data each day. However, the current infrastructure for data transmission is inadequate; a standard satellite link with a 25Mbps upload speed requires nearly four days to transmit a single day’s worth of data—a terabyte—to a data center. This bottleneck in data transmission underscores the urgent need for on-site data processing solutions.
The vast quantities of data generated by large-scale refineries, pipeline inspections, and seismic surveys are equally staggering, with volumes reaching up to 10 terabytes per survey. Pipeline inspections yield roughly 1.5 terabytes for every 300 miles (480 km) inspected, and ultrasound scans produce about 1.2 terabytes over eight hours. However, the processing capabilities at present, especially in remote or bandwidth-restricted locations, fall short. This inadequacy results in exorbitant service costs and significant maintenance challenges.
The diverse nature of data, encompassing structured, semi-structured, and unstructured formats, presents its own set of challenges. The value of storing unstructured data in its raw form is increasingly scrutinized due to its frequent lack of meaningful or actionable insights. For example, a 4K camera operating at 30 frames per second with H.265 encoding (0.015 compression factor) produces a 4Mbps data stream, yet typically less than 20% of this data is operationally relevant. With a 25Mbps satellite link, streaming capabilities are limited to just six cameras for cloud processing. The emergence of sophisticated AI offers a promising resolution. Leveraging video-to-text generative AI and multimodal AI for data consolidation enables the industry to refine and extract pertinent information from large unstructured datasets, significantly improving real-time decision-making and post-event retrieval.
The critical need for real-time data processing in the oil and gas industry is paramount, especially in high-stakes situations where rapid response can avert catastrophic blowouts. Immediate analysis of well data—from flow-out sensors, pit volume totalizers, and pressure and temperature readings from downhole and surface sensors—is crucial during drilling operations to detect early warning signs, known as ‘kicks’, that can lead to blowouts if not addressed promptly. The adoption of on-site data processing technologies, such as those using Recurrent Neural Networks with long short-term memory architecture, epitomizes the move toward edge computing, facilitating instantaneous predictions that are vital for ensuring the safety and continuity of operations.
Standing at the precipice of a data eagre, the oil and gas industry’s future is predicated on the adept management and application of AI to its data. This evolution encompasses a critical transition from cloud-based batch processing to on-premises real-time analysis, signifying a fundamental shift in the role of data: from mere output to a driving force for innovation, safety, and operational efficiency within one of the world’s most vital industries.
The goal is unequivocal: to revolutionize the industry’s AI and data strategy, ensuring that each byte lays the foundation for a safer, more productive, and environmentally responsible future. As we chart a course through the zettabyte surge, the destiny of the oil and gas sector is anchored to the advanced application of AI and edge computing, leveraging torrents of raw data into the currency of instantaneous, strategic decisions. In this expansive data sea, every byte harbors the promise of being as critical as a barrel of oil.
