Data Pipelines for Operational Intelligence
Differences between Operational and Business Intelligence from a Data Engineering perspective.
In recent years, the use of operational intelligence has become a rule as the use of decision-making during operations can improve overall efficiency. Here we will discuss how it has become separated from business intelligence and the challenges from a data engineering perspective to support the use of real-time data to optimize day-to-day operations within organizations.
Operational Intelligence
Operational intelligence refers to the use of KPIs which are aggregations of real-time data that is used to support front-line workers in their day-to-day operations. It is intended to optimize operations by making use of data insights which are delivered as dashboards similarly as it is done in Business Intelligence.
Before operational intelligence came along, it was necessary to monitor operations for a long period of time until the data was aggregated and loaded into data warehouses with the help of data engineers. Then business intelligence analysts and engineers worked to create dashboards and KPIs that reflect the operation details and allow them to obtain insights that could be used to improve the process. Needless is to say that this required a long time, both for the observation and for the development of the pipelines, tables and dashboards. This is the common approach used by BI engineers to create dashboards and data visualizations. The problem with this approach was that the time difference, which could be weeks or months, made it impossible to use information for daily operations because the data would always be a picture of the past.
The key difference between business intelligence and operational intelligence is that the latter is real-time data, with a focus on the activities that are being undertaken during operations. Business intelligence focuses on the data itself and might involve the use of several data assets, and the decisions that ultimately are being taken will have a long-term approach.
Data Pipelines for Operational Intelligence
The use of data engineering tools to create reliable operational intelligence pipelines, different tools that are used, as well as best practices.
The data that is required for operational intelligence has its origins in data collector systems. Data collectors are systems that capture data about the activities that are taking place during the operations. This data can be obtained through server and application logs, events on specific systems that are captured, devices streaming data to pub-sub databases, and so on. These systems can be loggers that are embedded in applications or can directly be measurements of variables that might be useful to obtain insight that could help optimize processes. Although it was previously mentioned that this data is consumed in real-time, the actual definition of what “real-time” means will differ from the type of operation that we are monitoring. It is also important to consider the latency of data collector systems as they can range from milliseconds to minutes, depending on the volume and type of data, as well as the technical specifications of the data collector systems themselves.
The data that is collected from these systems is in its raw format, therefore it is impossible to make sense of it before normalizing it. This is the part where data engineering comes into play, by creating data platforms that can consume these streams of data, process them and aggregate them into analytics dashboards that can be later be used to visualize these insights. Once the data has been aggregated into dashboard and data marts there is the possibility to use this information to automatically trigger actions based on a predefined set of rules. These are called action dashboards and they allow you to visualize not only the values derived from the observation but also see the automatic triggers that are put in place.
Conclusion
The use of Operational Intelligence will continue in the future as more companies start to use the vast amounts of data that modern systems generate. It is crucial in this context for data engineers to be able to develop pipelines that can process and serve data in almost real-time. Technologies such as Apache Flink or Spark, as well as streaming databases, have become of paramount importance, as well as the development of APIs that allow consuming the insights obtained to automate actions. Time is valuable, and more when it comes to day-to-day operations. Data engineers are just starting to adopt this trend, although there is a lot of time to catch up as operational intelligence will become a day-to-day tool.
