Implementing monitoring strategies in data engineering pipelines to catch faulty data. The modern data platform is becoming more complex and data downtime, which is defined as the period of time for which we have missing or wrong data, is one of the results of this trend. The rise of cloud…

The need for version control and accountability is also applicable to the way in which we handle data. DevOps started around 20 years ago when the business requirements started to outpace the software development times. This led to the development of a series of methods that had as objective to…

Differences between Operational and Business Intelligence from a Data Engineering perspective. In recent years, the use of operational intelligence has become a rule as the use of decision-making during operations can improve overall efficiency. Here we will discuss how it has become separated from business intelligence and the challenges from…

Accountability through data quality and open access are the keys to trustworthy data With the explosion in the volume and complexity of the data being used everyday for decision making, the importance of ensuring transparency in the use of data has become paramount. Companies and organizations rely on data more…

Challenges and best practices of the design of modern cloud platforms Data has become more and more centric for organizations in the past years. The idea of being able to democratize the insights obtained from data to create more informed decisions throughout an organization is a must for various companies…

How to tackle the growing complexity of extracting business value from data. The role of the data engineer has been around in some sort for the last decades. Since the introduction of relational databases, it has been there to support the development and implementation of the infrastructure as well as…

Fostering data democratization and a data-driven culture with better data engineering practices Data Democratization is a term that seeks to provide transparency and accessibility to data for the average non-technical users within an organization. In doing this, companies and organizations can ensure that any decision being taken is supported on…

Modern data applications require lengthy processing cycles and early data checks can save time, catch errors, and ultimately save resources. The concept of a Data Pipeline makes a reference to industrial pipelines that are tasked with moving liquids or solids from one stage of the process to another. These “physical”…

How an efficient building design philosophy can save us headaches in the future. In recent years, organizations have been striving to make use of the existing data available. The increase in the amount of volume of data, as well as the capabilities and tools to process it, have caused organizations…