The rise of Big Data has made it the next frontier for innovation, waiting to be conquered by data professionals who will unlock and harness its value for organisations. Data Science is a team sport whose power players function in three fields: data engineers, data analysts and data scientists.
Recognition of these highly sought-after data professionals have led to a global talent shortage and Malaysia is no exception. Much focus has been placed on the top of the pyramid where resides the data scientist whom is commonly dubbed “as rare as Unicorns”. However, through our conversations with business leaders, public sector organisations and vendors alike, the role that is currently hardest to fill is that of data engineers.
The data engineer’s role is at the foundation of the data life cycle and crucial in the successful delivery of any reporting or analytics projects. Reporting, analytics or any model build for that matter relies on good data management and processes. These data engineers are responsible for capturing, cleansing, storage, use and care processes of an organisation’s data, essentially, the cradle-to-grave management of data. The roles within the data engineer sphere includes data architects, database analysts (DBA), data governance specialists and data managers.
What makes a good data engineer? Although there are several different specialist roles to occupy, to make it in data management in general, one must have skills in advanced programming, building extract-transform-load (ETL) processes, fundamentals of data management, developing and managing a data warehouse and databases, knowledge of data governance, developing and maintaining an enterprise-wide data dictionary and process mapping.
Data engineers should also understand the fundamentals of experiment set-up, model deployment and basic statistics. Of course, the ability to communicate, present at and lead meetings accompanied by a strong understand of the business is an added bonus!
Malaysia is aiming to have developed 20,000 data professionals by 2020, 2,000 of which are data scientists while the rest will consist of data analysts and engineers. If majority truly rules, perhaps in Malaysia the title of “as rare as Unicorns” should be bestowed to the Data Engineers’ role?