From CIA to Morgan Stanley, data engineers are in high demand
From Morgan Stanley to Ericsson, IBM and even the CIA, the US spy agency, nearly every organization is looking to hire Data Engineers. A LinkedIn search shows that almost 15,000 openings for data engineers while Indeed.com shows over 10,000 openings. However, there is a confusion about the difference between a data engineer and a data scientist, the latter having a higher glam quotient in the minds of the average person. In simple terms data engineers process the raw data while data scientists explore the data to find actionable insights. So, without data engineering, data analysis won’t be possible.
The Dice 2020 Tech Job Report labelled data engineer as the fastest growing job in technology in 2019 in the US, with a 50% year-over-year growth in the number of open positions. Interest in the position has been increasing over the years, as organisations discover data engineers are they key personnel for unlocking the value of their data. The Dice report, which analysed job postings from the past year as well as the US’ top tech-job hubs, says the top five skills descending order). The Dice report also notes that AWS (i.e.,Cloud) skills are in high demand for data engineering jobs, as well as Scala and Hive skills.
Data Engineers, in any organisation, work together with data consumers and Information and Data Management Officers to determine, create, and populate optimal data architectures, structures, and systems. Let’s look at what CIA wants its future data engineers to do. The primary goal is to increase discoverability and retrievability, facilitate dissemination, and ensure the delivery of timely and relevant intelligence. The agency is being overwhelmed by the amount of data it collects. The job description for Data Engineers would, therefore, include designing how the data will be stored, consumed, integrated, and managed by different data entities and digital systems.
Military and intelligence agencies around the world deal with a multitude of sensors like, for instance, the kind of tech found on drones. The CIA’s own sensors suck up incalculable mountains of data per second. Officers badly want to develop massive computational power within a relatively small, low-power sensor so the sorting can be done readily on the device itself, instead of being sent back to a central system. Data Engineers – with their extensive knowledge of data manipulation, databases, data structures, data management, and best engineering practices – are therefore going to be crucial players in a complex data-crunching organisation like the CIA.
In a prior development, CIA Labs, the agency’s newly-formed solutions arm, had already announced recruiting and fresh technical talent in diverse cutting-edge domains. These includes areas like artificial intelligence, data analytics, biotechnology, advanced materials, and high-performance quantum computing. Incentives are also being considered by CIA Labs to compete with private research establishments. CIA scientists will now be able to publicly file patents on the intellectual property they work on – and collect a portion of the profits too.
The Hired’s2020 State of Software Engineers report also show that demand for data engineers rose a respectable 45%. The clear leader in the engineering profession are those who can build virtual reality and augmented reality (VR and AR) applications. According to Hired, demand for AR/VR engineers rose a whopping 1,400% in 2019. The growth is due to the maturation of AR and VR technologies, the company says. Glassdoor estimated the average base pay for data engineers at US$102,864annually! This is based on earnings reported by thousands of companies.