What Makes A Great Data Engineer?

Mad Skills

The vast majority of technologies are easily consumable, i.e. they can be learnt with a keen mind and effort, there are some that are not.   The starting block should be a programming language, preferably two. The key is not the language itself, but exposure to thinking like a programmer and the ecosystem that comes with it. I expect an engineer to know how to write a for loop but why use this construct rather than foreach or filter or reduce functions – question the why rather than the how.

Depending on their experience I would expect some level of understanding of SQL, their knowledge should go beyond simple querying. They should be able to solve the same problem in a database and in code, and demonstrate flexibility of mind when it comes to problem solving.  Writing the code isn’t that important, understanding how to do it using set logic (or Venn diagrams even) is enough.

Using the command line seems increasingly like a lost art, but generally those who do have some knowledge have a better chance of becoming an outstanding data engineer. After all, being able to perform a Map Reduce function using cut, sort and wc requires a good bit of mental gymnastics.

Infrastructure is one of the areas where experience will impact knowledge. If someone has worked on Hadoop for a few years I would expect them to have an understanding of how the infrastructure impacts performance.  Cloud or on premise, it doesn’t matter.  If their experience is more cloud, I would expect their understanding of networking to be better, especially around security.

Most software engineers worth their salt will have views on design and architecture. They should be able to defend their views using what they have delivered, but should also be flexible to each situation and problem. Case studies are good for this, get them to solve a problem you have looked at recently, if they present a better solution that you… hire them!  But if you expect rote answers from Gamma, et al, you probably won’t find the right people.

Have them evaluate two different technologies they have used and look for characteristics they value (I look for simplicity and supportability as features). Can they see the benefits of the technology? Can they see why the grass isn’t always greener?

Ask about prioritization and project management (agile) skills. This is a skill that can be taught but a person who doesn’t understand that they don’t have to say yes to every request is going to be problematic.

Look for the edges of their knowledge. Look for animation and the things that excite them. After speaking to a candidate about the delegated authentication process for Hadoop (very niche), I pushed for her to be hired despite being pretty junior. She was a great hire.  A candidate needs to be curious about what we do and how we do it. Even if this knowledge has little actual value, the pursuit of it is key.  Intellectual curiosity is a key characteristic for rapid progression.

Who should do the interviewing?

Technical interviewers are a must. This is an investment so choose your best engineers and make it clear that they are to weed out anyone not as good as them.  Show them what a good interview looks like: a search for what someone is capable of rather than what they can do now. Ensure they know how to get candidates to be comfortable and confident that they are in a safe, trusted space.

So, what makes someone a great engineer?

I can sum it up in a single word:  Curiosity. Do they enjoy data and software? Find someone who just can’t shake that problem in their head, and are willing learn new things just to solve it.  Those are the kind of people we look for at bigspark.