Stephen Matthew

Here's how to run the hiring process. Start with describing the ideal profile - skills, experience, personality and other essential attributes. Then design the interview funnel to test for each of these, making sure that you have rubrics for each. The process should be quick, repeatable and data-driven but allowing for gut instincts. Overall it should be a great experience for candidates, who should become recommenders of your company and team even if ultimately unsuccessful.

Successfully resolving conflict begins before it arises with setting the right ground conditions ahead of time. Here's an example. If you find that your team is often in major disagreement over design to the detriment of delivery then pivot the culture to one that falls in love with the problem and not the solution. Focussing on outcome makes resolving design conflicts easier to achieve. On implementation, encourage strong but lightly held opinions. People can disagree but they should be able to commit to the outcome.

As many contributors have already suggested, data quality checks should be implemented in-situ of the pipeline. As a way of thinking about the data quality checks to implement and when: Start with foundational checks with the highest ROI. For example column level checks on: * Non-emptiness * Non-duplicates * Uniqueness * Formatting (for emails, dates etc) Then move to more nuanced checks (but with higher probability of false-positives): - Semantic validations (e.g. prices greater than 0 and less than $100) - Day-over-day checks on expected record counts. - Week-over-week row count checks - Distribution checks (e.g. customer demographics) - Business Rules (asserting required relationships between entities)

As with any testing effort, given we live in a time-constrained world, start implementing test coverage that gives the biggest-bang for your buck. For example, if your pipeline ends in data being served via an API then test the data being returned from the API. Subsequent areas of focus would be guided by a rubric that considers factors such as pipeline stages with the highest complexity (and therefore risk) and tests with the lowest chance of false-positives.

Overcome any reluctance you might have reaching out to prospective companies. If you show genuine interest in their business and lead with what you can offer then many managers are open to an intro chat. As a personal reflection this has absolutely been my experience within climatetech. A few useful tips: - Find an advocate in the target company within your near network. This person may not be in the target team but they can help make that connection on your behalf. - Demonstrate genuine interest. Ask a relevant question or even better suggest a product or tech improvement if these are visible to you. - Be a "giver" and not just a "taker" - highlight what you can offer and even how you could pay-it-forward in the long run.

Fundamental to growing you career is developing your pattern recognition, ability to make appropriate trade-offs, troubleshoot and find or ask for the right information. All these can be facilitated by the experience of working on real projects. Almost as good is taking a tutorial or blog article and adapting it to a different data stack. In this way you use a real world example to up-skill on new approaches to integrate, store or serve data.