Lead Data Engineer

Company: Together Labs
Location: Redwood City
Posted on: February 21, 2021

Job Description:

The Data and Analytics team is focused on creating competitive advantage for Together Labs and our customers through novel data infrastructure, metrics, insights and data services.--We are a small but rapidly growing analysis and engineering team that builds and leverages state-of-the-art analytics systems. Our work informs decisions and direction for our business, while also impacting our products. The petabytes of data we store and the hundreds of GB of transactional data that we process each day provide a great opportunity to apply machine learning techniques to design powerful and meaningful experiences to our users.We are looking for a seasoned Lead engineer to not only build data pipelines to efficiently and reliably move data across systems, but also to build the next generation of data tools to enable us to take full advantage of this data. In this role, your work will broadly influence the company's products, data consumers and data scientists. The Role:* Architect and implement best practices of big data tools such as Hadoop, Spark, Airflow, Kafka, Presto and Time-series Databases * Set processes and policies for data governance and data pipeline * Set and define the standards and best practices in data team * Be the point of reference for solving a challenging technical problem* Architect and implement Machine Learning Pipelines for Data Science team* Build and expand IMVU's Data Platform's Hadoop Open source Big data Clusters within an agile collaboration environment* Observe the thriving open source projects in the descriptive and predictive analytics big data space and bring it into IMVU* Must love coding - prepare to spend 80% of the time on hands-on development with development teams* Deliver end-to-end applications and frameworks ranging from system programming to micro-services to simple front-end applications* Security-first mindset to ensure that any software built will satisfy Visa's stringent security standards* Ability to own complete project or a subject area delivery while leading team members in a scrum setting* Use your knowledge of Java, Go or Python to create new features in the cloud.* Understand and influence logging to support our Data flow.Qualifications:* Bachelor's or Master's--degree in Computer Science, Engineering or related discipline * 10+ years of experience in design and development of large-scale data platforms * 4+ years of software development experience in Scala or Java.* Expert in tools such as Spark, Airflow, Presto * Expert in design and implement reliable, scalable, and performant distributed systems and data pipelines * Expert in containerization, including Docker and Kubernetes * Experience with running large-scale distributed computing infrastructure such as load balancing, Zookeeper, Micro service architecture * Experience with Columnar database such as Redshift, Vertica or Parquet* Hand-on experience in optimizing existing Big Data applications written in Spark, Hive. Optimizing * Experience in building Open source projects, contributed to open source projects and deploying into containers platform * Good understanding of building utilization frameworks on Hadoop, AWS, Azure or GCP (Restful services, Self-service BI and Analytics).* Experience in one or more analytical modeling tools/workbench (Jupyter or Zeppelin)* Worked on AWS services ---Lambda, S3, Redshift, SNS (or Apache Kafka), SQS, Redis, Data-Lakes, and AWS cloud deployment models* Strong communication skills with ability to present complex ideas and document in a clear and concise way* Quick learner; self-starter, detailed and thorough* Proactive sense of urgency and "can do" attitude* A technical leader: you make intuitive decisions about what services, frameworks, and capabilities need to be in place before they are needed.* Self Driven - able to own a project from inception to completion* Mentor - open and active in sharing knowledge as well as excellent communication skillsBonus Points:* Experience with Free-to-play/social games* AWS certified: Architect/DevOps/Big Data

Keywords: Together Labs, Redwood City , Lead Data Engineer, Engineering , Redwood City, California