Responsibilities :
- Building robust large-scale social media crawler for various social media platforms
- Maintain day-to-day operation of said crawlers alongside our data quality team to make sure the completeness and timeliness of incoming data
- Work in team along with several stakeholders to better understand and communicate about data requirements and issues
Qualifications :
- Intermediate or advanced understanding of SQL
- Have proficiency in several programming languages, mainly in Javascript (Node) and Python. Having a good experience in using Scala for production (not related to Spark) is a big plus
- Have intermediate or advanced understanding of several data-related technologies such as MariaDB, Postgres, Redis, Kafka, and Elasticsearch
- Have ample experience using Linux-based system
- Have understanding on several data acquisition techniques such as webscraping, API-based data consumption, and so on
