About this position
Work on a challenging mobility project, in which real time data is crucial to give the best service to the traveler and valuable data insights to the operators.
Together with the data engineering team, you'll provide your real time expertise to ingest data, manage messages from different data providers and consolidate everything into the common data lake, all of this in a highly scalable Kubernetes infrastructure.
Key responsibilities
- Design and implement real-time data processing solutions using Kafka and Spark Streaming (and possibly Flink)
- Collaborate with cross-functional teams to ensure the successful integration of real-time data solutions into our products and services
- Optimize the performance and scalability of real-time data systems
- Maintain and troubleshoot Kafka clusters and Spark Streaming applications
- Keep up-to-date on the latest trends and developments in the field of real-time data processing
Requirements
-
University degree in IT or similar
- Level of English (B2 or higher)
-
1+ years of experience in implementation of Kafka clusters and real-time analytics
-
1+ years of experience in Spark Streaming
-
2+ years of experience in Python
-
Strong data manipulation skills (specially SQL, Spark SQL and the Dataframe API)
- Ability to design, develop and maintain real-time data processing solutions
- Excellent problem-solving skills and ability to make informed decisions in real time
- Good knowledge of Git
- Good communication skills
- Results faced, self-challenging mentality
- Eager to learn
Nice to have
-
Experience with Apache Flink
- Experience with other data-relate Apache products
-
Experience with Kubernetes clusters or Rancher
-
Experience with Delta Lake
-
Experience with Spark in general and the Hadoop ecosystem
For more information feel free to contact us: hr@nucleoo.com
