GCP Data Engineer

Job Summary:

We are seeking a highly skilled and experienced GCP (Google Cloud Platform) Data Engineer with more than 6 years of expertise to join our dynamic team. The GCP Data Engineer will play a pivotal role in designing, implementing, and optimizing data solutions on Google Cloud Platform. The ideal candidate should have a deep understanding of GCP services, strong data engineering skills, and a proven track record of delivering robust and scalable data solutions.

Key Responsibilities:

• Architecture Design: Design and implement scalable and efficient data architectures on Google Cloud Platform (GCP) to meet business requirements and performance goals.
• Data Integration: Integrate data from various sources into GCP using services like Cloud Storage, Cloud Pub/Sub, Dataflow, and BigQuery.
• Data Processing: Develop and implement data processing pipelines using Apache Beam, Apache Spark, or other suitable frameworks to transform and cleanse data for analysis.
• Data Modeling: Design and implement data models for optimal performance and efficiency in BigQuery or other GCP data storage solutions.
• Streaming Analytics: Implement real-time data processing and analytics solutions using GCP streaming services such as Cloud Dataflow and Cloud Pub/Sub.
• Data Security and Governance: Ensure data security and compliance with regulatory requirements by implementing appropriate access controls, encryption mechanisms, and data governance policies.
• Performance Optimization: Optimize data pipelines, queries, and data storage configurations to maximize performance and minimize costs.
• Monitoring and Logging: Set up monitoring and logging solutions to track system performance, detect anomalies, and troubleshoot issues in GCP data environments.
• Automation and DevOps: Implement automation scripts and workflows using tools like Terraform, Deployment Manager, or Google Cloud Deployment Manager to automate provisioning and management of GCP resources.
• Collaboration: Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements and deliver data solutions that meet business needs.
• Documentation and Knowledge Sharing: Create and maintain technical documentation, best practices, and guidelines for data engineering on GCP. Share knowledge and mentor junior team members.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Minimum of 6 years of experience in data engineering, with at least 3 years of hands-on experience working with Google Cloud Platform.
• Strong understanding of GCP services such as BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Dataproc.
• Proficiency in programming languages such as Python, Java, or Scala for data processing and scripting tasks.
• Experience with data warehouse design, implementation, and optimization.
• Knowledge of data streaming and real-time analytics technologies.
• Familiarity with machine learning concepts and frameworks (e.g., TensorFlow, scikit-learn) is a plus.
• Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment.