Data Platform Engineer

Dallas, Texas
(View all jobs)

Function:

DS IoT

Requisition ID:

1035802

Our Company

We’re Hitachi Digital Services, a global digital solutions and transformation business with a bold vision of our world’s potential. We’re people-centric and here to power good. Every day, we future-proof urban spaces, conserve natural resources, protect rainforests, and save lives. This is a world where innovation, technology, and deep expertise come together to take our company and customers from what’s now to what’s next. We make it happen through the power of acceleration.

Imagine the sheer breadth of talent it takes to bring a better tomorrow closer to today. We don’t expect you to ‘fit’ every requirement – your life experience, character, perspective, and passion for achieving great things in the world are equally as important to us.

The team

You will join a high-performing Data Engineering & Platform team responsible for building and scaling the organization’s cloud-based data ecosystem. The team treats data as a strategic product and partners closely with engineering, analytics, and business stakeholders to power enterprise-wide insights and innovation. Operating at the intersection of data engineering, DevOps, and cloud architecture, the team focuses on building modern, secure, and scalable solutions using Databricks and AWS.

The role

As a Data Platform Engineer, you will lead the design, development, and optimization of our large-scale, cloud-native data platform. You will architect and build robust ETL/ELT pipelines using PySpark and Databricks, leveraging Delta Lake, Unity Catalog, and Delta Live Tables. You will own DevOps automation through GitHub Actions and ensure fast, reliable deployments of Databricks assets. You will also manage the AWS infrastructure supporting the platform—focusing on secure, scalable, and high-performing environments. This role requires deep expertise in distributed data processing, Databricks engineering, CI/CD automation, and cloud infrastructure.

Key Areas of Ownership:

Build and maintain high-scale ETL/ELT pipelines across diverse data sources
Implement and optimize Databricks workflows using PySpark, Python, DLT, and Unity Catalog
Configure and manage AWS environments including VPCs, IAM, S3, and secure connectivity
Establish CI/CD pipelines using GitHub Actions for automated deployment of Databricks notebooks, jobs, and pipelines
Drive data quality via automated testing frameworks (unit, integration, performance)
Optimize cluster performance and cost efficiency
Lead best practices in Medallion Architecture, ACID data principles, and high-performance SQL
Create clear technical documentation, architecture diagrams, and design specifications

What you’ll bring

Required Qualifications

5+ years of experience building scalable Data Engineering platforms and production-grade pipelines
3+ years of hands-on Databricks development, including expertise in:
- Delta Lake (ACID, time travel, optimization)
- Unity Catalog (security, governance, metadata)
- Delta Live Tables (DLT)
- Workspaces, Repos, Jobs, and Databricks SQL
3+ years of AWS experience, including:
- VPC, Subnets, Endpoints, Routing
- IAM roles, policies, cross-account access
- S3-based data lake implementation
Expert programming skills in Python (4+ years)
Deep hands-on experience with PySpark and advanced SQL
Proven CI/CD experience using GitHub Actions or similar tools
Strong understanding of ETL/ELT, Data Lake, Data Warehouse, and distributed computing concepts
Agile (Scrum) experience and Git proficiency

Preferred Qualifications

Experience with AWS data services such as Glue, Athena, Redshift, RDS, DynamoDB
Knowledge of real-time streaming (Kafka, Spark Structured Streaming)
Experience building ML feature pipelines
Background in performance tuning and capacity planning for large Spark clusters

About us

We’re a global, team of innovators. Together, we harness engineering excellence and passion to co-create meaningful solutions to complex challenges. We turn organizations into data-driven leaders that can make a positive impact on their industries and society. If you believe that innovation can bring a better tomorrow closer to today, this is the place for you.

#LI-RS2

Fostering innovation through diverse perspectives

Hitachi is a global company operating across a wide range of industries and regions. One of the things that sets Hitachi apart is the diversity of our business and people, which drives our innovation and growth.

We are committed to building an inclusive culture based on mutual respect and merit-based systems. We believe that when people feel valued, heard, and safe to express themselves, they do their best work.

How we look after you

We help take care of your today and tomorrow with industry-leading benefits, support, and services that look after your holistic health and wellbeing. We’re also champions of life balance and offer flexible arrangements that work for you (role and location dependent). We’re always looking for new ways of working that bring out our best, which leads to unexpected ideas. So here, you’ll experience a sense of belonging, and discover autonomy, freedom, and ownership as you work alongside talented people you enjoy sharing knowledge with.

We’re proud to say we’re an equal opportunity employer and welcome all applicants for employment without attention to race, colour, religion, sex, sexual orientation, gender identity, national origin, veteran, age, disability status or any other protected characteristic. Should you need reasonable accommodations during the recruitment process, please let us know so that we can do our best to set you up for success.