Get the app Chrome extension Freelance ideasGigsSign inFind my gigs

ProfilePush / H1B jobs / Sr Data Engineer (Azure Databricks)

Sr Data Engineer (Azure Databricks)

Insight Global · Woonsocket, RI · contract · posted 19 days ago

$71 an hourpay
Woonsocket, RIlocation
contractjob type
5+ yearsexperience

Apply on Insight Global's siteCheck my AI Match score

Skills

About the job

Required Skills & Experience

• 5-7+ years of Data Engineering experience building large-scale data platforms

• Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)

• Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning

• Experience developing and maintaining Delta Live Tables (DLT) pipelines

• Strong SQL background with experience supporting legacy data warehouse environments

• Experience migrating SQL-based pipelines to modern Python/PySpark architectures

• Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies

• Proficiency with YAML, JSON, and configuration-driven development patterns

• Experience automating CI/CD deployments using DevOps pipelines

• Strong understanding of Medallion Architecture (Bronze, Silver, Gold)

• Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers

• Experience troubleshooting, monitoring, and supporting production data pipelines

• Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred

• Healthcare industry experience preferred

• Medicare, payer, member enrollment, or claims data experience strongly preferred

Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

Nice to Have Skills & Experience

*Important* Core Databricks concepts we rely on daily:

We need candidates with solid data engineering fundamentals and proven hands-on Databricks experience, specifically across: • Performance tuning & optimization (Spark internals, join strategies, skew handling) • Efficient use of LDP APIs (CDF, CDC) • Data quality frameworks (expectations, quarantine logic, alerting) • Compliance & governance (PII/PHI masking, access controls) and lastly on Orchestrating and stiching these pipelines together

Job Description

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables

• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions

• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns

• Develop and maintain CI/CD deployment processes for data engineering workflows

• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers

• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability

• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments

• Enhance monitoring, observability, and operational support processes

• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets

• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability

More Insight Global jobs this week

Sr. DevOps & Microsoft AI Solutions EngineerInsight Global · Portland, OR · contract · today
View
Sr. Power Platform EngineerInsight Global · Portland, OR · contract · today
View
Technical Support Admin IInsight Global · Lower Nazareth, PA · contract · yesterday
View
Sr. Applications EngineerInsight Global · Hillsboro, OR · full-time · yesterday
View
ASIC Subsystem EngineerInsight Global · El Segundo, CA · contract · yesterday
View
QA Engineering AdvisorInsight Global · Hunter Mill, VA · contract · yesterday
View

Questions

Does Insight Global sponsor H1B for this job?

Insight Global is on record as an H1B sponsor. The posting doesn't say it won't sponsor. A past filing doesn't guarantee sponsorship for a particular role, so confirm with Insight Global.

How do I apply for this Sr Data Engineer (Azure Databricks) job?

Apply on Insight Global's own careers page with the Apply button. To see how well your resume fits first, ProfilePush's AI Match scores it 0 to 100 for free.

How well do you fit this job?

Upload your resume and AI scores this job, and every sponsor job like it, 0 to 100. Free to start.

Get my match score