Data Engineer / Data Architect – Databricks & Unity Catalog

Dear Consultant,

We are collaborating with a leading global professional services organization, recognized as one of the "Big Four" accounting firms.

Our client specializes in providing audit, consulting, tax, and financial advisory services across various industries. They are seeking a skilled Data Engineer/Data Architect with expertise in Databricks and Unity Catalog to design, implement, and manage scalable data solutions. If you match the required skills, send CV to marcillina.tietjen@dcvtechnologies.co.uk

Key Responsibilities:

• Design and implement scalable data pipelines and ETL workflows using Databricks.
• Implement Unity Catalog for data governance, access control, and metadata management across multiple workspaces.
• Develop Delta Lake architectures for optimized data storage and retrieval.
• Establish best practices for data security, compliance, and lineage tracking in Unity Catalog.
• Optimize data lakehouse architecture for performance and cost efficiency.
• Collaborate with data scientists, engineers, and business teams to support analytical workloads.
• Monitor and troubleshoot Databricks clusters, performing performance tuning and cost management.
• Implement data quality frameworks and observability solutions to maintain high data integrity.
• Work with Azure/AWS/GCP cloud environments to deploy and manage data solutions.

Required Skills & Qualifications:

• 8-19 years of experience in data engineering, data architecture, or cloud data solutions.
• Strong hands-on experience with Databricks and Unity Catalog.
• Expertise in PySpark, Scala, or SQL for data processing.
• Deep understanding of Delta Lake, Lakehouse architecture, and data partitioning strategies.
• Experience with RBAC, ABAC, and access control mechanisms within Unity Catalog.
• Knowledge of data governance and compliance standards (e.g., GDPR, HIPAA) and audit logging.
• Familiarity with cloud platforms (Azure, AWS, or GCP) and their respective data services.
• Strong understanding of CI/CD pipelines, DevOps, and Infrastructure as Code (IaC).
• Experience integrating BI tools (Tableau, Power BI, Looker) and ML frameworks is a plus.
• Excellent problem-solving, communication, and collaboration skills.

If you are a proactive and experienced data professional looking to contribute to innovative projects in a dynamic environment, we encourage you to apply.