Logic Software Solutions

About the Role
Loghic Hire is seeking a&nbsp;Senior Data Engineer&nbsp;with deep, hands-on expertise in&nbsp;Azure Databricks&nbsp;and&nbsp;AWS&nbsp;to design, build, and operate modern cloud data platforms for our enterprise clients. You will own the end-to-end data lifecycle — from ingestion and transformation to serving and governance — and act as a technical anchor for data engineering across the organization.
This is a&nbsp;build-and-lead&nbsp;role. You will write production-grade&nbsp;PySpark / SQL / Python&nbsp;code, architect&nbsp;medallion (bronze/silver/gold) pipelines, tune&nbsp;Databricks clusters and Delta Lake workloads, and integrate&nbsp;AWS-native services&nbsp;where they add value. You will also mentor engineers, define standards, and partner with analytics, data science, and platform teams.
You will work in a&nbsp;hybrid model based in Lisbon, collaborating on-site with the team and client stakeholders while retaining flexibility for remote work.
Key Responsibilities1. Data Platform Architecture &amp; Design
<ul>
<li>
Design and deliver&nbsp;cloud-native data platforms&nbsp;on&nbsp;Azure Databricks&nbsp;and&nbsp;AWS, covering ingestion, storage, processing, serving, and governance layers.
</li>
<li>
Architect&nbsp;medallion architectures (Bronze / Silver / Gold)&nbsp;using&nbsp;Delta Lake&nbsp;with ACID transactions, time travel, and schema evolution.
</li>
<li>
Define&nbsp;lakehouse vs. warehouse boundaries&nbsp;and integrate with&nbsp;Azure Synapse, Snowflake, Redshift, or Athena&nbsp;as appropriate.
</li>
<li>
Design&nbsp;batch, micro-batch, and streaming&nbsp;pipelines using&nbsp;Spark Structured Streaming, Kafka, Event Hubs, or Kinesis.
</li>
<li>
Establish&nbsp;data modeling standards&nbsp;— dimensional, Data Vault, and wide-table patterns — fit for analytics and ML consumption.
</li>
<li>
Produce&nbsp;architecture decision records (ADRs), design documents, and reference architectures.
</li>
<li>
Evaluate and recommend&nbsp;build-vs-buy&nbsp;decisions across the Azure and AWS data ecosystems.
</li>
</ul>
2. Data Ingestion &amp; Integration
<ul>
<li>
Build&nbsp;scalable ingestion frameworks&nbsp;from relational databases, SaaS APIs, event streams, files, and third-party sources.
</li>
<li>
Implement&nbsp;CDC (Change Data Capture)&nbsp;patterns using&nbsp;Debezium, Azure Data Factory, AWS DMS, or Databricks Lakeflow Connect.
</li>
<li>
Orchestrate pipelines with&nbsp;Apache Airflow, Azure Data Factory, Databricks Workflows, or AWS Step Functions.
</li>
<li>
Integrate&nbsp;on-premises and hybrid sources&nbsp;into cloud data platforms securely and reliably.
</li>
<li>
Own&nbsp;idempotency, replayability, and backfill strategies&nbsp;for all ingestion paths.
</li>
<li>
Implement&nbsp;data contracts&nbsp;between producers and consumers to prevent upstream breakage.
</li>
</ul>
3. Transformation &amp; Processing
<ul>
<li>
Write&nbsp;production-grade PySpark, Spark SQL, and Python&nbsp;for large-scale transformations.
</li>
<li>
Optimize&nbsp;Databricks jobs and clusters&nbsp;— partitioning, Z-ordering, liquid clustering, photon, autoscaling, and job clusters vs. all-purpose clusters.
</li>
<li>
Tune&nbsp;Spark performance: shuffle, skew handling, broadcast joins, caching, and AQE.
</li>
<li>
Build&nbsp;reusable transformation frameworks&nbsp;and libraries that enforce standards across teams.
</li>
<li>
Implement&nbsp;Delta Live Tables (DLT) / Lakeflow Declarative Pipelines&nbsp;for declarative, testable pipelines.
</li>
<li>
Apply&nbsp;data quality checks&nbsp;using&nbsp;Great Expectations, Soda, or DLT expectations, with quarantine and alerting paths.
</li>
<li>
Manage&nbsp;schema evolution, versioning, and backward compatibility&nbsp;rigorously.
</li>
</ul>
4. AWS &amp; Azure Cloud Engineering
<ul>
<li>
Provision and manage data infrastructure with&nbsp;Terraform&nbsp;(and/or Bicep / CloudFormation) as code.
</li>
<li>
Work hands-on with&nbsp;AWS services: S3, Glue, Athena, Redshift, Kinesis, Firehose, Lambda, Step Functions, EMR, IAM, KMS.
</li>
<li>
Work hands-on with&nbsp;Azure services: Data Lake Storage Gen2, Azure Data Factory, Synapse, Event Hubs, Key Vault, Entra ID, Databricks workspace administration.
</li>
<li>
Design&nbsp;cost-efficient storage and compute&nbsp;strategies — lifecycle policies, tiering, spot/preemptible instances, and cluster policies.
</li>
<li>
Implement&nbsp;networking and security&nbsp;for data platforms: private endpoints, VNet/VPC injection, NSGs/security groups, and firewalls.
</li>
<li>
Own&nbsp;IAM / RBAC / Unity Catalog&nbsp;access models across both clouds.
</li>
</ul>
5. Data Governance, Security &amp; Compliance
<ul>
<li>
Implement&nbsp;Unity Catalog&nbsp;for fine-grained access control, lineage, and auditing on Databricks.
</li>
<li>
Integrate with&nbsp;AWS Lake Formation&nbsp;and&nbsp;Azure Purview / Microsoft Purview&nbsp;for cataloging and lineage.
</li>
<li>
Enforce&nbsp;data classification, masking, tokenization, and encryption&nbsp;at rest and in transit.
</li>
<li>
Ensure compliance with&nbsp;GDPR&nbsp;and other applicable regulations, including&nbsp;right-to-erasure&nbsp;and data residency.
</li>
<li>
Establish&nbsp;audit logging, monitoring, and alerting&nbsp;for sensitive data access.
</li>
<li>
Partner with security and compliance teams on&nbsp;control evidence and audits.
</li>
</ul>
6. Reliability, Observability &amp; Operations
<ul>
<li>
Define&nbsp;SLAs / SLOs&nbsp;for data freshness, completeness, and availability.
</li>
<li>
Instrument pipelines with&nbsp;metrics, logs, and traces&nbsp;(OpenTelemetry, CloudWatch, Azure Monitor, Datadog, Grafana).
</li>
<li>
Build&nbsp;data observability&nbsp;using tools like&nbsp;Monte Carlo, Elementary, or custom frameworks.
</li>
<li>
Lead&nbsp;root-cause analysis&nbsp;for pipeline failures and drive permanent fixes.
</li>
<li>
Implement&nbsp;CI/CD for data pipelines&nbsp;— Git-based workflows, automated testing, and environment promotion.
</li>
<li>
Own&nbsp;disaster recovery and business continuity&nbsp;for critical data assets.
</li>
<li>
Participate in an&nbsp;on-call / escalation rotation&nbsp;for production data platforms.
</li>
</ul>
7. Technical Leadership &amp; Collaboration
<ul>
<li>
Mentor and coach&nbsp;data engineers — code reviews, design reviews, and pairing.
</li>
<li>
Set and enforce&nbsp;coding, testing, and documentation standards.
</li>
<li>
Partner with&nbsp;data scientists, analysts, and product teams&nbsp;to translate requirements into robust pipelines.
</li>
<li>
Represent Loghic Hire in&nbsp;client architecture and delivery reviews.
</li>
<li>
Contribute to&nbsp;internal accelerators, templates, and best-practice playbooks.
</li>
<li>
Help&nbsp;interview, onboard, and grow&nbsp;the data engineering practice.
</li>
</ul>
Tech StackCategoryTechnologiesCloud PlatformsMicrosoft Azure, AWSData PlatformsAzure Databricks, Delta Lake, Delta Live Tables (Lakeflow), Unity CatalogCompute &amp; ProcessingApache Spark, PySpark, Spark SQL, Spark Structured Streaming, Databricks WorkflowsProgrammingPython, SQL, Scala (nice to have), BashOrchestrationApache Airflow, Azure Data Factory, Databricks Workflows, AWS Step FunctionsStreaming &amp; MessagingApache Kafka, Azure Event Hubs, AWS Kinesis, AWS FirehoseStorageADLS Gen2, Amazon S3, Parquet, Delta, Avro, ORCAWS ServicesS3, Glue, Athena, Redshift, EMR, Lambda, DMS, IAM, KMS, Lake FormationAzure ServicesData Lake Storage Gen2, Synapse, Event Hubs, Key Vault, Entra ID, PurviewWarehouses / QuerySnowflake, Redshift, Synapse, Athena, Databricks SQLData QualityGreat Expectations, Soda, DLT Expectations, ElementaryGovernance &amp; CatalogUnity Catalog, AWS Lake Formation, Microsoft PurviewIaC &amp; DevOpsTerraform, Bicep, CloudFormation, GitLab CI / GitHub Actions, Azure DevOpsObservabilityCloudWatch, Azure Monitor, Datadog, Grafana, OpenTelemetrySecurityIAM, RBAC, KMS, Key Vault, encryption, masking, tokenizationCDC &amp; IntegrationDebezium, AWS DMS, Azure Data Factory, Lakeflow ConnectRequirements
<ul>
<li>
10+ years&nbsp;in data engineering, with at least&nbsp;5+ years&nbsp;building production data platforms on&nbsp;Azure Databricks&nbsp;and&nbsp;AWS.
</li>
<li>
Deep, hands-on expertise in&nbsp;Apache Spark / PySpark&nbsp;and&nbsp;Delta Lake, including performance tuning at scale.
</li>
<li>
Strong&nbsp;SQL&nbsp;and&nbsp;Python&nbsp;skills; experience writing production-grade, tested, maintainable code.
</li>
<li>
Proven experience designing and operating&nbsp;medallion / lakehouse architectures&nbsp;end to end.
</li>
<li>
Hands-on with&nbsp;AWS data services&nbsp;(S3, Glue, Athena, Redshift, Kinesis, IAM) and&nbsp;Azure data services&nbsp;(ADLS Gen2, ADF, Synapse, Event Hubs, Key Vault).
</li>
<li>
Strong&nbsp;orchestration&nbsp;experience (Airflow, ADF, Databricks Workflows, or Step Functions).
</li>
<li>
Experience with&nbsp;CI/CD for data pipelines&nbsp;and&nbsp;Infrastructure as Code&nbsp;(Terraform preferred).
</li>
<li>
Solid grasp of&nbsp;data governance, security, and compliance&nbsp;in regulated environments (GDPR, PCI, SOC 2).
</li>
<li>
Demonstrated&nbsp;technical leadership&nbsp;— mentoring engineers, setting standards, and driving architecture.
</li>
<li>
Excellent&nbsp;communication skills&nbsp;in English; able to work with technical and non-technical stakeholders.
</li>
</ul>
Nice to Have
<ul>
<li>
Databricks Certified Data Engineer Professional&nbsp;or Associate
</li>
<li>
AWS Certified Data Analytics — Specialty&nbsp;or&nbsp;AWS Solutions Architect
</li>
<li>
Microsoft Certified: Azure Data Engineer Associate
</li>
<li>
Experience with&nbsp;Snowflake&nbsp;or&nbsp;Microsoft Fabric
</li>
<li>
Exposure to&nbsp;MLOps / feature stores&nbsp;(MLflow, Feast, Databricks Feature Store)
</li>
<li>
Knowledge of&nbsp;data mesh&nbsp;principles and domain-oriented data ownership
</li>
<li>
Experience with&nbsp;Terraform modules&nbsp;for data platform provisioning
</li>
<li>
Familiarity with&nbsp;Kafka / Confluent&nbsp;in production
</li>
<li>
Prior consulting or client-facing delivery experience
</li>
</ul>
Other Requirements
<ul>
<li>
Hybrid role based in Lisbon, Portugal&nbsp;— expected on-site presence for team collaboration and client workshops, with flexibility for remote work.
</li>
<li>
Must be&nbsp;eligible to work in Portugal / the EU.
</li>
<li>
Occasional travel to client sites within Europe may be required.
</li>
<li>
Participation in an&nbsp;on-call / escalation rotation&nbsp;for production data platforms.
</li>
<li>
Background screening&nbsp;may be required depending on client engagement.
</li>
</ul>
What We Offer
<ul>
<li>
Competitive compensation aligned with seniority and market
</li>
<li>
Hybrid working model based in Lisbon
</li>
<li>
Opportunity to work on enterprise-scale cloud data platforms across Azure and AWS
</li>
<li>
Access to certifications and continuous learning
</li>
<li>
A collaborative, engineering-led culture with real technical ownership
</li>
<li>
Exposure to financial services and other regulated industries
</li>
</ul>