- Salary: up to £80,000
- Location: Newcastle, on site 3 days a week
- ****Active SC required****
Our client is a multi-award-winning digital consultancy that builds and runs services for UK government departments, the sort of platforms tens of millions of people end up using without ever thinking about it. This role sits hands-on in their data engineering team, building the AWS pipelines that carry that work.
You'll write Python and PySpark for Spark workloads on Amazon EMR, orchestrate them through Apache Airflow DAGs, and provision the whole environment as Infrastructure as Code (IaC) with Terraform, containerised in Docker and shipped through GitLab CI/CD.
Non-Negotiables
- Python: primary language for pipeline development and processing
- PySpark and Apache Spark: distributed processing on AWS EMR
- SQL: querying, transformation and validation across data stores
- Apache Airflow: writing DAGs and orchestrating tasks at scale
- AWS EMR: running Spark workloads and reading output logs
- Terraform and Infrastructure as Code (IaC) on AWS
- Docker: containerising data engineering solutions
- GitLab CI/CD: pipeline config, release tagging, component versioning
- Dimensional modelling and slowly changing dimensions (SCD Types 1, 2, 3)
- ****Active SC****
What You'll Work With
- Amazon Athena: ad hoc querying over S3
- AWS S3, IAM, CloudWatch, EC2 and ECR day to day
- Jupyter Notebooks: exploratory analysis and prototyping
- Server-side and client-side encryption patterns in AWS
- Root cause analysis across pipeline failures and data quality issues
- Data governance standards in a regulated government setting
- You'll turn customer expectations into working technical solutions
Nice to Haves
- Amazon Textract: document extraction
- Amazon Comprehend: natural language processing
- AI and ML services applied to document-heavy datasets
- Public sector or other regulated delivery experience
Why Join / Projects
The data is public service data, so it matters and it gets used. You'll see your pipelines carrying real caseloads rather than sitting in a proof of concept, and you'll move across departments and problems instead of maintaining one warehouse for years.
You own your pipelines end to end. When something breaks you're in the EMR logs and CloudWatch metrics working out why, not raising a ticket for someone else. Infrastructure goes in as Terraform, deployments go out through GitLab, and the engineering standards are yours to push on where they need tightening.
Employee Benefits
- Salary: up to £80,000
- Private health cover
- Pension scheme
- Rewards Gateway discounts and perks platform
- 25 days holiday
- Sponsorship through Security Check (SC) clearance
AWS Data Engineer | Python, PySpark, Airflow, AWS EMR, Terraform | Award-Winning Public Services Consultancy | ****Active SC Clearence required****
