Executive Summary

Bayleys Real Estate is one of New Zealand’s largest full-service real estate companies, operating across commercial, industrial, residential and rural property. Its valuers rely on an internal Commercial Valuation Evidence (CVE) application to record, search and report on the sales and rental evidence behind every commercial valuation. 

Bayleys partnered with Peritos Solutions to replace the ageing Access-based system with a modern platform that valuers could use from any device. Peritos designed and delivered a fully serverless AWS application in under two months. Valuers can now search evidence on a map, attach photos and documents, and export reports to Word, Excel and PDF, all through their existing Bayleys login.

“The team was quick to respond, very cooperative, and genuinely focused on delivering quality results.”

Biswa Mishra, Director, Wizmonk (Peritos AWS client)

Project outcomes

Results & Impact

> 99%
Up time
Platform availability
120+hrs/mo
Hrs Saved
Time retur
200+ User
Cost Saved

Background

The CVE application had run on a Microsoft Access database for many years. It was a standalone, desktop-bound tool with no cloud access, no modern search and no ability to scale with a growing team of valuers. 

Bayleys needed the application moved to a cloud-native platform that could be accessed from anywhere, aligned with its existing Azure AD identity and roles, and gave valuers a much better experience for entering and finding property evidence.

Challenge

Moving years of property data to the cloud was far more than a lift-and-shift. The Access database had an unstructured schema built through years of ad hoc data entry. It had to be remapped, converted and validated record by record before go-live. 

Bayleys staff needed to sign in with their corporate Azure AD accounts, which meant integrating Microsoft identity with an AWS-native application. The new platform also had to handle specialist valuation fields such as yield, IRR and WALT exactly as valuers expect, and restrict all address searches to New Zealand. All of this had to be delivered in under two months. 

Solution

Peritos built a fully serverless platform on AWS, following the AWS Well-Architected Framework. The application uses a React.js frontend on AWS Amplify, C# APIs on AWS Lambda and an Amazon Aurora PostgreSQL database. There are no servers to manage, the platform scales automatically, and Bayleys pays only for what it uses. The total running cost is around USD $124 a month. 

Staff sign in through AWS Cognito connected to Bayleys’ Azure AD, with admin rights managed through existing Microsoft groups. Valuers can search evidence by property type, price, area, tenure or keyword, and use Google Maps radius search to find comparable evidence near any NZ address. The platform is secured with AWS WAF, Shield, KMS encryption and Secrets Manager, and every data export is captured in an audit log.

Inside the application

Approach

Peritos began with a structured assessment before any build started. The team reviewed the existing database, designed the target cloud architecture, completed a Well-Architected review across all five pillars, and validated AWS costs upfront. 

Delivery then ran as parallel workstreams across infrastructure, backend, frontend and data migration, with daily sprints and fast feedback from Bayleys valuers. The core application was built first, followed by exports, map search and audit logging. Data migration, user testing and go-live came last. All 45+ test cases passed with only three bugs raised, and a month of free hypercare followed launch before Bayleys moved to ongoing support.

“Peritos are very easy to work with and have excellent development skill. They were dedicated and committed to our project and provided great service.”

John Leeves, Director, Tonkin + Taylor (Peritos AWS client

About the Client

Client  Bayleys Real Estate 
Industry  Commercial & residential property  |  sales · valuation · property management 
Region  New Zealand  |  one of NZ’s largest full-service real estate companies 
Application  CVE — Commercial Valuation Evidence  |  internal property evidence platform 
Legacy  Microsoft Access database  →  migrated to Aurora PostgreSQL on AWS 
Delivered by  Peritos Solutions  |  Oct 2023 – Dec 2023 

Bayleys is one of New Zealand’s leading full-service real estate companies, operating across commercial, industrial, residential and rural property. Their internal CVE application is used by valuers to record, search, and report on commercial sales and rental evidence across the New Zealand market.

Project Background

Bayleys’ existing Commercial Valuation Evidence system ran on a legacy Microsoft Access database — a standalone, desktop-bound solution with no cloud access, no modern search capability, and no ability to scale to a growing user base. The requirement was to migrate the entire application to a cloud-native platform that could be accessed from any device, support role-based access aligned to Bayleys’ existing Azure AD identity, and provide a significantly improved user experience for valuers entering and searching property evidence. 

 Peritos Solutions was engaged to design, build, and deploy the full application on AWS — from architecture design and Access database migration through to a React.js frontend, serverless C# API layer, Aurora PostgreSQL database, and production deployment — in under two months.

Project Timelines

Phase  Period  Key Activities  Status 
Phase A  Oct 2023  Requirement gathering, AS IS / TO BE flows, business architecture, application design, AWS account structure setup (Dev/Test/Prod OUs)  Done 
Phase B  Oct – Nov 2023  AWS infrastructure setup, React.js frontend, C# Lambda APIs, AWS Cognito + Azure AD SSO, property search, sales entry, rental entry, photo/attachment upload, CI/CD pipelines, unit testing  Done 
Phase C  Nov – Dec 2023  Word/Excel export, Google Maps radius search, audit logs, email report delivery, advanced search enhancements  Done 
Phase D  Dec 2023  Access DB data migration to Aurora PostgreSQL, UAT, bug fixing, production deployment, documentation & handover  Done 
Hypercare  Dec 2023  1-month free hypercare post go-live, then transition to support contract  Done 

Results & Impact

Observability & Visibility

Assessment Report

Phase A delivered a structured assessment before any build commenced: 

Cloud Formation

Serverless AWS Architecture

The entire application was built on a consumption-based serverless model: 

Scope & Requirements

Phase A — Requirements & Architecture
Phase B — Core Application
Phase C — Additional Features
Phase D — Data Migration & Production

Implementation

Serverless Architecture

The backend was built entirely on AWS Lambda using C#, invoked via Amazon API Gateway. This eliminated server management entirely — Lambda scales automatically to handle any volume of concurrent requests, and the consumption-based pricing model means Bayleys pays only for actual usage. At 50,000 Lambda invocations per month, the Lambda cost is effectively zero under AWS’s free tier thresholds. 

Authentication — Azure AD + AWS Cognito

Bayleys users authenticate using their existing Bayleys corporate Azure AD credentials. AWS Cognito is configured with Azure AD as the federated identity provider — users click ‘Sign in with Bayleys account’, are redirected to Azure AD for authentication, and returned to the CVE app with a JWT token. Admin role assignment is handled via Azure AD group membership, surfaced in the JWT claim and enforced on every Lambda endpoint. 

Property Search & Google Maps Integration

The property search module allows Bayleys valuers to search commercial sales and rental evidence by property type, category, subcategory, location, date, price, building area, tenure, and keyword. NZ address lookup is powered by the Google Maps Places API with results restricted to New Zealand addresses only. The radius search feature allows valuers to find all comparable evidence within a defined kilometre radius of any NZ address — critical for commercial valuation work. 

Data Model & Migration

The existing Microsoft Access database was analysed, mapped to a normalised PostgreSQL schema, and migrated to Amazon Aurora Serverless. Aurora’s serverless configuration means the database scales its capacity units automatically based on load — dropping to near-zero cost during idle periods and scaling up for peak search and reporting activity. The database is deployed in a private subnet with no public internet access; Lambda functions access it over the private network only. 

Security Architecture

All API endpoints are protected by AWS WAF with AWS managed rule groups covering common web exploits, SQL injection, and bot traffic. AWS Shield Standard provides DDoS protection at the network layer. All sensitive configuration (database connection strings, API keys, third-party credentials) is stored in AWS Secrets Manager in encrypted format. Data at rest in Aurora and S3 is encrypted using AWS KMS. All data in transit is secured via HTTPS/TLS.

Technology & Architecture

Frontend  React.js  |  AWS Amplify  |  responsive  |  Chrome · Edge · Firefox · Safari 
Backend  C# (.NET)  |  AWS Lambda (serverless)  |  AWS API Gateway  |  event-driven architecture 
Database  Amazon Aurora Serverless (PostgreSQL)  |  99.99% SLA  |  auto-scaling  |  private subnet 
Auth  AWS Cognito  |  Azure AD as identity provider  |  JWT tokens  |  role-based access (admin / user) 
Storage  Amazon S3  |  PDFs · photos · attachments  |  private endpoint access from Lambda  |  KMS encryption 
Networking  AWS VPC  |  private subnets for Lambda & Aurora  |  AWS WAF on API Gateway  |  AWS Shield Standard 
Security  AWS WAF (managed rule groups)  |  AWS Shield Standard  |  AWS KMS  |  AWS Secrets Manager  |  SSL/TLS 
Notifications  Amazon SNS  |  Amazon SQS  |  email delivery for property reports 
Monitoring  Amazon CloudWatch  |  AWS CloudTrail  |  alert & monitoring dashboards 
DevOps  AWS CloudFormation (IaC)  |  CI/CD pipelines  |  Dev · Test · Prod environment parity 
Integrations  Google Maps API  (NZ address lookup · radius search)  |  Azure AD SSO  |  Word/Excel/PDF export 
Region  Asia Pacific — Sydney  |  monthly AWS cost: USD $124.22  |  12-month total: USD $1,490.69 
Architecture Overview

A high level architecture diagram as below

Challenges

Access Database Migration to Aurora PostgreSQL

The legacy Microsoft Access database had an unstructured schema built over many years of ad hoc property data entry. Peritos mapped the full Access data model to a normalised PostgreSQL schema before migration, handled data type conversions, and validated migrated data record-by-record against the source before go-live. 

Azure AD Federated Identity via AWS Cognito

Integrating Bayleys’ corporate Azure AD as the identity provider for an AWS-native application required careful configuration of OIDC federation in Cognito, JWT claim mapping for admin role detection, and testing across all authentication edge cases — including token expiry, re-authentication, and admin privilege escalation via group membership. 

NZ-Only Address Restriction

Google Maps Places API returns global results by default. Enforcing NZ-only address lookup required both frontend component configuration (restricting the Places autocomplete to NZ) and backend validation on address submission — ensuring no non-NZ property records could be created through direct API calls. 

Complex Valuation Analysis Fields

The sales evidence entry form includes specialist commercial valuation fields — Building Rate ($/psm), Equivalent Market Yield (%), IRR (%), WALT (years to 2dp), Vacancy (%) — each with specific unit formatting requirements and conditional display rules (e.g. subcategory only active for Commercial Office and Commercial Retail). Getting the field logic right required close collaboration with Bayleys valuers through multiple UAT cycles. 

Under Two Months End-to-End

Delivering a fully serverless custom application — including data migration, three AWS environments, CI/CD pipelines, Azure AD integration, Google Maps integration, and report export — in under two months required strict parallel workstreaming across infrastructure setup, backend development, frontend development, and data migration, with daily sprint cycles and rapid UAT feedback turnarounds. 

Key Benefits

Post-Launch Support

Hypercare Period

A 1-month free hypercare period was included from the go-live date. Peritos monitored CloudWatch metrics, resolved all post-launch issues, and provided guidance on Aurora capacity tuning and Lambda concurrency configuration. All Phase B UAT test scenarios (45+ test cases across sales entry, rental entry, and search) were confirmed passed before the hypercare period commenced, with only 3 bugs raised and resolved during testing. 

A support contract proposal was prepared for ongoing AMC support following the hypercare period, covering Lambda, Aurora, Amplify, and application-level support. 

About Client

DEEPIQ simplifies industrial analytics and offers the only self-service application for end-to-end (Data + AI) OPs for the industrial world. With this app, your SMEs can digitize their expertise, and your data teams can build and deploy robust analytic pipelines.

Project Background

DeepIQ required a proof-of-concept integration layer to extract Vendor master data and Work Order information from their SAP system and expose it to external applications. The goal was to evaluate viable integration patterns — SOAP, REST, BAPI/RFC — and deliver a working, reusable service layer that could be extended to production. 

The project was executed by Peritos Solutions under a phased delivery model, beginning with an assessment and requirement-gathering sprint before moving into build and testing phases. 

Scope & Requirements

In Scope

Out of Scope

Project Timelines

Phase  Duration  Key Deliverables  Status 
Assessment  < 2 wks  Requirement gathering, SAP system access, Risk & Gap Analysis, Customised Assessment Report  Done 
Phase B  2–3 wks  Custom BAPI design, ABAP OOP coding, WSDL web service exposure, SOAP UI unit testing  Done 
Phase C  2–3 wks  REST-based SICF service (z_ven_wrk_ords), JSON output, Postman testing, 1-week support  Done 
Hypercare  1 wk  Post-go-live support, bug fixing, documentation handover  Done 

Results & Impact

Observability & Visibility

Assessment Report

A comprehensive assessment was completed in under two weeks. This included: 

Cloud Formation

Integration — Delivered in < 2 Weeks per Phase

Each integration phase was scoped, built, and tested in a rapid sprint cadence: 

Key Outcome Metrics

✔  Observability & Visibility 

Full assessment report delivered with risk & gap analysis within 2 weeks. 

 ☁  Cloud-Ready Integration 

REST & SOAP web services exposed from SAP — ready for cloud consumption. 

 ⚡  Rapid Time-to-Value 

Custom BAPI developed & tested end-to-end in under 8 weeks. 

Implementation

Architecture Overview

Below is a high level Architecture Diagram overview


Technology & Architecture

SAP Core  SAP R/3 · S/4HANA  |  ABAP OOP  |  BAPI / RFC 
Integration  SOAP Web Services (WSDL)  |  REST / HTTP (SICF)  |  JSON 
Tooling  SOAP UI  |  Postman  |  Informatica ETL 
Data Scope  Vendor Master  |  Customer Master  |  Work Orders 
Auth  HTTP Basic Auth  |  Service User credentials 
Output Format  JSON (serialised via /ui2/cl_json)  |  XML / WSDL 


Challenges

Authentication Configuration

SAP’s web service authentication settings required careful tuning. Multiple configurations were tested before identifying the correct combination of HTTP handler and service user credentials that allowed external SOAP calls to succeed. 

Date Filter Logic

Initial work-order retrieval returned data outside the requested date range. The ABAP selection logic was corrected to enforce strict date boundary matching (GSTRP GE i_start AND GSTRP LE i_end), and output dates were reformatted to DD/MM/YYYY. 

Vendor Number Formatting

Vendor numbers required CONVERSION_EXIT_ALPHA_OUTPUT normalisation to strip leading zeros and match expected formats in the consuming application. 

Compressed Timeline

All three phases were delivered in parallel with the client’s internal scheduling constraints, requiring iterative development with 2–3 day feedback cycles and daily stand-ups. 

Key Benefits

Post-Go-Live Support

Hypercare Period

A dedicated one-week hypercare period was included post-delivery. During this period DeepIQ resolved all outstanding defects, provided guidance on SAP navigation, and completed final documentation. All six functional test scenarios and both raised bugs were closed with a Pass status before handover. 

Application screenshots are included below as placeholders — replace with actual Postman / SOAP UI test captures. 

About Client

Electric Kiwi is an autonomous online New Zealand electricity retailer. Established in 2014, Electric Kiwi uses cloud computing and intelligent meters to service customers nationwide in major urban areas. Its services depend on the presence of the smart meters in the customer’s home. For the client, we did the POC for integrating AWS Support Services with Datadog’s CloudFormation template.

Project Background

The case study of the project provides a synopsis of how we did the POC for the client for integrating AWS Support Services, an AWS account with Datadog employing Datadog’s CloudFormation template. Creating an IAM role and associated policy enables Datadog’s AWS account to make API calls for collecting or pushing data into your AWS account. Using the CloudFormation template supplies all the tools required to send this data to your Datadog account, and Datadog helps the CloudFormation template to provide the latest functionality. The template even deploys the Datadog Forwarder Lambda operation for sending logs to Datadog.

After setting the initial connection, enable the individual AWS service integrations appropriate to your AWS domain. With a single click, Datadog provisions the required resources in your AWS account and starts analyzing the metrics and events for the services you use. For general AWS services you are using, Datadog needs out-of-the-box dashboards providing prompt and customizable visibility. The project background demonstrates setting up the integration and establishing the Datadog Agent on an Amazon Linux EC2 instance and feeds a broad outline of the integration’s capabilities. Visit the Enable integrations for individual AWS service factions for a list of the general sub-integrations.

This procedure can be repeated for multiple AWS accounts as required, or you can also use the API, AWS CLI, or Terraform to set up various accounts simultaneously. For more details, read the Datadog-Amazon CloudFormation guide.

Scope & Requirement

The scope & requirements involved the following:

Implementation

Technology and Architecture

Technology/Services used

We used Datadog integration with AWS services to masure and do observability matrix and helped them to setup below

Security & Compliance:
Backup and Recovery
Network Architecture
Cost Optimization
Code Management, Deployment

Challenges

Support

About Client

Managing AWS Environment

Wine-Searcher is a web search engine that helps find the price and availability of any wine, whiskey, spirit, or beer worldwide. It has been in operation since 1999 and has offices in New Zealand and the UK. They provide an easy-to-use search engine, price comparison tools, an extensive database of wines and spirits, an encyclopedia, and news pages that aim to provide all “wine-finding” needs.

Project Background

As part of their plan to launch a full suite of digital products, Wine-Searcher chose AWS as their cloud environment. Strategic resource allocation and cost optimization are critical to ensure a cost-effective operation. Peritos helped as the reliable AWS partner on AWS Cost Explorer and AWS Budgets, like valuable tools for implementing ongoing discounted billing. Furthermore, leveraging reserved instances and spot instances and optimizing usage based on peak hours and demand patterns can result in significant cost savings. Experts from the Peritos team helped regularly monitor and fine-tune the AWS environment based on Winesearcher’s needs, allowing for continuous optimization while adhering to budgetary constraints and maintaining the required scalability and performance for their operations.

Scope & Requirement for Managing AWS Environment

In the 1st Phase of the AWS Environment Setup, implementation was discussed as follows:

Implementation

Technology and Architecture of Managing AWS Environment

Furthermore, Read on the key components that defined the Architecture for managing the AWS Environment Setup for Wine-Searcher

Technology/ Services used

We used AWS services and helped them to setup below

Security & Compliance:

Network Architecture

Backup and Recovery

Cost Optimization

Code Management, Deployment

Cloudformation scripts for creating stack sets and scripts for generating AWS services were handed over to the client

Challenges in Implementing Managing AWS Environment

Support

Next Phase

We are now looking at the next phase of the project, which involves:

About Client

AWS Environment Setup

Tonkin + Taylor is New Zealand’s leading environment and engineering consultancy with offices located globally. They shape interfaces between people and the environment which includes earth, water and air. They have won awards like Beaton Client Choice Award for Best Provider to Government and Community-2022 and IPWEA Award for Excellence in Water Projects for the Papakura Water Treatment Plan- 2021.

Project Background

Tonkin + Taylor were embarking on the journey for launching a full suite of digital product and zeroed upon AWS as thier choice for cloud environment. They wanted to create new applications and migrate to cloud services to improve their scalability, ensure availability, minimize latency and reduce costs. They were also looking to accelerate their digital transformation and add greater business value through AWS Development Environment best practices. Lastly with the new apps they wanted to monetize and create SAAS based offerings for the end users. For all this it was needed to do AWS Environment Setup which follows best practices and meets compliance which can be used as a base for implementing more applications.

Scope & Requirement

In the 1st Phase of the AWS Environment Setup, implementation was discussed as follows:

Implementation

AWS Environment Setup

Technology and Architecture

Read more on the key components which defined the Architecture for AWS Environment Setup for Tonkin + Taylor

Technology/ Services used
Security & Compliance:
Network Architecture
Backup and Recovery

Cloud systems and components used followed AWS’s well-Architected framework and the resources were all Multi-zone availability with uptime of 99.99% or more.

Cost Optimization

Alerts and notifications are configured in the AWS cost

Code Management, Deployment

Cloudformation scripts for creating stacksets and scripts for generating AWS services was handed over to the client

Challenges of AWS Environment Setup

Support

Next Phase

We are now looking at the next phase of the project which involves:

About Client

AWS Control Tower Setup

Wine-Searcher is a web search engine that helps find the price and availability of wine, whiskey, spirits, and beer worldwide.
It has been operating since 1999, with offices in New Zealand and the UK. The platform offers search tools, price comparison,
an extensive database, an encyclopedia, and news content to support all wine-finding needs.

Project Background

Peritos led the AWS Control Tower setup for Wine-Searcher, optimizing their cloud infrastructure.
The implementation streamlined governance, improved compliance, and enabled secure scalability.
Multiple accounts were consolidated and managed using AWS Organizations within Control Tower.

The environment was configured to meet specific business needs, ensuring efficient resource management
and cost control. With built-in automation and governance, Wine-Searcher gained a strong foundation
for future growth while focusing on innovation and user experience.

Scope & Requirement for AWS Control Tower Setup

Implementation

Technology and Architecture of AWS Control Tower Setup

Technology / Services Used
Security & Compliance
Network Architecture
Backup and Recovery
Cost Optimization

Alerts and notifications are configured to monitor AWS costs and prevent budget overruns.

Code Management & Deployment

CloudFormation scripts for stack sets and AWS service provisioning were handed over to the client.

Challenges in Implementing AWS Control Tower Setup

Support

Executive Summary

The Tellabs AWS Glue integration project was designed to establish a secure, scalable, and automated data integration pipeline between Oracle NetSuite and AWS. Leveraging AWS Glue, Amazon S3, and orchestration services such as Step Functions and EventBridge, the solution ensures efficient data extraction, transformation, and storage. The Objective was to build and implement automated ETL pipelines to ensure key reporting metrics are improved in terms of speed and accuracy. 

A strong emphasis was placed on security, governance, monitoring, and automation, ensuring that all workloads align with AWS best practices. The implementation provides enhanced visibility, operational resilience, and cost optimization for data processing workflows. 

About Client

Tellabs is a technology-driven organization requiring secure and scalable cloud infrastructure to support its data integration and analytics workloads. The customer needed a robust AWS-based solution to handle data ingestion from Oracle NetSuite while maintaining strict governance, security, and compliance standards.


Objectives

Scope of Engagement

Architecture Overview

The architecture consists of: 

This architecture ensures scalability, automation, and secure data processing pipelines. 

Solution Overview – To implement automated ETL pipelines using AWS Glue

The solution integrates Oracle NetSuite with AWS using Glue-based ETL pipelines. Data is extracted, transformed, and stored in S3, enabling downstream analytics and reporting. 

Key capabilities: 

Security & Governance

AWS Account Governance SOP 

Identity & Access Management

Network Security (VPC) 

Data Security 

Implement automated ETL pipelines using AWS Glue manage Security and Access control

Implement automated ETL pipelines using AWS Glue and ensure to manage Security and Access control

Monitoring & Logging

Implement automated ETL pipelines using AWS Glue and ensure monitoring and Observability by automating data pipelines

Implement automated ETL pipelines using AWS Glue and ensure monitoring and Observability by automating data pipelines

Implementation

Infrastructure as Code 

CI/CD Automation 

Glue Implementation – automated ETL pipelines using AWS Glue  

Runbook & Troubleshooting

Routine Monitoring Tasks 

Troubleshooting Scenarios 

Lambda Errors 

API Gateway Latency 

Aurora DB Issues 

High Latency Scenario (OPE-001) 

Deployment Readiness Checklist

Testing 

Automation 

Documentation 

Validation 

Cost Optimization & Performance Tuning for running and implementing automated ETL pipelines using AWS Glue

Challenges & Resolutions

Challenge  Resolution 
Secure access management  Implemented IAM roles & SSO 
Monitoring complexity  Centralized CloudWatch dashboards 
Cost visibility  Implemented tagging & cost analysis 
Data security compliance  Used KMS CMKs & encryption 
Deployment consistency  Adopted CloudFormation IaC 

Project Completion

Note

This implementation follows AWS best practices for: 

Implement automated ETL pipelines using AWS Glue

Implement automated ETL pipelines using AWS Glue high-level system setup.

Reference Links

https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-etl-service-pipeline-to-load-data-incrementally-from-amazon-s3-to-amazon-redshift-using-aws-glue.html

Read more about Glue

https://aws.amazon.com/glue/

Read more here about our services

AWS Glue Services

AWS consulting Services

About Client

1Place is a technology-driven organization focused on leveraging data to enhance decision-making, reporting, and operational intelligence. The company manages a growing data ecosystem comprising multiple data sources, analytics tools, and business applications. AWS offers glue as a fully managed etl service

To address scalability challenges and improve data automation, 1Place collaborated with Peritos Solutions, an AWS Advanced Consulting Partner, to design and implement a serverless data platform using AWS Glue. The goal was to replace manual data workflows with a secure, automated ETL framework that ensures accuracy, consistency, and governance. 

Project Background – Data Modernization through AWS Glue ETL service

1Place data operations were previously dependent on traditional ETL tools that lacked automation and flexibility. These manual processes caused data silos, inconsistent data quality, and delayed reporting. 

Peritos Solutions proposed an AWS Glue-based serverless ETL framework to transform 1Place data landscape. The solution automated schema detection, data cataloging, and transformation pipelines while ensuring end-to-end visibility through AWS CloudWatch and centralized governance. 

This transformation allowed 1Place to manage large-scale data workloads with minimal operational overhead while maintaining compliance, traceability, and cost efficiency.

Objectives of the Engagement

Scope & Requirements

Scope 

The project’s scope included the design, deployment, and optimization of AWS Glue components to enable seamless data flow across 1Place AWS environment. 

Key deliverables included: 

Requirements

Functional: 

AWS Glue ETL service Implementation support for Pipeline Automation

AWS Glue ETL service Implementation support for Pipeline Automation

Solution Overview -AWS Glue ETL service

Business Problem Addressed 

1Place existing ETL infrastructure was time-intensive and not scalable. Manual intervention led to delays, higher costs, and inconsistent data quality. 

Proposed AWS Glue-Based Solution 

Peritos Solutions implemented an end-to-end AWS Glue solution integrating multiple data sources and automating data transformation and cataloging. Using Glue Crawlers, Jobs, and Workflows, the entire ETL process became event-driven, reducing human intervention and operational latency. 

Key Benefits 

Implementation -AWS Glue ETL service

Architecture Overview 

The architecture consisted of: 

Technology Stack 

AWS Glue Components Implemented 

Security and Compliance 

Runbook and Troubleshooting Scenarios

Routine Operational Tasks 

Common Troubleshooting Scenarios 

AWS Glue ETL service Implementation support

AWS Glue ETL service Implementation support

Deployment Readiness Checklist

Testing 

Automation 

Documentation 

Monitoring & Validation 

Evidence: Deployment logs, Glue job screenshots, and automation reports attached to project documentation. 

Cost Optimization and Performance Tuning

 Challenges and Resolutions

Challenge  Resolution 
Schema evolution from multiple data sources  Automated schema updates via Glue Crawlers 
Long-running ETL jobs  Spark job optimization and dynamic partitioning 
Data duplication in catalogs  Automated Data Catalog cleanup and versioning 
Integration with legacy databases  Implemented secure JDBC connections and Glue connections 
Monitoring job failures  Integrated CloudWatch alerts with email/SNS notifications 

Project Completion – AWS Glue ETL service

Deliverables 

Support 

Post-implementation support for two months, including 20 hours/month of operational support, bug fixes, and performance optimization. 

Next Phase

AWS Glue ETL service

AWS Glue ETL service Implementation support

Reference Links AWS Glue ETL service

https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-etl-service-pipeline-to-load-data-incrementally-from-amazon-s3-to-amazon-redshift-using-aws-glue.html

Read more about Glue

https://aws.amazon.com/glue/

Read more here about our services

AWS Glue Services

AWS consulting Services