ETL Process: Integrate Surveillance Sensor Data

by ybautista
ETL Process: Integrate Surveillance Sensor Data
ETL Process: Integrate Surveillance Sensor Data
ETL Process: Integrate Surveillance Sensor Data
ETL Process: Integrate Surveillance Sensor Data
ETL Process: Integrate Surveillance Sensor Data

ETL Process: Integrate Surveillance Sensor Data - Email attachments & FTP (Data Integration / Data Warehousing) My client wanted to integrate plain text files coming from 2 separate surveillance software platforms, where sensors and cameras are setup in doors of most stores located in Shopping Centers. Data files included hourly counts of people coming in and out of stores, and were delivered daily as e-mail attachments, or placed in FTP location. I developed Java-based ETL process(*) to extract text file attachments, or download them from FTP, parse and transform their proprietary format, and load dimension and fact tables in target data warehouse. Data warehouse database was deployed on premises using SQL Server as back-end. Now users are able to access the data warehouse using BI tools such as Power BI. * The solution was developed using Talend Open Studio for Data Integration.

image of username ybautista Flag of Peru AREQUIPA, Peru

About Me

I develop software you can use to get the most out of your data, by pulling it out from your data sources, and putting it in a format ready for analysis: Data Warehouse, Hadoop data lake, and AWS S3 buckets. Experience, Skills & interests: [3+ yrs] - Developed expertise building large-scale web scraping solutions using Python (Beautiful Soup, Selenium/Chrome headless), and Django/Celery/Redis stack as backend - Hands-on experience writing Python scripts for Algorithmic trading and financial data analysis (Pandas/Numpy/Ta-lib.) [4+ yrs] - Hands-on experience working on cloud integration projects Platforms: AWS/Microsoft Azure/Google Cloud - Strong command of object-oriented programming skills in Java [5+ yrs] - Substantial experience in Data Integration (ETL)/ Data Warehousing projects using Talend Open Studio - Proven skills in Transact-SQL as a server-side programming language for both on-premises SQL Server & Azure SQL Database.

$50 USD/hr

29 reviews
6.4

Tags