ETL Process: Import CSV files into Azure SQL Database

by ybautista
ETL Process: Import CSV files into Azure SQL Database
ETL Process: Import CSV files into Azure SQL Database
ETL Process: Import CSV files into Azure SQL Database
ETL Process: Import CSV files into Azure SQL Database
ETL Process: Import CSV files into Azure SQL Database

ETL Process: Import CSV files into Azure SQL Database (Data Integration / Data Warehousing) My client wanted a process to integrate CSV files available in 2 different formats. Data files tracked loan commissions on a monthly basis, and were copied by user to folder location that would be monitored by ad-hoc program. I developed a Java-based ETL process(*) to monitor a folder for changes, parse and transform each CSV file, and load dimension and fact tables in data warehouse. Data warehouse database was deployed to the cloud using Azure SQL Database service as back-end. Now users are able to access the data warehouse using BI tools such as Power BI. * The solution was developed using Talend Open Studio for Data Integration.

image of username ybautista Flag of Peru AREQUIPA, Peru

About Me

I develop software you can use to get the most out of your data, by pulling it out from your data sources, and putting it in a format ready for analysis: Data Warehouse, Hadoop data lake, and AWS S3 buckets. Experience, Skills & interests: [3+ yrs] - Developed expertise building large-scale web scraping solutions using Python (Beautiful Soup, Selenium/Chrome headless), and Django/Celery/Redis stack as backend - Hands-on experience writing Python scripts for Algorithmic trading and financial data analysis (Pandas/Numpy/Ta-lib.) [4+ yrs] - Hands-on experience working on cloud integration projects Platforms: AWS/Microsoft Azure/Google Cloud - Strong command of object-oriented programming skills in Java [5+ yrs] - Substantial experience in Data Integration (ETL)/ Data Warehousing projects using Talend Open Studio - Proven skills in Transact-SQL as a server-side programming language for both on-premises SQL Server & Azure SQL Database.

$50 USD/hr

29 reviews
6.4

Tags