ETL Process: Load CSV files attached to emails into BigQuery
ETL Process: Load CSV files attached to emails into BigQuery (Big Data Integration) I developed an automated Java-based import solution that integrates Google Analytics reports delivered as CSV files attached to e-mails sent daily, and makes it available as BigQuery tables (Google Cloud), so Marketing users from a leading advertising agency (Ogilvy) can analyse campaign statistics and metrics. ETL process does the following: - Download e-mails sent on process date to specific e-mail recipient - Export attachments as CSV files - Upload files to a bucket in Google Cloud Storage - Create BigQuery's table schema based on columns from CSV, and insert records - Append new records if table already exist. * The solution was developed using Talend Open Studio for Big Data.
About Me
I develop software you can use to get the most out of your data, by pulling it out from your data sources, and putting it in a format ready for analysis: Data Warehouse, Hadoop data lake, and AWS S3 buckets. Experience, Skills & interests: [3+ yrs] - Developed expertise building large-scale web scraping solutions using Python (Beautiful Soup, Selenium/Chrome headless), and Django/Celery/Redis stack as backend - Hands-on experience writing Python scripts for Algorithmic trading and financial data analysis (Pandas/Numpy/Ta-lib.) [4+ yrs] - Hands-on experience working on cloud integration projects Platforms: AWS/Microsoft Azure/Google Cloud - Strong command of object-oriented programming skills in Java [5+ yrs] - Substantial experience in Data Integration (ETL)/ Data Warehousing projects using Talend Open Studio - Proven skills in Transact-SQL as a server-side programming language for both on-premises SQL Server & Azure SQL Database.
