In Progress

Azure Flatten JSON files from Data Lake upload to SQL

From my existing Azure Data Lake, I have daily JSON files that are heavily nested. I need these files flattened and put into a tabular SQL database so that I can access the data more easily from other tools & apps.

Goal: Pull the data from the JSON files in the Azure Data Lake, extract the data, flatten the data into a structure that can be uploaded to a SQL database.

Requirements:

• Use the Azure Cloud stack (Data Factory, Function, Logic App, etc)

• Flatten JSON files in existing Azure Data Lake -- There is a file for each day

flattened data out

• Upload flattened data into SQL database

• Need some documentation on how this process is done so that I can replicate it

o What Azure tool

o How to make any needed adjustments for future use (ie: changing dates, file locations, SQL database, etc)

I have found this example using Azure Logic App (the solution doesn't have to be Logic App) that could be a template: [login to view URL]

I've attached an example of the daily JSON file.

Skills: Microsoft Azure, SQL, Python, Relational Databases, JSON

See more: azure data factory dynamic content, copy data from a rest endpoint by using azure data factory, u sql json extractor, azure data factory flatten json, azure data factory json to sql, azure data factory xml, azure data factory flatten hierarchy, u-sql json outputter, upload sql file database, convrt data date format sql, upload sql, foxpro data entry forms sql server, aspnet document upload sql, import data csv file sql 2005, download upload sql php, insert data xml file sql, asp files upload sql server, upload sql database command prompt, asp file upload sql, store data time oracle sql server aspnet

About the Employer:
( 7 reviews ) Atlanta, United States

Project ID: #25598430

Awarded to:

sangam21

Hi there, I am an IT professional having 9+ years of experience in IT in developing and deploying application/software using Java, Python, Spring, AWS, Azure etc. I have good knowledge on Azure services like Data Fact More

$90 USD in 7 days
(1 Review)
0.3

9 freelancers are bidding on average $185 for this job

khanicha

Hello, You can use Azure Trigger method, so every a new json file generated it will be automatically processed by the apps I will write for you. Here is the reference: [login to view URL] More

$150 USD in 2 days
(11 Reviews)
7.7
kozachenkodr

Dear client. I am very well experienced in complex nested json processing with PYTHON. I am sure I will provide the perfect python script to upload json into SQL. Please ping me to discuss more, I will give you the bes More

$80 USD in 7 days
(26 Reviews)
5.6
fabienbenoit1984

Hello! I'd like to deliver optimized data access. I'm familiar with c++ processing of json, python, windows platform. I'll do the job fairly good. Please, give me a try!

$900 USD in 1 day
(5 Reviews)
4.6
bitnetservices

Hi there, I have done similar activity where Azure Function was used using C#. However, looking at length and breadth and even depth, it seems Python will come handy. On similar stuff, I created a web scrapping to More

$150 USD in 5 days
(1 Review)
1.2
mneo

Hi, We can create a simple ADF pipeline for this. Logic Apps might not be the best option in my opinion. We can discuss further on the solution. Could you please share the json schema with me. Is it a single Json sche More

$90 USD in 7 days
(4 Reviews)
1.0
bathini31

Hello, I am expertise in azure data tasks, having great hands on experience. Handling similar tasks in my current project... Please share more details and reach me if you are fine with my proposal .. I can handover More

$56 USD in 2 days
(0 Reviews)
0.0
Dineshtwi

Programming language I will use C#. You are ok please give a call on this number +91 7598412675 Relevant Skills and Experience Azure developer , [login to view URL], c#, azure devops , java

$50 USD in 10 days
(0 Reviews)
0.0
pranaytt

HI, I am a cloud architect and working with Azure Stack since few years. I have designed several ETL pipelines, parsed / flattened JSON documents in python and apache spark, Azure functions and even container insta More

$100 USD in 5 days
(0 Reviews)
0.0