Find Jobs
Hire Freelancers

AWS Lambda / Python Guru - Re-Architecture

$750-1500 USD

Completed
Posted over 6 years ago

$750-1500 USD

Paid on delivery
Each month we receive a CSV data file that has about 5-6k records. We need that data processed and put into a MySQL data base. Here is the background and workflow: 1) We receive the raw data file as a CSV (via email) 2) We have a script that uploads that CSV to a AWS S3 bucket (called raw_data) 3) A Python Lambda is triggered that iterates over each of the records in the file.....for each record we pull a unique identifier and use that identifier to call an external api and pull in other relevant infoation 4) the results from #3 are inserted into a AWS MySQL RDS instance. 5) Move the raw data file from the raw_data bucket to a processed_data bucket So if it's working what else do we need? We'll the max time a Lambda can run is 5 mins, and the script is timing out before if finished so we need to refactor it. What we would like, is to update it to the following pattern: 1 Master lambda that reads all of the records and fans out the processing via SQS (as 5-6k messages). Then have another lambda that is triggered that will only process one message at a time. The deliverables would include: 1) All source code (the current lambda python script would need to be updated) 2) Cloud Formation Templates 3) Configured security groups such that ports, etc aren't open to the world and the only the applicable resources can talk to one another (S3, SQS, Lambda, RDS, etc) Open to fair estimates for this project
Project ID: 15775786

About the project

23 proposals
Remote project
Active 6 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Great idea! I just think Redis is faster than SQS since you want to use the queues just from s3. SQS is fine when you want to use the queues from outside of AWS. Relevant Skills and Experience - AWS Lambda, SQS, RDS - Redis, Celery - Ansible Proposed Milestones $950 USD - all
$950 USD in 10 days
4.8 (234 reviews)
8.3
8.3
23 freelancers are bidding on average $1,168 USD for this job
User Avatar
Thank you for looking at our bid. Separate Lambada per record that would process (instead of one Lambda processing all records as being done currently), request your response. (guess externalizing api call through an independent Lambda would not make your current master process faster) Relevant Skills and Experience 100+ projects through Freelancer - Firebase, AWS, S3, dynamo, kindly visit our profile on Freelancer to review our past. Proposed Milestones $1500 USD - to be discussed Just need to review few things (1-2 days) & discuss before accepting award. We have an AWS dev account, may be inactive post trial period. We request your response to discuss.
$1,500 USD in 21 days
4.9 (79 reviews)
7.9
7.9
User Avatar
Hello, I'd be glad to complete this python project for you. Loking forward to chat with you soon for more details. Best regards, Relevant Skills and Experience Python , data processing, automation Proposed Milestones $882 USD - payment after project completion
$750 USD in 7 days
5.0 (180 reviews)
7.4
7.4
User Avatar
Hello sir I am a python developer with 7 years of experience. I am very familiar with aws s3, lambda, sqs and etc. I am confident with this job and can start the work right now. Best, Zhang Relevant Skills and Experience python, aws Proposed Milestones $888 USD - Re-Architecture
$888 USD in 10 days
4.9 (127 reviews)
7.7
7.7
User Avatar
I have read your project details. I have to ask a few questions. Can you please message me via chat so we can discuss all the details to elicit all the requirements and hence start the development? Relevant Skills and Experience I will showcase all the skills and my experience for this project over the chat. Please message me so we can resume this discussion. Stay tuned, I'm still working on this proposal.
$1,250 USD in 20 days
5.0 (42 reviews)
7.6
7.6
User Avatar
Hello, I always provide fastest results with quality,I have done more than 300 projects here..I have an experience of more than 10 years in web development and maintenance. Relevant Skills and Experience I have in-depth knowledge of php, mysql, javascript , jquery, paypal integrations, API's, css, html, html5, wordpress. I will look forward to discuss with you. Thank you. Proposed Milestones $988 USD - milestone
$988 USD in 20 days
4.6 (125 reviews)
7.7
7.7
User Avatar
Hi I am interested in this project. I am high skilled Python developer and know well about Amazon Lambda. I understood your requirements and ready to discuss. Let us discuss in detail Relevant Skills and Experience Python, Amazon Lambda Proposed Milestones $1411 USD - disucss
$1,411 USD in 20 days
5.0 (42 reviews)
7.0
7.0
User Avatar
Hi, .I specialise in web development. I live in Oxford. Your project looks exciting to me and I’ve got the skills required Relevant Skills and Experience Amazon Web Services, Aws Lambda, MySQL, Python, Software Architecture Proposed Milestones $1250 USD - project milestone I've done someting similar recently, will be happy to share.
$1,250 USD in 20 days
5.0 (18 reviews)
6.1
6.1
User Avatar
This bid is to modify an existing AWS lambda function by leveraging the SQS service and a sub-lambda to process the file incrementally in order to avoid timing out. Relevant Skills and Experience Python, AWS, S3, Data Processing, MySQL Proposed Milestones $1000 USD - Modify / test the existing Lambda function to overcome timeout issue.
$1,000 USD in 10 days
5.0 (44 reviews)
6.3
6.3
User Avatar
Thanks for sharing the details. I can help you with programming and infrastructure both. I have been dealing with these services since 14+ years. Relevant Skills and Experience PHP MySQL JAVAScript Ajax CSS HTML5 IOS Android Proposed Milestones $1250 USD - To be decided after discussion
$1,250 USD in 30 days
5.0 (51 reviews)
5.9
5.9
User Avatar
We love challenging opportunities. Looks like AWS configuration/services is a major issue here. We are clear about the requirements and our certified AWS solution architect can certainly solve it. Relevant Skills and Experience AWS Certified Developer Proposed Milestones $1323 USD - AWS Configuration + Python Script review Are you open to changes in AWS services being used?
$1,323 USD in 12 days
4.3 (51 reviews)
6.8
6.8
User Avatar
Amazon AWS Expert Team We are a competent team for AWS and proud member of Amazon AWS global partner network (APN) and a certified AWS service provider. Skills and Expertise We help our clients at every step of their Cloud journey including - Planning, Procurement, Deployment, Migration, Optimization, Support and security. Our managed service helps optimizing Cloud efficiently for Cost, Speed, Scale, Security and Automation. In order to understand more about your project requirements. Kindly initiate chat. Thanks! aastha
$750 USD in 7 days
4.9 (46 reviews)
6.0
6.0
User Avatar
Hello AWS Lambda / Python Guru - Re-Architecture Relevant Skills and Experience Amazon Web Services, Aws Lambda, MySQL, Python, Software Architecture Proposed Milestones $1250 USD - when i did it I have some questons for your project now thx
$1,250 USD in 20 days
4.9 (26 reviews)
5.7
5.7
User Avatar
Dear sir alredy done this type of job having 8 year exp in php , pls ping me for more detail am waiting your reply ! Relevant Skills and Experience 10 year exp in amazon web services Proposed Milestones $1666 USD - complete Stay tuned, I'm still working on this proposal.
$1,666 USD in 25 days
4.9 (30 reviews)
5.8
5.8
User Avatar
A cloud engineer with 2.5 years of hands-on experience with AWS clouds. Has delivered high quality, scalable, available, reliable and fault-tolerant solutions using AWS. seeks quality work Relevant Skills and Experience I have expertise in following areas 1. Lambda(Integration with SNS, SQS, S3, dynamodb, API gateway, Kinesis etc) 2. SQS(Decoupled architecture, SAM) 3. S3, EC2, ECS, Kinesis,Dynamodb etc 4. python etc Proposed Milestones $150 USD - Project Award $300 USD - First Milestone(deliverable tbd) $300 USD - Final Milestone(deliverable tbd) Can we discuss further and refine the architecture?
$750 USD in 20 days
5.0 (21 reviews)
5.2
5.2
User Avatar
I have been in this field for more than 7 years. I have experience in php,perl,wordpress,Html,postgress ,Web Services, Rest Api, Zend ,YIi,Mysql. Relevant Skills and Experience so I am right candidate for your project because I have experience in C#Aws,,Perl,php,node.js,Crypto currency,Wordpress,Ebay,Amazon,Instagram,Mysql,Mongo Db,Api projects. Proposed Milestones $1250 USD - AWS Lambda / Python Guru - Re-Architecture
$1,250 USD in 20 days
5.0 (20 reviews)
5.4
5.4
User Avatar
Hi, I am Python expert and have rich experiences with AWS. I can complete this project successfully. Hope to work with you. Stay tuned, I'm still working on this proposal.
$1,323 USD in 20 days
5.0 (11 reviews)
4.8
4.8
User Avatar
Can help you with this. Message me . I will deliver in 10 days. Relevant Skills and Experience more than 5 years experience in full stack development Proposed Milestones $1500 USD - code,deployment
$1,500 USD in 10 days
5.0 (6 reviews)
4.0
4.0
User Avatar
Hi, I am an experienced programmer and I'm sure I can complete your project asap. Please contact me and tell about your project details. I promise a high quality and punctual work. Thanks. AWS Lambda / Python Guru - Re-Architecture Amazon Web Services, Aws Lambda, MySQL, Python, Software Architecture,
$1,250 USD in 6 days
4.9 (8 reviews)
4.2
4.2
User Avatar
Lets introduce our company Gropse Technologiesthat has been in business for the last 5 years developing mobile apps and websites. We have an excellent track record for the best customer satisfaction. Stay tuned, I'm still working on this proposal.
$1,250 USD in 20 days
5.0 (7 reviews)
4.2
4.2
User Avatar
As a python expert, I will like to complete your project. we can discuss details over chat.
$750 USD in 5 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
BALTIMORE, United States
5.0
97
Payment method verified
Member since Jul 18, 2007

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.