In Progress

Create a dynamic script to scrape multiple websites. -- 2

Create a dynamic script in NodeJS to scrape 15+ sites using configurable json.

Requirement:

-Should be able to scrape lazy loading sites

-Download images

-Go through all pagination pages and scrape the child pages

-If element is a product link "click" that link and scrape that child page

-Get all below elements and retrieve correctly

See below for understanding how to set it up.

//Example configurable object

[

site1: {

url: "[url removed, login to view]",

parent: {

productList: "div.product-list",

product-link: "a.product-url",

product-name: ".product-info .product-title",

product-image: ".product-images img[src attribute]",

product-price: {

list-price: ".product-price span.old-price",

sale-price: ".product-price [url removed, login to view]"

}

pagination: "button.load-more-btn",

product-page: {

name: "form#product h1",

price: {

list-price: ".price .old",

sale-price: ".price new"

},

colors: "#options-articles li",

sizes: "#options-variants li",

description: ".description p",

images: {

main: "#images .product-image-box img[src attribute]",

thumbs: "#images .thumbs .product-thumbs li"

}

}

...

}

},

site2: {

Similar to above Site1

...

}

...

]

==================================================================================

//Product Class

Class Product() {

Product(name, link, image, price, colors, sizes, description) {

[url removed, login to view] = name;

[url removed, login to view] = image;

[url removed, login to view] = price;

[url removed, login to view] = colors;

[url removed, login to view] = sizes;

[url removed, login to view] = description;

}

}

===================================================================================

//Main Scraper Class

Class Scraper() {

Scraper(site) {

[url removed, login to view] = site;

}

getName(attr) {

return name value;

}

getLink(attr) {

return link value;

}

getMainImage(attr) {

return main value;

}

getThumbImages(attr) {

return thumb array;

}

getPrice(attr) {

return Price Object(list, sale);

}

getColor(attr) {

return Colors array;

}

getSizes(attr) {

return Sizes array;

}

getName(attr) {

return desc value;

}

hasPagination() {

if([url removed, login to view] element exist)

//scrape all pagination pages to the last page

}

downloadImage() {

if([url removed, login to view] element exist)

//download image

src = getImage();

download(src);

}

getProductInfo(product) {

//Go to product page and scrape using "product-page" attribute from json sample.

return {

name: String,

price {

old: String

new: String

},

images : {

main: String src,

thumbs: [array]

},

sizes: [array],

colors: [array],

description: String

}

}

getListOfProducts() {

//Use [url removed, login to view] to get listOfProducts

ArrayOfProducts = [];

for( items in listOfProducts ){

//create new product

product = new Product( getProductInfo() );

//add it to list

[url removed, login to view]( product );

}

}

}

Skills: Python, Software Architecture

See more: I like someone who can scrape data from multiple websites and arrange it in excel, python script scrape websites, magento create multiple websites, node js cheerio tutorial, web scraping dynamic content, client side web scraping, web scraping javascript tutorial, python parse web page beautifulsoup, javascript web scraping example, python web scraping dynamic content, python web scraping example, create multiple websites magento, scrape data multiple websites, scrape multiple websites, web scrape multiple websites, create multiple websites single magento, create multiple websites magento windows, script scrape websites, want create dashboard multiple websites page, script scrape data websites

About the Employer:
( 1 review ) San Jose, United States

Project ID: #15688261

Awarded to:

$375 USD in 7 days
(4 Reviews)
3.1

4 freelancers are bidding on average $446 for this job

LuckyQ

Hello , My Name is Sam , I am Full Stack web and app developer . I'd read the Project details extremely carefully and I am absolutely sure that I can do the It very well . Relevant Skills and Experience To Know idea a More

$555 USD in 10 days
(21 Reviews)
5.3
$300 USD in 10 days
(4 Reviews)
4.5
krishan725

I have gone through your project requirement and would like to discuss further on it. Please send me personal message for more clarification. [login to view URL] Relevant Skills and Experience Relavent exper More

$555 USD in 10 days
(1 Review)
2.4