Web Scrapping using Python | DS-1

Web Scrapping

Libraries used in Web Scrapping

Now follow this steps for web scrapping:

Installing Libraries:

pip install beautifulsoup4
pip install selenium

Importing Libraries in code:

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver

Webdriver:

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
driver=webdriver.Chrome('chromedriver',chrome_options=chrome_option)

Getting website content:

driver.get("https://www.flipkart.com/search?q=IPHONE&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off")content = driver.page_source
soup = BeautifulSoup(content)

Creating an empty List for our data:

products=[]
prices=[]
features=[]
ratings=[]

Appending Data to List:

for a in soup.findAll('a',href=True, attrs={'class':'_1fQZEK'}):
name=a.find('div',attrs={'class':'_4rR01T'})
price=a.find('div',attrs={'class':'_30jeq3 _1_WHN1'})
feature=a.find('div',attrs={'class':'fMghEO'})
rating=a.find('div',attrs={'class':'_3LWZlK'})products.append(name.text)
prices.append(price.text)
features.append(feature.text)
ratings.append(rating.text)

Creating Data Frame and extracting to CSV :

df = pd.DataFrame({'Product Name':products,'Price':prices , 'Feature':features, 'Rating': ratings})print(df.head())
df.to_csv('products.csv', index=False, encoding='utf-8')

Github Link:

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Being a CTO: Eating the Elephant

The current adoption status of cgroup v2 in containers

How Simpler and Shorter Code of Kotlin Has Changed the Android Development Experience?

We had to decline several orders ¡n case that the production was held

Binary Trees Quiz 1

Part 4 - Namespaces in Kubernetes

What can organizations who use RedHat OpenShift via IBM Cloud Paks do to ensure regulatory…

One stop guide for implementing scheduling in Spring Boot

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
PRASHIL VAISHNANI

PRASHIL VAISHNANI

student

More from Medium

Setup formatting / linting for Python when committing

Create executable file on window from your python code using Power shell.

A Fedex & geolocation scraper with Python

Auto-book COVID test appointment