Webscrape Interactive Chart In Python Using Beautiful Soup With Loops
The below code provide information from all the numeric tags in the page. Can I use a filter to extract once for each region For example : https://opensignal.com/reports/2019/04/
Solution 1:
Here is how I would do it for all regions. Requires bs4 4.7.1. AFAICS you have to assume consistent ordering of companies.
import requests
from bs4 import BeautifulSoup
import pandas as pd
r = requests.get("https://opensignal.com/reports/2019/04/uk/mobile-network-experience")
soup = BeautifulSoup(r.content,'lxml') #'html.parser'if lxml not installed
metrics = ['4g-availability', 'video-experience', 'download-speed' , 'upload-speed', 'latency']
headers = ['02', 'Vodaphone', '3', 'EE']
results = []
for region in soup.select('.s-regional-analysis__region'):
for metric in metrics:
providers = [item.text for item in region.select('.c-ru-chart:has([data-metric="' + metric + '"]) .c-ru-graph__number')]
row = {headers[i] : providers[i] for i in range(len(providers))}
row['data-metric'] = metric
row['region'] = region['id']
results.append(row)
df = pd.DataFrame(results, columns = ['region', 'data-metric', '02','Vodaphone', '3', 'EE'] )
print(df)
Sample output:
Solution 2:
Assuming fixed the order of companies (it is, indeed), you can simply reduce the content to examine to only those div's containing the information you need.
import requests
from bs4 importBeautifulSouphtml= requests.get("https://opensignal.com/reports/2019/04/uk/mobile-network-experience").textsoup= BeautifulSoup(html,'html.parser')
res = soup.find_all('div', {'id':'eastern'})
aval = res[0].find_all('div', {'data-chart-name':'4g-availability'})
avalname = aval[0].find('span', {'class':'js-metric-name'}).textupload= res[0].find_all('div', {'data-chart-name':'upload-speed'})
uploadname = upload[0].find('span', {'class':'js-metric-name'}).textcompanies= [i.text for i in aval[0].find_all('span', class_='c-ru-graph__label')]
row1 = [i.text for i in aval[0].find_all('span', class_='c-ru-graph__number')]
row2 = [i.text for i in upload[0].find_all('span', class_='c-ru-graph__number')]
import pandas as pddf= pd.DataFrame({avalname:row1,
uploadname:row2})
df.index = companiesdf= df.T
output
O2 Vodafone 3 EE
4G Availability 82.0 76.9 73.0 89.2
Upload Speed Experience 5.6 5.9 6.8 9.5
Post a Comment for "Webscrape Interactive Chart In Python Using Beautiful Soup With Loops"