Python - Extracting Data Between Specific Comment Nodes With Beautifulsoup 4

Looking to pick out specific data from a website such as prices, company info etc. Luckily, the website designer has put lots of tags such as '

Solution 1:

If you want to select the table element after those specific comment(s), then you can select all the comment nodes, filter them based on the desired text, and then select the the next sibling table element:

import requests
from bs4 import BeautifulSoup
from bs4 import Comment

response = requests.get(url)
soup = BeautifulSoup(response.content, "lxml")

comments = soup.find_all(string=lambda text:isinstance(text,Comment))

for comment in comments:
    if comment.strip() == 'Begin Services Table':
        table = comment.find_next_sibling('table')
        print(table)

Alternatively, if you want to get all data between those two comments, then you could find the first comment and then iterate over all the next siblings until you find the closing comment:

import requests
from bs4 import BeautifulSoup
from bs4 import Comment

response = requests.get(url)
soup = BeautifulSoup(response.content, "lxml")

data = []

for comment in soup.find_all(string=lambda text:isinstance(text, Comment)):
    if comment.strip() == 'Begin Services Table':
        next_node = comment.next_sibling

        while next_node and next_node.next_sibling:
            data.append(next_node)
            next_node = next_node.next_sibling

            if not next_node.name and next_node.strip() == 'End Services Table': break;

print(data)