Commit @32fe6114fbdc210f061ec9e19ffa3652c4436a02 - yjyoon/구미시-네이버-블로그-스크레퍼

윤영준 2023-12-12

Added a blog scrapper, finally able to supress annoying MarkupResemblesLocatorWarning, since, everything that bs4 handles are just html...

@32fe6114fbdc210f061ec9e19ffa3652c4436a02

81b7126

blog/구미라면/2022-10-30 (deleted)

--- blog/구미라면/2022-10-30

...	...	@@ -1,43 +0,0 @@
	1	-url,postdate,creator,title,contents
	2	-https://blog.naver.com/zoyongeun/222915073436,2022-10-30,,,
	3	-https://blog.naver.com/doremifa_sola/222914954215,2022-10-30,,,
	4	-https://blog.naver.com/kawaeev7/222915203534,2022-10-30,,,
	5	-https://blog.naver.com/wlstlf712/222914811663,2022-10-30,,,
	6	-https://blog.naver.com/moonwonwoo/222914588070,2022-10-30,,,
	7	-https://blog.naver.com/jjamsd/222914506977,2022-10-30,,,
	8	-https://blog.naver.com/love8117/222914518260,2022-10-30,,,
	9	-https://blog.naver.com/immersemyself/222914627542,2022-10-30,,,
	10	-https://blog.naver.com/yummyhouse1/222914540676,2022-10-30,,,
	11	-https://blog.naver.com/sangyun1106/222915008572,2022-10-30,,,
	12	-https://blog.naver.com/gimjoo0619/222914581517,2022-10-30,,,
	13	-https://blog.naver.com/jrrrr90/222915077573,2022-10-30,,,
	14	-https://blog.naver.com/janussunny/222914510031,2022-10-30,,,
	15	-https://blog.naver.com/conpanna88/222914558063,2022-10-30,,,
	16	-https://blog.naver.com/sidsid20000/222914511949,2022-10-30,,,
	17	-https://blog.naver.com/mybosom18/222914946895,2022-10-30,,,
	18	-https://blog.naver.com/yeo_un1217/222915061909,2022-10-30,,,
	19	-https://blog.naver.com/062810609/222915090337,2022-10-30,,,
	20	-https://blog.naver.com/s2silver_s2/222914651369,2022-10-30,,,
	21	-https://blog.naver.com/bobgawaa/222915083419,2022-10-30,,,
	22	-https://blog.naver.com/so_o_zero_/222915166919,2022-10-30,,,
	23	-https://blog.naver.com/thdus1138/222914628152,2022-10-30,,,
	24	-https://blog.naver.com/lhsss114/222915039917,2022-10-30,,,
	25	-https://blog.naver.com/lhw1067/222914981845,2022-10-30,,,
	26	-https://blog.naver.com/dmsdk1300/222914415346,2022-10-30,,,
	27	-https://blog.naver.com/entro_pi/222915141637,2022-10-30,,,
	28	-https://blog.naver.com/yeejilove1991/222914800407,2022-10-30,,,
	29	-https://blog.naver.com/wnwnynn/222915175278,2022-10-30,,,
	30	-https://blog.naver.com/j__h_1008/222915122861,2022-10-30,,,
	31	-https://blog.naver.com/jinheeyoung0613/222915025314,2022-10-30,,,
	32	-https://blog.naver.com/sohee3182/222915043481,2022-10-30,,,
	33	-https://blog.naver.com/12345uni/222915141315,2022-10-30,,,
	34	-https://blog.naver.com/h_matzip/222914870485,2022-10-30,,,
	35	-https://blog.naver.com/jjsh114/222914851308,2022-10-30,,,
	36	-https://blog.naver.com/karen021011/222915119121,2022-10-30,,,
	37	-https://blog.naver.com/jieuny118/222915119978,2022-10-30,,,
	38	-https://blog.naver.com/rlaalswjd930/222915192453,2022-10-30,,,
	39	-https://blog.naver.com/donguibogam1/222911962901,2022-10-30,,,
	40	-https://blog.naver.com/dany_jeong/222915215212,2022-10-30,,,
	41	-https://blog.naver.com/loulouya/222915069104,2022-10-30,,,
	42	-https://blog.naver.com/99shootforthemoon/222915140342,2022-10-30,,,
	43	-https://blog.naver.com/inbg1117/222915211409,2022-10-30,,,

81b7126

blog/구미라면/2022-10-31 (deleted)

--- blog/구미라면/2022-10-31

...	...	@@ -1,45 +0,0 @@
	1	-url,postdate,creator,title,contents
	2	-https://blog.naver.com/brightenedu/222914646585,2022-10-31,,,
	3	-https://blog.naver.com/tjsal0199/222915512669,2022-10-31,,,
	4	-https://blog.naver.com/bobgawaa/222911274493,2022-10-31,,,
	5	-https://blog.naver.com/dlwngml5351/222915972891,2022-10-31,,,
	6	-https://blog.naver.com/f40335/222915949828,2022-10-31,,,
	7	-https://blog.naver.com/linda5021/222915511172,2022-10-31,,,
	8	-https://blog.naver.com/kawaeev7/222916019604,2022-10-31,,,
	9	-https://blog.naver.com/nengsu68/222915689797,2022-10-31,,,
	10	-https://blog.naver.com/silver_0909/222915602485,2022-10-31,,,
	11	-https://blog.naver.com/nangman_lee/222915726003,2022-10-31,,,
	12	-https://blog.naver.com/qsdfw4568/222916075326,2022-10-31,,,
	13	-https://blog.naver.com/ichaeyeon24/222915831821,2022-10-31,,,
	14	-https://blog.naver.com/siz_0205/222916126018,2022-10-31,,,
	15	-https://blog.naver.com/ekaqoduwk79/222915999631,2022-10-31,,,
	16	-https://blog.naver.com/allsix6/222915383220,2022-10-31,,,
	17	-https://blog.naver.com/happy_ahy/222915378859,2022-10-31,,,
	18	-https://blog.naver.com/z990/222915748203,2022-10-31,,,
	19	-https://blog.naver.com/h30457/222916067190,2022-10-31,,,
	20	-https://blog.naver.com/jjamsd/222915387190,2022-10-31,,,
	21	-https://blog.naver.com/chaechae_0621/222915315808,2022-10-31,,,
	22	-https://blog.naver.com/rlaxogns1123/222916094935,2022-10-31,,,
	23	-https://blog.naver.com/llllsoft/222915604757,2022-10-31,,,
	24	-https://blog.naver.com/farrung/222915757565,2022-10-31,,,
	25	-https://blog.naver.com/reeree_effect/222914941143,2022-10-31,,,
	26	-https://blog.naver.com/9cns2/222915875506,2022-10-31,,,
	27	-https://blog.naver.com/nuri1453/222915658927,2022-10-31,,,
	28	-https://blog.naver.com/salty0salt/222915903441,2022-10-31,,,
	29	-https://blog.naver.com/gbsahouse/222915715955,2022-10-31,,,
	30	-https://blog.naver.com/rin1554/222916047988,2022-10-31,,,
	31	-https://blog.naver.com/bexrryx/222915683483,2022-10-31,,,
	32	-https://blog.naver.com/deadamhan/222915759681,2022-10-31,,,
	33	-https://blog.naver.com/bravewoman3/222915374395,2022-10-31,,,
	34	-https://blog.naver.com/studio_vtoil/222915933077,2022-10-31,,,
	35	-https://blog.naver.com/n_oeyha/222916132334,2022-10-31,,,
	36	-https://blog.naver.com/newhopeman/222915346531,2022-10-31,,,
	37	-https://blog.naver.com/bcfarm33/222915684358,2022-10-31,,,
	38	-https://blog.naver.com/cloudmanuse/222915800551,2022-10-31,,,
	39	-https://blog.naver.com/koreha_gumi/222915763407,2022-10-31,,,
	40	-https://blog.naver.com/iaan6098/222913841276,2022-10-31,,,
	41	-https://blog.naver.com/chaela/222915793598,2022-10-31,,,
	42	-https://blog.naver.com/sevun77/222915180794,2022-10-31,,,
	43	-https://blog.naver.com/bcfarm33/222915673730,2022-10-31,,,
	44	-https://blog.naver.com/syw4262/222915218129,2022-10-31,,,
	45	-https://blog.naver.com/wdlje220/222915901314,2022-10-31,,,

81b7126

blog/구미라면/2022-11-01 (deleted)

--- blog/구미라면/2022-11-01

...	...	@@ -1,36 +0,0 @@
	1	-url,postdate,creator,title,contents
	2	-https://blog.naver.com/luckseven123/222916784549,2022-11-01,,,
	3	-https://blog.naver.com/july24/222916680980,2022-11-01,,,
	4	-https://blog.naver.com/leejhda/222916910200,2022-11-01,,,
	5	-https://blog.naver.com/ximotors/222915500803,2022-11-01,,,
	6	-https://blog.naver.com/vivianna2015/222917090120,2022-11-01,,,
	7	-https://blog.naver.com/lieblich_pearl/222916693866,2022-11-01,,,
	8	-https://blog.naver.com/yuna579/222916856153,2022-11-01,,,
	9	-https://blog.naver.com/abc7578/222916187511,2022-11-01,,,
	10	-https://blog.naver.com/tkd0516/222916619160,2022-11-01,,,
	11	-https://blog.naver.com/eye0902/222916686958,2022-11-01,,,
	12	-https://blog.naver.com/knmknmknm/222916810577,2022-11-01,,,
	13	-https://blog.naver.com/skdusol/222913062733,2022-11-01,,,
	14	-https://blog.naver.com/rai122/222916211086,2022-11-01,,,
	15	-https://blog.naver.com/gagegaja/222896475249,2022-11-01,,,
	16	-https://blog.naver.com/ydr2812/222916951750,2022-11-01,,,
	17	-https://blog.naver.com/alswls113500/222916212467,2022-11-01,,,
	18	-https://blog.naver.com/mooorr/222916708066,2022-11-01,,,
	19	-https://blog.naver.com/dobby1102/222916137282,2022-11-01,,,
	20	-https://blog.naver.com/ujiny1957/222917124770,2022-11-01,,,
	21	-https://blog.naver.com/noteella/222916426538,2022-11-01,,,
	22	-https://blog.naver.com/carpediem1120/222916884899,2022-11-01,,,
	23	-https://blog.naver.com/godqhrgmlwn/222916531017,2022-11-01,,,
	24	-https://blog.naver.com/kidhyeji/222916503740,2022-11-01,,,
	25	-https://blog.naver.com/khg6482/222916207394,2022-11-01,,,
	26	-https://blog.naver.com/rudals8227/222916684538,2022-11-01,,,
	27	-https://blog.naver.com/patently/222916173924,2022-11-01,,,
	28	-https://blog.naver.com/koreha_gumi/222916623427,2022-11-01,,,
	29	-https://blog.naver.com/homeless17/222916920117,2022-11-01,,,
	30	-https://blog.naver.com/gae7vtabamz/222916513057,2022-11-01,,,
	31	-https://blog.naver.com/susierose7/222916206945,2022-11-01,,,
	32	-https://blog.naver.com/parkcy6737/222916478783,2022-11-01,,,
	33	-https://blog.naver.com/world2013v/222917025006,2022-11-01,,,
	34	-https://blog.naver.com/gkgkgk11k/222916947602,2022-11-01,,,
	35	-https://blog.naver.com/prome6328/222916277490,2022-11-01,,,
	36	-https://blog.naver.com/haelove21/222916834867,2022-11-01,,,

81b7126

blog/구미라면/2022-11-02 (deleted)

--- blog/구미라면/2022-11-02

...	...	@@ -1,46 +0,0 @@
	1	-url,postdate,creator,title,contents
	2	-https://blog.naver.com/koreha_gumi/222917663605,2022-11-02,,,
	3	-https://blog.naver.com/qpalxn0/222917887179,2022-11-02,,,
	4	-https://blog.naver.com/rozy_hyebin/222917300286,2022-11-02,,,
	5	-https://blog.naver.com/wlstlf712/222918009529,2022-11-02,,,
	6	-https://blog.naver.com/jrrrr90/222917093020,2022-11-02,,,
	7	-https://blog.naver.com/lr0249/222917256887,2022-11-02,,,
	8	-https://blog.naver.com/darockhouse/222917710245,2022-11-02,,,
	9	-https://blog.naver.com/ljr041313/222917908160,2022-11-02,,,
	10	-https://blog.naver.com/rehouse/222917517973,2022-11-02,,,
	11	-https://blog.naver.com/2n_yang/222918074318,2022-11-02,,,
	12	-https://blog.naver.com/woqjftjsrnjs/222917729542,2022-11-02,,,
	13	-https://blog.naver.com/delta1003560/222917480535,2022-11-02,,,
	14	-https://blog.naver.com/silver_gi03/222917840653,2022-11-02,,,
	15	-https://blog.naver.com/kimbeauti/222917217389,2022-11-02,,,
	16	-https://blog.naver.com/taijiaa/222918109912,2022-11-02,,,
	17	-https://blog.naver.com/sosin7/222917608467,2022-11-02,,,
	18	-https://blog.naver.com/clarin_/222917180233,2022-11-02,,,
	19	-https://blog.naver.com/4e5rd7ugthr/222918007240,2022-11-02,,,
	20	-https://blog.naver.com/sosin7/222917615615,2022-11-02,,,
	21	-https://blog.naver.com/hwyj0412/222917359502,2022-11-02,,,
	22	-https://blog.naver.com/bittenorange/222917895348,2022-11-02,,,
	23	-https://blog.naver.com/cocom213/222917746323,2022-11-02,,,
	24	-https://blog.naver.com/dingading_/222917482526,2022-11-02,,,
	25	-https://blog.naver.com/sujitasan_x/222917148883,2022-11-02,,,
	26	-https://blog.naver.com/tkdals6794/222917397544,2022-11-02,,,
	27	-https://blog.naver.com/kls9212/222917095831,2022-11-02,,,
	28	-https://blog.naver.com/gpdnjs691/222917820717,2022-11-02,,,
	29	-https://blog.naver.com/myjeje1301/222917734565,2022-11-02,,,
	30	-https://blog.naver.com/ghn06233/222918109114,2022-11-02,,,
	31	-https://blog.naver.com/ekfaam/222917390881,2022-11-02,,,
	32	-https://blog.naver.com/0ooo000/222917868020,2022-11-02,,,
	33	-https://blog.naver.com/choue00011/222917799046,2022-11-02,,,
	34	-https://blog.naver.com/koreha_gumi/222917609379,2022-11-02,,,
	35	-https://blog.naver.com/gksl917/222917883132,2022-11-02,,,
	36	-https://blog.naver.com/wildcherry03/222918094168,2022-11-02,,,
	37	-https://blog.naver.com/nanodental2288/222917497669,2022-11-02,,,
	38	-https://blog.naver.com/koreha_gumi/222917616884,2022-11-02,,,
	39	-https://blog.naver.com/vrenop6335/222917474383,2022-11-02,,,
	40	-https://blog.naver.com/rag1127/222917576631,2022-11-02,,,
	41	-https://blog.naver.com/koreha_gumi/222917643235,2022-11-02,,,
	42	-https://blog.naver.com/gkgkgk11k/222917423823,2022-11-02,,,
	43	-https://blog.naver.com/goldstar114/222917683934,2022-11-02,,,
	44	-https://blog.naver.com/vbb13rl/222917622953,2022-11-02,,,
	45	-https://blog.naver.com/kqt20/222916706028,2022-11-02,,,
	46	-https://blog.naver.com/mygodis111/222917734864,2022-11-02,,,

32fe611

naver_blog_info_gatherer.py (added)

+++ naver_blog_info_gatherer.py

...	...	@@ -0,0 +1,229 @@
	1	+import re
	2	+import regex
	3	+import time
	4	+import math
	5	+import pandas as pd
	6	+import os
	7	+import bs4
	8	+import warnings
	9	+from bs4 import MarkupResemblesLocatorWarning
	10	+from datetime import datetime, timedelta
	11	+from selenium import webdriver
	12	+from selenium.webdriver.common.by import By
	13	+from webdriver_manager.chrome import ChromeDriverManager
	14	+from selenium.webdriver.chrome.service import Service as ChromeService
	15	+from tqdm import tqdm
	16	+from joblib import Parallel, delayed
	17	+from utils.cache_clear import delete_cache
	18	+
	19	+
	20	+# To slience the MarkupResemblesLocatorWarning of bs4, which is annoying since I KNOW EVERYTHING IS HTML
	21	+warnings.filterwarnings("ignore", category=MarkupResemblesLocatorWarning, module='bs4')
	22	+
	23	+HEADER = {"User-Agent": "Mozilla/119.0 (Windows NT 10.0; Win64; x64) Chrome/98.0.4758.102"}
	24	+
	25	+
	26	+# Function to remove tags
	27	+def remove_tags(html):
	28	+ # parse html content
	29	+ soup = bs4.BeautifulSoup(html, "html.parser")
	30	+
	31	+ for data in soup(['style', 'script']):
	32	+ # Remove tags
	33	+ data.decompose()
	34	+
	35	+ # return data by retrieving the tag content
	36	+ return ' '.join(soup.stripped_strings)
	37	+
	38	+
	39	+def get_url(keyword, start_date, end_date, page_num, by_sim=True):
	40	+ if by_sim:
	41	+ orderBy = "sim"
	42	+ else:
	43	+ orderBy = "recentdate"
	44	+ url = f"https://section.blog.naver.com/Search/Post.naver?pageNo={page_num}&orderBy={orderBy}&startDate={start_date}&endDate={end_date}&keyword={keyword}"
	45	+ return url
	46	+
	47	+
	48	+def get_article_count(plain_html):
	49	+ regex_match_count = regex.search(r'\d(,\d{3}\|d{3})\b건\b<', plain_html)
	50	+ regex_match_count = regex.sub(r'건\b*<', '', regex_match_count.group())
	51	+ match_count = regex_match_count.replace(',', '')
	52	+ return int(match_count)
	53	+
	54	+
	55	+def get_total_page(url):
	56	+ # Chrome WebDriver 설치 경로 가져오기
	57	+ chrome_path = ChromeDriverManager().install()
	58	+ # WebDriver 설정
	59	+ options = webdriver.ChromeOptions()
	60	+ options.add_experimental_option("excludeSwitches", ["enable-automation"])
	61	+ options.add_experimental_option("useAutomationExtension", False)
	62	+ options.add_argument('headless')
	63	+ options.add_argument('window-size=1920x1080')
	64	+ options.add_argument("disable-gpu")
	65	+ # options.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko")
	66	+
	67	+ # WebDriver 초기화
	68	+ driver = webdriver.Chrome(service=ChromeService(chrome_path), options=options)
	69	+ driver.get(url)
	70	+ time.sleep(3)
	71	+ plain_html = driver.page_source
	72	+ article_count = get_article_count(plain_html)
	73	+ total_page = math.ceil(int(article_count) / 7)
	74	+ return total_page
	75	+
	76	+
	77	+def scrap_agent(url):
	78	+ # Chrome WebDriver 설치 경로 가져오기
	79	+ chrome_path = ChromeDriverManager().install()
	80	+ # WebDriver 설정
	81	+ options = webdriver.ChromeOptions()
	82	+ options.add_experimental_option("excludeSwitches", ["enable-automation"])
	83	+ options.add_experimental_option("useAutomationExtension", False)
	84	+ options.add_argument('headless')
	85	+ options.add_argument('window-size=1920x1080')
	86	+ options.add_argument("disable-gpu")
	87	+ # options.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko")
	88	+
	89	+ # WebDriver 초기화
	90	+ driver = webdriver.Chrome(service=ChromeService(chrome_path), options=options)
	91	+ driver.get(url)
	92	+ time.sleep(2)
	93	+ single_page_search_result = driver.find_elements(By.CSS_SELECTOR, ".area_list_search")
	94	+
	95	+ search_result_plain_html = single_page_search_result[0].get_attribute('innerHTML')
	96	+ formatter = bs4.formatter.HTMLFormatter(indent=1)
	97	+ search_result_plain_html_beautified = bs4.BeautifulSoup(search_result_plain_html, 'html.parser').prettify(
	98	+ formatter=formatter)
	99	+
	100	+ search_result_contents = regex.findall(
	101	+ r'(<!-- ngRepeat: post in postSearchCtrl.resultList -->)?'
	102	+ r'(.*?)'
	103	+ r'<!-- end ngRepeat: post in postSearchCtrl.resultList -->',
	104	+ search_result_plain_html_beautified,
	105	+ re.DOTALL
	106	+ )
	107	+
	108	+ ret = {
	109	+ "url": [],
	110	+ "title": [],
	111	+ "blog_name": [],
	112	+ "post_date": [],
	113	+ "text": []
	114	+ }
	115	+ for content in search_result_contents:
	116	+ content = content[1].replace("\n", "")
	117	+ href = regex.search(
	118	+ r"https:\/\/blog\.naver\.com\/[\uAC00-\uD7AFa-zA-Z0-9_-]+\/[\uAC00-\uD7AFa-zA-Z0-9_-]+",
	119	+ content, re.DOTALL)
	120	+ title = regex.search(r'<strong class="title_post">(.*?)<\/strong>', content)
	121	+ text = regex.search(r'<!-- ngIf: post.contents -->(.*?)<\/a>', content)
	122	+ author_name = regex.search(r'<em class="name_author">(.*?)<\/em>', content)
	123	+ post_date = regex.search(r'<span class="date">(.*?)<\/span>', content)
	124	+ href = href.group(0)
	125	+ title = remove_tags(title.group(1))
	126	+ text = remove_tags(text.group(1))
	127	+ author_name = remove_tags(author_name.group(1))
	128	+ post_date = remove_tags((post_date.group(1)))
	129	+
	130	+ ret["url"].append(href)
	131	+ ret["title"].append(title)
	132	+ ret["blog_name"].append(author_name)
	133	+ ret["post_date"].append(post_date)
	134	+ ret["text"].append(text)
	135	+ delete_cache(driver)
	136	+ driver.close()
	137	+ return ret
	138	+
	139	+
	140	+def merge_dicts(dict_list):
	141	+ # Initialize the result dictionary with empty lists
	142	+ result = {
	143	+ "url": [],
	144	+ "title": [],
	145	+ "blog_name": [],
	146	+ "post_date": [],
	147	+ "text": []
	148	+ }
	149	+
	150	+ # Iterate through each dictionary and merge the lists
	151	+ for d in dict_list:
	152	+ for key in result.keys():
	153	+ result[key].extend(d.get(key, []))
	154	+
	155	+ return result
	156	+
	157	+
	158	+def generate_date_range(start_date, end_date, interval):
	159	+ """
	160	+ :param start_date: start date in datetimestr formatted in %Y-%m-%d
	161	+ :param end_date: end date in datetimestr formatted in %Y-%m-%d
	162	+ :param interval: interval of time in DAYS
	163	+ :return: returns list of time interval that will be feeded into naver_blog_scrapper.
	164	+ for example, [[2023-10-01, 2023-10-07],[2023-10-08, 2023-10-14]] for generate_date_range("2023-10-01", "2023-10-14", 7)
	165	+ Also, this function can handle when the time range of start and end date is not perfectly divisible by interval
	166	+ generate_date_range("2023-10-01", "2023-10-14", 10) will produce output [['2023-10-01', '2023-10-10'], ['2023-10-11', '2023-10-14']]
	167	+ """
	168	+
	169	+ # Convert the start and end date strings to datetime objects
	170	+ start = datetime.strptime(start_date, "%Y-%m-%d")
	171	+ end = datetime.strptime(end_date, "%Y-%m-%d")
	172	+
	173	+ # Generate a range of dates from start to end
	174	+ date_ranges = []
	175	+ current_date = start
	176	+ while current_date < end:
	177	+ current_end = min(current_date + timedelta(days=interval - 1), end)
	178	+ date_ranges.append([current_date.strftime("%Y-%m-%d"), current_end.strftime("%Y-%m-%d")])
	179	+ current_date = current_end + timedelta(days=1)
	180	+
	181	+ return date_ranges
	182	+
	183	+def range_of_ranges(start, end, step):
	184	+ ranges = []
	185	+ current_start = start
	186	+
	187	+ while current_start <= end:
	188	+ # Calculate the end of the current range
	189	+ current_end = min(current_start + step - 1, end)
	190	+
	191	+ # Append the current range as a list
	192	+ ranges.append([current_start, current_end])
	193	+
	194	+ # Update the start for the next range
	195	+ current_start += step
	196	+
	197	+ return ranges
	198	+
	199	+
	200	+def naver_blog_scrapper(keyword, start_day, end_day, date_interval, page_step=10, start_page_num=1, browser_thread_count=12):
	201	+ url = get_url(keyword, start_day, end_day, 1)
	202	+ total_page_num = get_total_page(url)
	203	+ print(f"total_page_num : {total_page_num}")
	204	+
	205	+ date_range = generate_date_range(start_day, end_day, date_interval)
	206	+
	207	+
	208	+ for i, date_range in enumerate(date_range):
	209	+ url = get_url(keyword, date_range[0], date_range[1], 1)
	210	+ total_page_num = get_total_page(url)
	211	+ task_pool = range_of_ranges(1, total_page_num+1, page_step)
	212	+ for task in task_pool:
	213	+ def parallel_scraping(keyword):
	214	+ urls = Parallel(n_jobs=-1)(delayed(get_url)(keyword, date_range[0], date_range[1], i) for i in range(task[0], task[1]))
	215	+ results = Parallel(n_jobs=browser_thread_count)(delayed(scrap_agent)(url) for url in urls)
	216	+ return results
	217	+
	218	+ ret = parallel_scraping(keyword)
	219	+ merged_result = merge_dicts(ret)
	220	+ out_df = pd.DataFrame.from_dict(merged_result)
	221	+ if not os.path.exists(f"blog/{keyword}"):
	222	+ os.mkdir(f"blog/{keyword}")
	223	+ out_df.to_csv(f"blog/{keyword}/{date_range[0]}-{date_range[1]}%{task[0]}-{task[1]}.csv", index=False)
	224	+
	225	+
	226	+
	227	+
	228	+if __name__ == "__main__":
	229	+ naver_blog_scrapper("선산읍", "2022-01-01", "2023-10-31", 7, 50, 1, 12)(파일 끝에 줄바꿈 문자 없음)

81b7126

32fe611

naver_cafe_info_gatherer.py

--- naver_cafe_info_gatherer.py

+++ naver_cafe_info_gatherer.py

...	...	@@ -6,6 +6,7 @@
6	6	import pandas as pd
7	7	import os
8	8	import bs4
	9	+from bs4 import MarkupResemblesLocatorWarning
9	10	from datetime import datetime, timedelta
10	11	from selenium import webdriver
11	12	from selenium.webdriver.common.by import By
...	...	@@ -14,6 +15,11 @@
14	15	from tqdm import tqdm
15	16	from joblib import Parallel, delayed
16	17	from utils.cache_clear import delete_cache
	18	+import warnings
	19	+
	20	+
	21	+# To slience the MarkupResemblesLocatorWarning of bs4, which is annoying since I KNOW EVERYTHING IS HTML
	22	+warnings.filterwarnings("ignore", category=MarkupResemblesLocatorWarning, module='bs4')
17	23
18	24	HEADER = {"User-Agent": "Mozilla/119.0 (Windows NT 10.0; Win64; x64) Chrome/98.0.4758.102"}
19	25
...	...	@@ -42,6 +48,7 @@
42	48	options = webdriver.ChromeOptions()
43	49	options.add_experimental_option("excludeSwitches", ["enable-automation"])
44	50	options.add_experimental_option("useAutomationExtension", False)
	51	+ options.add_argument('--blink-settings=imagesEnabled=false')
45	52	options.add_argument('headless')
46	53	options.add_argument('window-size=1920x1080')
47	54	options.add_argument("disable-gpu")
...	...	@@ -89,7 +96,7 @@
89	96	}
90	97	for content in search_result_contents:
91	98	content = content.replace("\n", "")
92		- href = regex.search(r"https:\/\/cafe\.naver\.com\/[\uAC00-\uD7AFa-zA-Z0-9_]+\?iframe_url\=[\uAC00-\uD7AFa-zA-Z0-9\.\%_]+", content, re.DOTALL)
	99	+ href = regex.search(r"https:\/\/cafe\.naver\.com\/[\uAC00-\uD7AFa-zA-Z0-9__-]+\?iframe_url\=[\uAC00-\uD7AFa-zA-Z0-9\.\%__-]+", content, re.DOTALL)
93	100	title = regex.search(r'<strong class="title">(.*?)<\/strong>',content)
94	101	text = regex.search(r'<p class="text">(.*?)<\/p>',content)
95	102	cafe_name = regex.search(r'<span class="cafe_name">(.*?)<\/span>',content)
...	...	@@ -147,6 +154,7 @@
147	154	def naver_cafe_scrapper(keyword, step = 100, start_page_num=1, browser_thread_count=12):
148	155	url = get_url(keyword, 1)
149	156	total_page_num = get_total_page(url)
	157	+ print(f"total_page_num : {total_page_num}")
150	158
151	159	for scope in range_of_ranges(start_page_num, total_page_num+1, step):
152	160	def parallel_scraping(keyword):
...	...	@@ -160,7 +168,7 @@
160	168	if not os.path.exists(f"cafe/{keyword}"):
161	169	os.mkdir(f"cafe/{keyword}")
162	170	out_df.to_csv(f"cafe/{keyword}/{scope[0]}-{scope[1]}.csv", index=False)
163		- pass
	171	+
164	172
165	173	if __name__ == "__main__":
166		- naver_cafe_scrapper("구미시장", start_page_num=4901) (파일 끝에 줄바꿈 문자 없음)
	174	+ pass (파일 끝에 줄바꿈 문자 없음)

81b7126

32fe611

utils/cache_clear.py

--- utils/cache_clear.py

+++ utils/cache_clear.py


     actions.perform()
     sleep(1)
     driver.switch_to.window(driver.window_handles[0])
-    print("Succesfully cleared the browsing data")
+    # print("Succesfully cleared the browsing data")
 
 # delete_cache(driver)

Add a comment

Open 0
Closed 0

List

...	...	@@ -27,6 +27,6 @@
27	27	actions.perform()
28	28	sleep(1)
29	29	driver.switch_to.window(driver.window_handles[0])
30		- print("Succesfully cleared the browsing data")
	30	+ # print("Succesfully cleared the browsing data")
31	31
32	32	# delete_cache(driver) (파일 끝에 줄바꿈 문자 없음)

Delete comment