mischbeck/CollectUSSenatorTweets
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

adds comments. changes logfile format to .log

Browse Source
This commit is contained in:
Michael Beck 2023-06-23 20:34:46 +02:00
parent dc2e17cc2f
commit 02c3d055bd
1 changed files with 5 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
collect.py
View File

@ -7,8 +7,6 @@ Created on Thu Jun 8 01:08:21 2023
# https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html # https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
Following files are necessary: Following files are necessary:
config.py
Used to configure everything that's needed for this script.
funs/TimeSlice.py funs/TimeSlice.py
Function get_Tslices slices the defined timespan in config.py into N Function get_Tslices slices the defined timespan in config.py into N
slices. Is necessary due to possible blocking of requests by twitter. slices. Is necessary due to possible blocking of requests by twitter.
@ -17,6 +15,8 @@ Following files are necessary:
Function deDupe reads each line of inFile and removes duplicate lines. Function deDupe reads each line of inFile and removes duplicate lines.
A file outFile is saved without the duplicate lines. Generates A file outFile is saved without the duplicate lines. Generates
"keywords.txt". "keywords.txt".
funs/Scrape.py
scrapes using snscrape.modules.twitter. See docstring.
data/keywords-raw.txt data/keywords-raw.txt
Contains all keywords that are used to detect whether a tweet contains Contains all keywords that are used to detect whether a tweet contains
information about Covid19. information about Covid19.
@ -152,8 +152,8 @@ from funs.Scrape import scrapeTweets
# Create logfile & log all outputs # Create logfile & log all outputs
# there are three logfile types to be found in /log. # there are three logfile types to be found in /log.
# should be self explanatory. # should be self explanatory.
logfilen = logfile + datetime.now().strftime(fTimeFormat) + ".txt" logfilen = logfile + datetime.now().strftime(fTimeFormat) + ".log"
logfileErrors = logfile + datetime.now().strftime(fTimeFormat) + "_err" + ".txt" logfileErrors = logfile + datetime.now().strftime(fTimeFormat) + "_err" + ".log"
sys.stderr = open(logfileErrors, "w") sys.stderr = open(logfileErrors, "w")
sys.stdout = open(logfilen, "w") sys.stdout = open(logfilen, "w")
@ -240,7 +240,7 @@ for handle in accounts:
suffix = tslice['suffix'] suffix = tslice['suffix']
AllFilesList.append(f"Tweets-{handle}{suffix}.csv") AllFilesList.append(f"Tweets-{handle}{suffix}.csv")
# report missing files to "log_*_missing.txt" # report missing files to "log_*_missing.txt"
with open(f"{logfile}"+timeStartScrape.strftime(fTimeFormat)+"_missing.txt", "w") as fout: with open(f"{logfile}"+timeStartScrape.strftime(fTimeFormat)+"_missing.log", "w") as fout:
for file in AllFilesList: for file in AllFilesList:
if file not in tweetfiles: if file not in tweetfiles:
fout.write(f'Missing: {file}.\n') # if file is not in tweetfiles, print error message. fout.write(f'Missing: {file}.\n') # if file is not in tweetfiles, print error message.