🛒 View HF's Premium Marketplace ◆ [#1] PURE RAT v4 // BEST HVNC RAT ◆ Private Crypter - Unique Private Stubs ◆ Spoof Call and Spoof SMS set any caller ◆
Home Upgrade Search Memberlist Extras Hacker Tools Award Goals Help Wiki Contact
Hack Forums Tech Coding Software Development › [Python] Save only unique entries to text file

New Reply 
Python 

Save only unique entries to text file
Drazard
Drazard Offline
[n00b@HF:]
*
Registered
Posts:
5
Threads:
1
Credibility:
000
Popularity:
0
βytes:
β 0
Game XP:
0
Mar 13, 2017 07:58 AM (This post was last modified: Mar 13, 2017 09:35 AM by Drazard.)
Hello and thank you for viewing my thread. I am new here to HF and need some advice. Below i have some source code for a script i am trying to write.
The idea is simple, grab the source code from a website, parse ip's from source and dump into a file.

Code
while True:
    try:
        req = requests.get(url, cookies=cookies)
        result = req.text
        soup = str(BeautifulSoup(result, "html.parser"))
        # - return URL source code in a string

        parsedLogs = re.findall("\d+\.\d+\.\d+\.\d+] logged in", soup)
        # - Search Source for IP logs example: "255.255.255.255 logged in"

        f = open("log.txt", "a+")
        HElog = f.read()
        # - Opens and reads the log file
        
        for ip in parsedLogs:
            if ip not in HElog:
                print (ip)
                f.write(ip+'\n')
        # - Each IP found in the souce code is saved to a log file

    except:
        print("ERROR... Restarting script")
    # - The script restarts on error
Output:
Spoiler (Click to View)

Please note ive only been messing around with python for a short while. but i really cant put it down it is a great language.
If anyone could outline where i am going wrong with this i would be so great-full.

The output i am looking for is to only print and log one entry of each ip address found, and append new ip's to the log if this makes sense? the script is to run in a loop scanning the page over and over catching new entries.
(╯°□°）╯︵ ┻━━┻
(╯°□°）╯︵ ┻━━┻ Offline
i code stuff
***
HF Ub3r
Posts:
3,747
Threads:
231
Credibility:
000
Popularity:
0
βytes:
β 0
Game XP:
0
Mar 13, 2017 08:12 AM
Delete your log file and run it once. Does it look fine?

You're appending (the "a" in your file opener) to the end of the file. You may want it to be "w" which overwrites the file.

[Well Vouched] Python Programming Service - Automate Your Life
[Image: ouURNlq.png]
Drazard
Drazard Offline
[n00b@HF:]
*
Registered
Posts:
5
Threads:
1
Credibility:
000
Popularity:
0
βytes:
β 0
Game XP:
0
Mar 13, 2017 08:16 AM (This post was last modified: Mar 13, 2017 08:29 AM by Drazard.)
Sorry, i should explain further.

The script is written to run 24/7 and check the web source for new entries and save the log to a file, appending all new entries.

The ip addresses in the websource disappear after a few seconds and i am trying to harvest them, if this makes sense? so i require the code to be looped and constantly parse the source grabbing it all, sifting through and only adding new ip addresses to the log file.

EDIT:

if ip not in HElog:
print (ip)
f.write(ip+'\n')

This should only write the ip to the text file if its NOT already there. but its not wroking, so i am doing something wrong

EDIT2:

Actually, i understand why it is still dupicating. using "for ip in HElog:" it checks both ip addresses against each in the list and adds the non duplicate, and then goes to the second line and adds the non duplicate again. so my syntax is incorrect for what i am trying to do.

I need it to check all the ip addresses in the log file against the new ones. what is the best way to read an entire file without using too much memory?
Goodies
Goodies Offline
Rewriting Intro to Pentesting
*****
HF Ub3r
Posts:
12,586
Threads:
755
Credibility:
000
Popularity:
598
βytes:
β 5,520
Game XP:
0
Mar 13, 2017 12:37 PM
1) use a set to make a collection unique before writing to file.
2) use a pythonic storage mechanism like shelves and store a set in there.

Depending on how you want it stored...
SRPP Discord Server

I am a professional network/cloud security architect.
My Old Tutorials
Drazard
Drazard Offline
[n00b@HF:]
*
Registered
Posts:
5
Threads:
1
Credibility:
000
Popularity:
0
βytes:
β 0
Game XP:
0
Mar 13, 2017 03:42 PM (This post was last modified: Mar 13, 2017 03:56 PM by Drazard.)
i have no idea what im doing. this isnt working.

Isnt there simple way to read a file and check the whole file for a string??
Code
# - Opens and reads the log file
        stuff = set()
        for ip in parsedLogs:
                stuff.add(ip)
                print(stuff)


Code
Python 3.5.3rc1 (v3.5.3rc1:de530d7f21c0, Jan  2 2017, 06:29:54) [MSC
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
{'238.65.255.57', '193.171.10.214', '229.4.158.187'}
(╯°□°）╯︵ ┻━━┻
(╯°□°）╯︵ ┻━━┻ Offline
i code stuff
***
HF Ub3r
Posts:
3,747
Threads:
231
Credibility:
000
Popularity:
0
βytes:
β 0
Game XP:
0
Mar 13, 2017 05:03 PM
Code
with open("file.txt", "r") as f:
    file_contents = f.read()

for foo in bar:
    if foo not in file_contents:
        result.append(foo)

Something like that?

[Well Vouched] Python Programming Service - Automate Your Life
[Image: ouURNlq.png]
Drazard
Drazard Offline
[n00b@HF:]
*
Registered
Posts:
5
Threads:
1
Credibility:
000
Popularity:
0
βytes:
β 0
Game XP:
0
Mar 13, 2017 10:03 PM (This post was last modified: Mar 14, 2017 05:45 AM by Drazard.)
(Mar 13, 2017 05:03 PM)(╯°□°）╯︵ ┻━━┻ Wrote:  
Code
with open("file.txt", "r") as f:
    file_contents = f.read()

for foo in bar:
    if foo not in file_contents:
        result.append(foo)

Something like that?


Code
req = requests.get(url, cookies=cookies)
        result = req.text
        soup = str(BeautifulSoup(result, "html.parser"))
        # - return URL source code in a string

        parsedLogs = re.findall("\d+\.\d+\.\d+\.\d+", soup)
        # - Search Source for IP logs example: "255.255.255.255 logged in"
        ips = str(parsedLogs)
        # - Opens and reads the log file
        with open("log.txt", "r") as f:
                        file_contents = f.read()

        for ip in parsedLogs:
                if ip not in file_contents:
                                parsedLogs.append(ip)
                                print (parsedLogs)
                                f = open("log.txt", "a+")
                                f.write(ip+'\n')

Returns the same:

Spoiler (Click to View)



EDIT:

I aplogise, i have changed the code slightly after re reading all your suggestions and come to a reasonably good outcome.

Code
with open("log.txt", "r") as f:
                        file_contents = f.read()

        for ip in parsedLogs:
                if ip not in file_contents:
                        parsedLogs.append(ip)
                        #print (parsedLogs)
                        f = open("log.txt", "w+")
                        f.write(file_contents+ip+'\n')

Its not perfect, and any tweaking advice would be great but it does work.

as it is now i have changed the file to re-writable and added that it writes the contents of the file + the new entries, effectively adding only the new ones. so far it seems successful :D

Thank you for all your help.
Goodies
Goodies Offline
Rewriting Intro to Pentesting
*****
HF Ub3r
Posts:
12,586
Threads:
755
Credibility:
000
Popularity:
598
βytes:
β 5,520
Game XP:
0
Mar 14, 2017 11:07 AM
Did you not do what I said? Put the data into a set first. This will make it unique.
SRPP Discord Server

I am a professional network/cloud security architect.
My Old Tutorials
Drazard
Drazard Offline
[n00b@HF:]
*
Registered
Posts:
5
Threads:
1
Credibility:
000
Popularity:
0
βytes:
β 0
Game XP:
0
Mar 14, 2017 09:32 PM
(Mar 14, 2017 11:07 AM)Goodies Wrote:  Did you not do what I said? Put the data into a set first. This will make it unique.

I tried your method using sets, but not enough ws explained on how to use the set module, besides i worked out how to do what i wanted using text files.

:D
Next Oldest | Next Newest
New Reply 


Possibly Related Threads...
Thread / Author Replies Views Last Post
[Java] by KatasAug 23, 2021 07:50 AM 1 150 Aug 23, 2021 07:50 AM
Last Post: john__john
[C] by pebblakOct 15, 2019 05:24 AM 6 308 Oct 15, 2019 05:24 AM
Last Post: Joseph Ducreux
[C] by VoxiFeb 27, 2019 03:55 PM 3 260 Feb 27, 2019 03:55 PM
Last Post: Voxi