(Jan 5, 2019 05:26 PM)Ghxst Wrote:  Are you sending valid cookies and headers?

Capture the requests made with your program so you can compare.

(Jan 6, 2019 03:33 AM)ZyngaChips Wrote:  Yes i am sending valid cookies and headers. I captured the request with Charles, when i repeat the request i am getting same result which is 'f*ck off'.

(Jan 6, 2019 04:08 AM)Hexecutor Wrote:  I personally use Beautiful soup which is really good for your purpose, Session keeps any info and you don’t need much work for cookies or headers :

Code

ua = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'
  s = requests.Session()
  s.headers = {'User-Agent': ua}

Here a full example for tumblr login+scraping

Code

#####    HEXECUTOR      #####
#####  Hackforums.net   #####
#####      Tumbler          #####
#####     jpg,png,gif       #####
#####   DOWNLOADER   #####


import os, requests, time, re
from urllib.request import urlopen
from bs4 import BeautifulSoup as bs




def HexeTumblerDownloader():
  username = input("Email/Username: ")  
  password = input("Password: ")

  ua = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'
  s = requests.Session()
  s.headers = {'User-Agent': ua}
 
  # form_key
  print('Start...')
  r = s.get('https://www.tumblr.com/login')
  formKey = r.text.split('<input type="hidden" name="form_key" value="')[1]
  formKey = formKey.split('"')[0]


  data = {'user[email]': username, 'user[password]': password, 'user[tos]': 1,
          'context': 'login', 'version': 'STANDARD', 'follow': '',
          'http_referer': 'https://www.tumblr.com/login', 'form_key': formKey}


  # login
  print('Logging...')
  r = s.post('https://www.tumblr.com/login', data=data)

  if 'logged_in' not in s.cookies or s.cookies['logged_in'] is not '1':
      print('Error logging in.')
      return
  else:
      print('Successfully logged in.')  

  data = {}
  email = ''
  password = ''
  i=0

  #urls input cycle
  while True:
      page=input("Please paste the URL or Ctrl+z to stop (IT MUST END WITH .com NOT WITH .com/): ")
      maxpage=input("Scanning max pages: ")
      directory=input("Please write the directory Path e.g. C:\\Users\\NAMEUSER\\Desktop\\  IT MUST END WITH \\ to save inside the last directory specified (if empty every pic will be saved into the python folder):")
      pagenumb=1
      while pagenumb <= int(maxpage):
          #the first page doesn't require page number specification (it can cause no contents)
          if pagenumb!=1:
              r = s.get(page+'/page/'+str(pagenumb))
          else:
              r = s.get(page)
  #save jpg png gif
          linkregex = re.compile('([a-z]+:\/\/66(?:[^\/]+\/)*[^\/]+\.(?:jpg|png|gif))')
          matched = linkregex.findall(r.text)
          #sometimes it drops https after the first page so let's use http if the output is empty
          if matched == [] :
              r = s.get(page.replace('https','http')+'/page/'+str(pagenumb))
              matched = linkregex.findall(r.text)
             
          for img in matched:
              print('-------------- Page '+str(pagenumb)+' Pic#'+str(i)+' ---------------')
              if img.endswith('jpg'):
                  extension='.jpg'
              elif img.endswith('png'):
                  extension='.png'
              else:
                  extension='.gif'
              f = urlopen(img)
              with open(directory+'img'+str(i)+extension, 'wb') as w:
                  w.write( f.read() )
              i=i+1
              print(img)
          pagenumb=pagenumb+1
  return


HexeTumblerDownloader()

(Jan 6, 2019 08:46 AM)Infamia Wrote:  some websites have anti-bot protections, if you're sure it's the exact same request then you're probably retarded and don't know to change certain session cookies or session data; otherwise, if there is no session data, then it's probably a protection mechanism of the website itself.