+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [Python] A simple pastebin written in Python (/showthread.php?tid=5482860)
Pages: Pages
1
2
A simple pastebin written in Python - Fudster - Dec 3, 2016
I made a simple pastebin. You can view and install it here
Its released under the MIT License
Feedback is welcome, from both a coding and user standpoint, Coding changes, feature requests etc.
* - Ensky - Dec 3, 2016
Just checked it, it's pretty damn cool to say the least. I've encountered a bug however:
The "paste" only contains the first character of any data I've inputted.
* - Fudster - Dec 3, 2016
(Dec 3, 2016 02:44 AM)Ensky_ Wrote: Just checked it, it's pretty damn cool to say the least. I've encountered a bug however:
The "paste" only contains the first character of any data I've inputted.
That...can't be possible.
I guess this is what I get for changing code and not testing (It worked before) . Let me look into it.
EDIT: Confirmed bug. Let me look into it. Give me a bit. Shouldn't take long.
EDIT2: https://github.com/Fudster/python-pastebin/commit/a9540b994d05bfd5cc10abce3fa3c06edc9f8b62 fixed.
* - Ensky - Dec 3, 2016
(Dec 3, 2016 02:47 AM)Fudster Wrote: That...can't be possible.
I guess this is what I get for changing code and not testing (It worked before) . Let me look into it.
EDIT: Confirmed bug. Let me look into it. Give me a bit. Shouldn't take long.
Works perfectly, good work man!
* - Fudster - Dec 3, 2016
(Dec 3, 2016 02:56 AM)Ensky_ Wrote: Works perfectly, good work man!
Thanks. One of my first python scripts.
One thing I gotta do is figure out how to allow the pastebin title to be changed (header and footer)
I can do a config variable, Yes just haven't figured out the best way to implent it.
I will get around to it tomorrow, Its late. Bedtime for me.
* - Ensky - Dec 3, 2016
(Dec 3, 2016 02:59 AM)Fudster Wrote: Thanks. One of my first python scripts.
One thing I gotta do is figure out how to allow the pastebin title to be changed (header and footer)
I can do a config variable, Yes just haven't figured out the best way to implent it.
I will get around to it tomorrow, Its late. Bedtime for me.
Try to add the amount of lines that are in the code, that'd be perfect
* - Fudster - Dec 3, 2016
(Dec 3, 2016 03:00 AM)Ensky_ Wrote: Try to add the amount of lines that are in the code, that'd be perfect
Huh? You got my lost. I have no idea what oyu are referring to.
Examples?
* - Ensky - Dec 3, 2016
(Dec 3, 2016 03:03 AM)Fudster Wrote: Huh? You got my lost. I have no idea what oyu are referring to.
Examples?
On the real pastebin, there's line numbers if you paste anything, useful for pasting code.
* - Fudster - Dec 3, 2016
(Dec 3, 2016 03:07 AM)Ensky_ Wrote: On the real pastebin, there's line numbers if you paste anything, useful for pasting code.
OH, line numbers. I haven't figured that out either.
I might be able to run a for loop for each line using \n
I was not able to find any JS scripts that actually have numbering on them.
* - Goodies - Dec 5, 2016
Pretty simple. No major complaints, honestly, provided you understand that this can only be a very small server. It'd be perfect for hosting personal pastes for you and your coworkers, but should never go beyond that.
It's certainly not scalable for two main reasons:
~ You're using Flask, which is definitely not designed to be for large applications
~ You're synchronously using a database (which is a definite no-no in writing servers).
You would have substantially better and more scalable performance if you use the aiohttp.web module (which boasts of a Flask-like API) for hosting the site and something like aioodbc to asynchronously query the database.
* - Fudster - Dec 6, 2016
(Dec 5, 2016 02:07 AM)Goodies Wrote: Pretty simple. No major complaints, honestly, provided you understand that this can only be a very small server. It'd be perfect for hosting personal pastes for you and your coworkers, but should never go beyond that.
It's certainly not scalable for two main reasons:
~ You're using Flask, which is definitely not designed to be for large applications
~ You're synchronously using a database (which is a definite no-no in writing servers).
You would have substantially better and more scalable performance if you use the aiohttp.web module (which boasts of a Flask-like API) for hosting the site and something like aioodbc to asynchronously query the database.
Thanks for the feedback.
I dont plan making it a huge site. But since you mentioned it I will look into it and make a new version of the pastebin. Just for learning purposes.
* - Blackjack - Dec 7, 2016
(Dec 5, 2016 02:07 AM)Goodies Wrote: Pretty simple. No major complaints, honestly, provided you understand that this can only be a very small server. It'd be perfect for hosting personal pastes for you and your coworkers, but should never go beyond that.
It's certainly not scalable for two main reasons:
~ You're using Flask, which is definitely not designed to be for large applications
~ You're synchronously using a database (which is a definite no-no in writing servers).
You would have substantially better and more scalable performance if you use the aiohttp.web module (which boasts of a Flask-like API) for hosting the site and something like aioodbc to asynchronously query the database.
I know several large sites that use Flask with no problems - no idea what you're talking about.
* - Fudster - Dec 7, 2016
(Dec 7, 2016 05:00 AM)Yani Wrote: I have no experience with Python, but I do have a few points.
- Add something against ID collision, keep generating new IDs if the generated one is already in the database, until you get a unique one
- Make the 'id' and 'password' columns in the db VARCHARs
- You should hash the passwords
Good luck 👍
I did the 1st and 3rd one.
Why should I make the VARCHARs?
* - Goodies - Dec 7, 2016
(Dec 7, 2016 04:47 AM)Blackjack Wrote: I know several large sites that use Flask with no problems - no idea what you're talking about.
MOST large sites that I've seen with Python as their back-end use Django as their main web app framework and Flask is used as some kind of an API or smaller, supplementary application.
Don't take my word for it, though. Take a look at the actual Design Decisions in Flask on their official documentation.
- Thread Locals
Flask uses thread local objects (context local objects in fact, they support greenlet contexts as well) for request, session and an extra object you can put your own things on (g). Why is that and isn’t that a bad idea?
Yes it is usually not such a bright idea to use thread locals. They cause troubles for servers that are not based on the concept of threads and make large applications harder to maintain. However Flask is just not designed for large applications or asynchronous servers. Flask wants to make it quick and easy to write a traditional web application.
In other words, it has a lot of overhead for thread creation and TLS.
Anyone who uses Flask for large-scale applications almost certainly (and absolutely should) follow their advice on Becoming Big which is also on their official documentation.
If you actually want to test it yourself, it's trivially easy. I'm not sure if you have Pycharm Professional, but I'd highly recommend it, particularly for performance testing.
Anyways, I wrote a quick server that simulates being on the internet (as opposed to running on localhost) and performing a time-consuming, CPU-bound task such as database queries by sleeping for a quarter of a second on each request.
Simple Flask Server (Click to View)
And a simple async client using aiohttp and a semaphore to prevent more than just 15 concurrent clients.
Async HTTP Client Example (Click to View)
With just 40 URLs allowing no more than 20 concurrent clients (which is fairly small in size). I get performance like this:
Code
Querying 40 URLs...
3.794 1.775 1.020 2.532 3.036 3.541 4.296
4.802 5.043 5.041 5.039 5.040 5.038 5.037
5.035 5.035 5.035 5.033 5.030 5.033 0.272
0.518 0.770 1.272 1.524 2.027 2.279 2.783
3.288 4.044 4.549 5.053 5.043 5.041 5.040
Done.Compare that vs the exact same thing done with aiohttp. Each request is asynchronously handled on a single thread with non-blocking sockets, and sleeps asynchronously, too:
Simple aiohttp server. (Click to View)
Look at the difference in the output:
Code
Querying 40 URLs...
0.284 0.269 0.273 0.285 0.275 0.280 0.276
0.280 0.250 0.245 0.267 0.263 0.259 0.255
0.252 0.249 0.247 0.250 0.484 0.483 0.282
0.279 0.281 0.270 0.275 0.271 0.286 0.274
0.284 0.273 0.276 0.270 0.248 0.245 0.266
Done.Every request gets responded to in the correct amount of time because of its asynchronous nature. To be quite honest, I wouldn't completely trust aiohttp to handle tens of thousands of users, particulraly because it doesn't have a template engine and is event driven but I damn sure wouldn't trust flask to do it, either. Flask is good for API's as it handles JSON very well, but anything with serious database usage would need to be asynchronous for me to consider it.
* - Fudster - Dec 7, 2016
(Dec 7, 2016 07:50 PM)Yani Wrote: It's better practice.
Here's some more info: https://stackoverflow.com/questions/2023481/mysql-large-varchar-vs-text
Hm. Okay do I have to set a size?
(Dec 7, 2016 08:06 PM)Goodies Wrote: Every request gets responded to in the correct amount of time because of its asynchronous nature. To be quite honest, I wouldn't completely trust aiohttp to handle tens of thousands of users, particulraly because it doesn't have a template engine and is event driven but I damn sure wouldn't trust flask to do it, either. Flask is good for API's as it handles JSON very well, but anything with serious database usage would need to be asynchronous for me to consider it.
Well like I said. I might look into it.
But I am not expecting huge usage on this script. Maybe personal and small IRC channels. (under 1,000 users)
I am doing this mostly for learning. I am about to move to a deal or no deal game soon.