[Doomslayer]

Hello,

I am an intermediate to advanced python programmer, however I am a beginner when it comes to web development and databases. I’m looking to have a server socket that will accept connections from multiple clients, and display all active connections on a dynamic table on a Flask webpage.

My idea is to store the connections in a SQL database and display each entry in the table. However I have a few problems I have been pondering.

Each socket connection will be handled in a separate thread. What is the best way to store all active connections, and remove them if the client closes the connection?

Would it be better to store a list of threads? Or, create a class for the clients with a thread attribute and store them in a list?

My code for detecting unexpected closed connections was pretty messy. Using try except blocks surrounding the entire thread function. I’m sure there is a better way to determine when the thread is closed, and to remove that connection from the list.

Thirdly, what would be the best way to implement this in flask? Should I make the listener a class and create the instance in my flask files? Or should I simply import listener.py and get the list of connections from there?

Last question is about the database. Should I create the database connection in my listener file? Or should I use the existing database connection from flask?

I know that there would be many solutions for this, just looking for a way to write cleaner code. I know there was a lot of questions, so here’s a TLDR:

1. What is the best way to store active client socket connections in a list?

2. What is the best way to handle an unexpected close from the client, and remove the connection from the list.

3. What is the best way to display the active connection on a dynamic table in flask?

[synchronizing]

Out of curiosity - why do you want to utilize multiple ports? This sounds like a XY problem to me.

[Doomslayer]

Out of curiosity - why do you want to utilize multiple ports? This sounds like a XY problem to me.

I do not want to utilize multiple ports.

I want to run a server socket that listens on one port. There will be multiple clients connecting via this port. I simply need a way to dynamically store all of these connections. When a client connects it gets added to the list. When a client disconnects that connection is removed.

[synchronizing]

I do not want to utilize multiple ports.

Understood. I got myself confused somewhere reading your thread.

I want to run a server socket that listens on one port. There will be multiple clients connecting via this port. I simply need a way to dynamically store all of these connections. When a client connects it gets added to the list. When a client disconnects that connection is removed.

Regarding this and the fact you mentioned Flask I'm assuming you are using the standard HTTP/1.0 protocol (unless you did a bit of hacking on Werkzeug's backend). Via this protocol, specifically HTTP/1.0 (*not HTTP/1.1*), the connection is not kept alive by the server by default. In other words, the client sends a request to the server, the server replies, and then the server/client connection closes on both end automatically (headers can be set that some clients might respect, but I won't get into it here).

A persistent connection via HTTP is only useful if there some useful meaningful reason for it (e.g. continuous request for multiple static resources on the server - css, js, etc. - this connection, however, normally close as soon as the client gets all of the resources he/she requests). In most other cases, however, HTTP/1.x is not used for such things and a different protocol is prefered (i.e. WebSockets) for back-and-forth communication between client/server (e.g. chat communication, or some other type of continuous data stream).

Let me know if I can clarify anything. In short, what you are trying to do (maintain a list of active clients connected to the server) is technically possible, but likely not how you imagine it to be. Clients (via HTTP and Flask) connect to the server for a brief period of time, request a resource, and then leave.

[Doomslayer]

Understood. I got myself confused somewhere reading your thread.


Regarding this and the fact you mentioned Flask I'm assuming you are using the standard HTTP/1.0 protocol (unless you did a bit of hacking on Werkzeug's backend). Via this protocol, specifically HTTP/1.0 (*not HTTP/1.1*), the connection is not kept alive by the server by default. In other words, the client sends a request to the server, the server replies, and then the server/client connection closes on both end automatically.

A persistent connection via HTTP is only useful if there some useful meaningful reason for it. In most cases, however, HTTP/1.x is not used for such things, and a different protocol is prefered (i.e. WebSockets).

I was not asking about HTTP connections. Sorry if it was confusing because I went on about flask, but I will try to make it clear.

The socket I was talking about is just a simple TCP server socket. There will be multiple clients connecting to that TCP server and those are the connections I want to store.

The flask webapp is just going to act as a control panel. I would like to display all active connections on that socket in a table. Of course, I would need to store the connections somewhere. My first thought is to use a List. It’s easy to store new connections, but the problem comes when the clients close the connection. When this happens, the connection must be removed from the list, and therefore no longer be displayed on the table in flask.

It just gets messy since each client connection is handled in a separate thread. This means once a client closes the connection, I would need to stop the thread, and then remove the connection from the list. I can’t think of any elegant solutions to do this.

Also, I know that to display the table in flask it would probably be better to store in a SQL database instead of a list. I just want to use a list for now to get it working, and then I would easily be able to implement it in SQL.

[synchronizing]

I was not asking about HTTP connections. Sorry if it was confusing because I went on about flask, but I will try to make it clear.

Oh, understood now. :)

The socket I was talking about is just a simple TCP server socket. There will be multiple clients connecting to that TCP server and those are the connections I want to store.

...

It just gets messy since each client connection is handled in a separate thread. This means once a client closes the connection, I would need to stop the thread, and then remove the connection from the list. I can’t think of any elegant solutions to do this.

Just to understand the problem set further, let me ask:

• Is the Flask server and the TCP socket server running on the same process with multithread?
  


• What module are you using for creating the socket server? (e.g. socketserver, sockets, asyncio, etc.)
  


• Are you purposely handling each client in a different thread? (this question is dependent on the one above)
  

My first thought is to use a List. It’s easy to store new connections, but the problem comes when the clients close the connection. When this happens, the connection must be removed from the list, and therefore no longer be displayed on the table in flask.

My first instinct is that a list should work fine as long as whatever module you are using implements the __hash__ dunder method on whatever object you are storing on the list. With __hash__ you should be able to easily lst.remove(connection) assuming you can catch the connection object when the user closes the connection with the server. When this happens the Flask server should now reply (on say /connections) with the new updated list - this, of course, will not be reflected on a client accessing the Flask webpage until he/she refreshes their page.

Regarding a database - yes, this would work, but, you will still face the same issue if you were using a memory-based data structure to contain the information (e.g. list), since you will need to keep track of the connections one way or another - the main difference being that a database will be more restrictive as (a) you need to manually flush it every time you restart the program, and (b) you will need to store the client information via some primitive data type (string, int, etc.) that maps to the connections. Arguably, a database is more overhead than what you might likely want assuming you are running only one process (albeit, multi-threaded).

[Doomslayer]

Oh, understood now. :)


Just to understand the problem set further, let me ask:

• Is the Flask server and the TCP socket server running on the same process with multithread?
  


• What module are you using for creating the socket server? (e.g. socketserver, sockets, asyncio, etc.)
  


• Are you purposely handling each client in a different thread? (this question is dependent on the one above)
  

My first instinct is that a list should work fine as long as whatever module you are using implements the __hash__ dunder method on whatever object you are storing on the list. With __hash__ you should be able to easily lst.remove(connection) assuming you can catch the connection object when the user closes the connection with the server. When this happens the Flask server should now reply (on say /connections) with the new updated list - this, of course, will not be reflected on a client accessing the Flask webpage until he/she refreshes their page.

Regarding a database - yes, this would work, but, you will still face the same issue if you were using a memory-based data structure to contain the information (e.g. list), since you will need to keep track of the connections one way or another - the main difference being that a database will be more restrictive as (a) you need to manually flush it every time you restart the program, and (b) you will need to store the client information via some primitive data type (string, int, etc.) that maps to the connections. Arguably, a database is more overhead than what you might likely want assuming you are running only one process (albeit, multi-threaded).

1. Most likely separate. It could just be easier to start a second process for the listener and then access the SQLite database.
2. I’m using socket
3. Yes. The main thread accepts connections and starts a new thread for each client.

Okay so every time I accept a connection, I can copy the socket object to the list before starting a new thread, but how will I remove it? What’s the best way to detect a closed connection? Before I just used a bunch of try except which was messy and unreliable.

[synchronizing]

1. Most likely separate. It could just be easier to start a second process for the listener and then access the SQLite database.
2. I’m using socket
3. Yes. The main thread accepts connections and starts a new thread for each client.

Okay so every time I accept a connection, I can copy the socket object to the list before starting a new thread, but how will I remove it? What’s the best way to detect a closed connection? Before I just used a bunch of try except which was messy and unreliable.

Here is a quick example:



In the above, you have a Flask and socket server running. The socket server can take up to 5 connections at the same time, and handle them separately. The connections list simply keeps track of IP addresses and port numbers that have connected to the server - this is what Flask will server. The socket server expects a \0 anywhere in the message to assume the client is done with the communication. A basic client looks like so:



The client connects and then hangs for 10 seconds before sending the EOL signal to terminate the connection. Running the server above and going to the page we see:



When the first client connects (and hangs):



When the second client connects while the first hangs:



When the clients individually leave they are removed from list. To clarify, I'm running two instances of the client code one after the other. Let me know if you need clarification on anything, but hopefully this helps out.

[Doomslayer]

Here is a quick example:



In the above, you have a Flask and socket server running. The socket server can take up to 5 connections at the same time, and handle them separately. The connections list simply keeps track of IP addresses and port numbers that have connected to the server - this is what Flask will server. The socket server expects a \0 anywhere in the message to assume the client is done with the communication. A basic client looks like so:



The client connects and then hangs for 10 seconds before sending the EOL signal to terminate the connection. Running the server above and going to the page we see:



When the first client connects (and hangs):



When the second client connects while the first hangs:



When the clients individually leave they are removed from list. To clarify, I'm running two instances of the client code one after the other. Let me know if you need clarification on anything, but hopefully this helps out.

This is just what I was looking for. Thank you so much for your help.

[synchronizing]

This is just what I was looking for. Thank you so much for your help.

My pleasure, glad it was of help.