Saturday, August 11, 2007
Python Server Start, Take 2
A couple of months ago, I posted Python Server Start, a simple template for starting implementation of a network server in Python. I got a comment from "dt" suggesting that what I really wanted to use was the standard Python SocketServer module.
Today, I had to write a "real server" in Python, so I finally got around to looking into SocketServer. The documentation wasn't helpful, but the source code for the module was straightforward, so I figured things out pretty quickly.
After work, I decided to create a more generic version of what I'd done while I was on the clock. What follows is my new "starting point" for implementing a server process in Python. It's about 300 lines long, which is a bit large for a "Hello, world!" kind of program, but it has these nifty new features:
- The server starts a daemon process, disconnected from the user's terminal, like it should.
- The server writes to a log file
- It implements a simple protocol between client and server. Basically, the client just sends its command-line arguments to the server, and the server processes the command and sends output back, which the client writes to standard output. (This protocol should, of course, be replaced with whatever protocol your real server has to handle; the template is just in place for testing and demonstration.)
- It can work with TCP/IP sockets, or can use UNIX domain sockets (on platforms that support them).
I've only tried it on Mac OS X and Linux. It will need some work for Windows, but thankfully, I haven't had to do much Windows programming lately, so I'm not going to worry about it.
Making the necessary changes to use a base of ForkingTCPServer, ThreadingTCPServer, ThreadingUnixStreamServer, or other variations is left as an exercise for the reader.
I welcome any suggestions for improvement.
#!/usr/bin/env python """Server start This is a template for a Python-based server daemon derived from SocketServer. Hack it up as needed. This script implements both the server daemon and a command-line client that can issue requests against it. The template client-server protocol is very simple: the client simply sends the command-line arguments to the server, and the server returns output which the client writes to its standard output. Change the protocol as needed for your purposes. The template contains a few UNIXisms. Modification may be needed for a Windows-based server. References: - Source for ServerSocket.py (standard Python module) - Source for BaseHTTPServer.py (standard Python module) - http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731 """ version = '1.0' usage = """usage: %prog [options] command [arg...] commands: start start the server daemon stop stop the server daemon status return server daemon status echo server echoes arguments add A B return A+B Example session: %prog start # starts daemon %prog status # print daemon's status %prog add 15 8 # prints "15 + 8 = 23" %prog stop # stops daemon""" import SocketServer import optparse import os import os.path import resource import socket import sys import tempfile import time # We can use either a TCPServer or a UnixStreamServer (assuming the OS # supports UNIX domain sockets). We just need to define the # appropriate ServerBase class and then customize a few things based # upon which base we're using. #ServerBase = SocketServer.TCPServer ServerBase = SocketServer.UnixStreamServer if ServerBase == SocketServer.TCPServer: # TODO: replace with appropriate port number server_address = ('', 54545) elif ServerBase == SocketServer.UnixStreamServer: # TODO: replace with appropriate socket file path server_address = os.path.join(tempfile.gettempdir(), 'server_socket') # Path to log file # TODO: Change to appropriate path and name server_log = os.path.join(tempfile.gettempdir(), 'server.log') class RequestHandler(SocketServer.StreamRequestHandler): """Request handler An instance of this class is created for each connection made by a client. The Server class invokes the instance's setup(), handle(), and finish() methods. The template implementation here simply reads a single line from the client, breaks that up into whitespace-delimited words, and then uses the first word as the name of a "command." If there is a method called "do_COMMAND", where COMMAND matches the commmand name, then that method is invoked. Otherwise, an error message is returned to the client. """ def handle(self): """Service a newly connected client. The socket can be accessed as 'self.connection'. 'self.rfile' can be used to read from the socket using a file interface, and 'self.wfile' can be used to write to the socket using a file interface. When this method returns, the connection will be closed. """ # Read a single request from the input stream and process it. # TODO: Change as needed for actual client-server protocol. request = self.rfile.readline() if request: self.server.log('request %s: %s', self.connection.getpeername(), request.rstrip()) try: self.process_request(request) except Exception, e: self.server.log('exception: %s' % str(e)) self.wfile.write('Error: %s\n' % str(e)) else: self.server.log('error: unable to read request') self.wfile.write('Error: unable to read request') def process_request(self, request): """Process a request. This method is called by self.handle() for each request it reads from the input stream. This implementation simply breaks the request string into words, and searches for a method named 'do_COMMAND', where COMMAND is the first word. If found, that method is invoked and remaining words are passed as arguments. Otherwise, an error is returned to the client. """ words = request.split() if len(words) == 0: self.server.log('error: empty request') self.wfile.write('Error: empty request\n') return command = words args = words[1:] methodname = 'do_' + command if not hasattr(self, methodname): self.server.log('error: invalid command') self.wfile.write('Error: "%s" is not a valid command\n' % command) return method = getattr(self, methodname) method(*args) def do_stop(self, *args): """Process a 'stop' command""" self.wfile.write('Stopping server\n') self.server.stop() def do_echo(self, *args): """Process an 'echo' command""" self.wfile.write(' '.join(args) + '\n') def do_status(self, *args): """Process a 'status' command""" self.wfile.write('Server Version: %s\n' % version) self.wfile.write('Process ID: %d\n' % os.getpid()) self.wfile.write('Parent Process ID: %d\n' % os.getppid()) self.wfile.write('Server Socket: %s\n' % str(server_address)) self.wfile.write('Server Log: %s\n' % server_log) def do_add(self, a, b): """Process an 'add' command""" answer = int(a) + int(b) self.wfile.write('%s + %s = %s\n' % (a, b, answer)) class Server(ServerBase): """Server implementation """ def __init__(self, server_address): """Constructor""" self.__daemonize() if ServerBase == SocketServer.UnixStreamServer: # Delete the socket file if it already exists if os.access(server_address, 0): os.remove(server_address) ServerBase.__init__(self, server_address, RequestHandler) def log(self, format, *args): """Write a message to the server log file""" try: message = format % args timestamp = time.strftime('%Y-%m-%d %H:%M:%S') f = open(server_log, 'a+') f.write('%s %s\n' % (timestamp, message)) f.close() except Exception, e: print str(e) def serve_until_stopped(self): """Serve requests until self.stop() is called. This is an alternative to BaseServer.serve_forever() """ self.log('started') self.__stopped = False while not self.__stopped: self.handle_request() self.log('stopped') def stop(self): """Stop handling requests. Calling this causes the server to drop out of serve_until_stopped(). """ self.__stopped = True def __daemonize(self): """Create daemon process. Based upon recipe provided at http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731 """ UMASK = 0 WORKDIR = '/' MAXFD = 1024 if hasattr(os, 'devnull'): REDIRECT_TO = os.devnull else: REDIRECT_TO = '/dev/null' try : if os.fork() != 0: os._exit(0) os.setsid() if os.fork() != 0: os._exit(0) os.chdir(WORKDIR) os.umask(UMASK) except OSError, e: self.log('exception: %s %s', e.strerror, e.errno) raise Exception, "%s [%d]" % (e.strerror, e.errno) except Exception, e: self.log('exception: %s', str(e)) maxfd = resource.getrlimit(resource.RLIMIT_NOFILE) if maxfd == resource.RLIM_INFINITY: maxfd = MAXFD for fd in range(0, maxfd): try: os.close(fd) except OSError: pass os.open(REDIRECT_TO, os.O_RDWR) os.dup2(0, 1) os.dup2(0, 2) def run_server(options, args): """Run a server daemon in the current process.""" svr = Server(server_address) svr.serve_until_stopped() svr.server_close() def do_request(options, args): """Send request to the server and process response.""" if ServerBase == SocketServer.UnixStreamServer: s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) elif ServerBase == SocketServer.TCPServer: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Send request # TODO: Change as needed for actual client-server protocol s.connect(server_address) s.sendall(' '.join(args) + '\n') # Print response # TODO: Change as needed for actual client-server protocol sfile = s.makefile('rb') line = sfile.readline() while line: print line, line = sfile.readline() # # MAIN # if __name__ == '__main__': optparser = optparse.OptionParser(usage=usage, version=version) (options, args) = optparser.parse_args() if len(args) == 0: optparser.print_help() sys.exit(-1) if args == 'start': run_server(options, args[1:]) else: do_request(options, args)
> The server writes to a log file
These days I write my server programs (Python, C, even Perl on the odd occasion) to run under djb's daemontools
(http://cr.yp.to/daemontools.html). In particular, supervise, multilog and the aoftlimit tools which takes care of both of these issues, and restarting the program should it abort. With logging the policy (i.e. log file rotation, maximum size of log file) is separated so you don't have to worry about log file management. Solaris has a similar mechanism with 10, I believe.
I do have a Python wrapper to do the "fork/setsid/etc" dance where daemontools isn't installed, but thankfully I have more say in how systems are configured these days ...