Saturday, August 11, 2007


Python Server Start, Take 2

A couple of months ago, I posted Python Server Start, a simple template for starting implementation of a network server in Python. I got a comment from "dt" suggesting that what I really wanted to use was the standard Python SocketServer module.

Today, I had to write a "real server" in Python, so I finally got around to looking into SocketServer. The documentation wasn't helpful, but the source code for the module was straightforward, so I figured things out pretty quickly.

After work, I decided to create a more generic version of what I'd done while I was on the clock. What follows is my new "starting point" for implementing a server process in Python. It's about 300 lines long, which is a bit large for a "Hello, world!" kind of program, but it has these nifty new features:

I've only tried it on Mac OS X and Linux. It will need some work for Windows, but thankfully, I haven't had to do much Windows programming lately, so I'm not going to worry about it.

Making the necessary changes to use a base of ForkingTCPServer, ThreadingTCPServer, ThreadingUnixStreamServer, or other variations is left as an exercise for the reader.

I welcome any suggestions for improvement.

#!/usr/bin/env python

"""Server start

This is a template for a Python-based server daemon derived from
SocketServer.  Hack it up as needed.

This script implements both the server daemon and a command-line
client that can issue requests against it.  The template client-server
protocol is very simple: the client simply sends the command-line
arguments to the server, and the server returns output which the
client writes to its standard output.  Change the protocol as needed
for your purposes.

The template contains a few UNIXisms.  Modification may be needed for
a Windows-based server.

- Source for (standard Python module)
- Source for (standard Python module)


version = '1.0'

usage = """usage: %prog [options] command [arg...]

  start     start the server daemon
  stop      stop the server daemon
  status    return server daemon status
  echo      server echoes arguments
  add A B   return A+B

Example session:
  %prog start     # starts daemon
  %prog status    # print daemon's status
  %prog add 15 8  # prints "15 + 8 = 23"
  %prog stop      # stops daemon"""

import SocketServer
import optparse
import os
import os.path
import resource
import socket
import sys
import tempfile
import time

# We can use either a TCPServer or a UnixStreamServer (assuming the OS
# supports UNIX domain sockets).  We just need to define the
# appropriate ServerBase class and then customize a few things based
# upon which base we're using.

#ServerBase = SocketServer.TCPServer
ServerBase = SocketServer.UnixStreamServer
if ServerBase == SocketServer.TCPServer:
    # TODO: replace with appropriate port number
    server_address = ('', 54545)
elif ServerBase == SocketServer.UnixStreamServer:
    # TODO: replace with appropriate socket file path
    server_address = os.path.join(tempfile.gettempdir(), 'server_socket')

# Path to log file
# TODO: Change to appropriate path and name
server_log = os.path.join(tempfile.gettempdir(), 'server.log')

class RequestHandler(SocketServer.StreamRequestHandler):

    """Request handler

    An instance of this class is created for each connection made
    by a client.  The Server class invokes the instance's
    setup(), handle(), and finish() methods.

    The template implementation here simply reads a single line from
    the client, breaks that up into whitespace-delimited words, and
    then uses the first word as the name of a "command."  If there is
    a method called "do_COMMAND", where COMMAND matches the
    commmand name, then that method is invoked.  Otherwise, an error
    message is returned to the client.


    def handle(self):
        """Service a newly connected client.

        The socket can be accessed as 'self.connection'.  'self.rfile'
        can be used to read from the socket using a file interface,
        and 'self.wfile' can be used to write to the socket using a
        file interface.

        When this method returns, the connection will be closed.
        # Read a single request from the input stream and process it.
        # TODO: Change as needed for actual client-server protocol.
        request = self.rfile.readline()
        if request:
            self.server.log('request %s: %s',
                            self.connection.getpeername(), request.rstrip())
            except Exception, e:
                self.server.log('exception: %s' % str(e))
                self.wfile.write('Error: %s\n' % str(e))
            self.server.log('error: unable to read request')
            self.wfile.write('Error: unable to read request')

    def process_request(self, request):
        """Process a request.

        This method is called by self.handle() for each request it
        reads from the input stream.

        This implementation simply breaks the request string into
        words, and searches for a method named 'do_COMMAND',
        where COMMAND is the first word.  If found, that method is
        invoked and remaining words are passed as arguments.
        Otherwise, an error is returned to the client.

        words = request.split()
        if len(words) == 0:
            self.server.log('error: empty request')
            self.wfile.write('Error: empty request\n')

        command = words[0]
        args = words[1:]

        methodname = 'do_' + command
        if not hasattr(self, methodname):
            self.server.log('error: invalid command')
            self.wfile.write('Error: "%s" is not a valid command\n' % command)
        method = getattr(self, methodname)

    def do_stop(self, *args):
        """Process a 'stop' command"""
        self.wfile.write('Stopping server\n')

    def do_echo(self, *args):
        """Process an 'echo' command"""
        self.wfile.write(' '.join(args) + '\n')

    def do_status(self, *args):
        """Process a 'status' command"""
        self.wfile.write('Server Version:    %s\n' % version)
        self.wfile.write('Process ID:        %d\n' % os.getpid())
        self.wfile.write('Parent Process ID: %d\n' % os.getppid())
        self.wfile.write('Server Socket:     %s\n' % str(server_address))
        self.wfile.write('Server Log:        %s\n' % server_log)

    def do_add(self, a, b):
        """Process an 'add' command"""
        answer = int(a) + int(b)
        self.wfile.write('%s + %s = %s\n' % (a, b, answer))

class Server(ServerBase):

    """Server implementation


    def __init__(self, server_address):

        if ServerBase == SocketServer.UnixStreamServer:
            # Delete the socket file if it already exists
            if os.access(server_address, 0):

        ServerBase.__init__(self, server_address, RequestHandler)

    def log(self, format, *args):
        """Write a message to the server log file"""
            message = format % args
            timestamp = time.strftime('%Y-%m-%d %H:%M:%S')
            f = open(server_log, 'a+')
            f.write('%s %s\n' % (timestamp, message))
        except Exception, e:
            print str(e)

    def serve_until_stopped(self):
        """Serve requests until self.stop() is called.

        This is an alternative to BaseServer.serve_forever()

        self.__stopped = False
        while not self.__stopped:

    def stop(self):
        """Stop handling requests.

        Calling this causes the server to drop out of

        self.__stopped = True

    def __daemonize(self):
        """Create daemon process.

        Based upon recipe provided at

        UMASK = 0
        WORKDIR = '/'
        MAXFD = 1024
        if hasattr(os, 'devnull'):
            REDIRECT_TO = os.devnull
            REDIRECT_TO = '/dev/null'

        try :
            if os.fork() != 0:


            if os.fork() != 0:

        except OSError, e:
            self.log('exception: %s %s', e.strerror, e.errno)
            raise Exception, "%s [%d]" % (e.strerror, e.errno)
        except Exception, e:
            self.log('exception: %s', str(e))
        maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
        if maxfd == resource.RLIM_INFINITY:
            maxfd = MAXFD
        for fd in range(0, maxfd):
            except OSError:
                pass, os.O_RDWR)
        os.dup2(0, 1)
        os.dup2(0, 2)

def run_server(options, args):
    """Run a server daemon in the current process."""
    svr = Server(server_address)

def do_request(options, args):
    """Send request to the server and process response."""
    if ServerBase == SocketServer.UnixStreamServer:
        s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    elif ServerBase == SocketServer.TCPServer:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Send request
    # TODO: Change as needed for actual client-server protocol
    s.sendall(' '.join(args) + '\n')

    # Print response
    # TODO: Change as needed for actual client-server protocol
    sfile = s.makefile('rb')
    line = sfile.readline()
    while line:
        print line,
        line = sfile.readline()

if __name__ == '__main__':
    optparser = optparse.OptionParser(usage=usage,
    (options, args) = optparser.parse_args()

    if len(args) == 0:

    if args[0] == 'start':
        run_server(options, args[1:])
        do_request(options, args)

> The server starts a daemon process, disconnected from the user's terminal, like it should.

> The server writes to a log file

These days I write my server programs (Python, C, even Perl on the odd occasion) to run under djb's daemontools
( In particular, supervise, multilog and the aoftlimit tools which takes care of both of these issues, and restarting the program should it abort. With logging the policy (i.e. log file rotation, maximum size of log file) is separated so you don't have to worry about log file management. Solaris has a similar mechanism with 10, I believe.

I do have a Python wrapper to do the "fork/setsid/etc" dance where daemontools isn't installed, but thankfully I have more say in how systems are configured these days ...

Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?