Persistence

I've had a few questions about how I plan to handle persistence, as in saving the states of game objects in case the server is restarted. The quick answer is I'm not -- with the exception of player character data. Bogboa is using two forms of data storage; YAML files and an SQLite database. I'll discuss each.


YAML Files

Currently, YAML is the only third-party library used in Bogboa. I didn't want to use any really, but YAML is such a human friendly format that I went with it.

On startup, the world is created from one or more directory trees filled with YAML files that define individual rooms, items, races, etc. This is a one-way conversion where each YAML file is parsed into a Python dictionary that is used to set attributes of game classes. One of my long-term goals is to support reading these YAML trees from zip files so that designers could release adventure modules that contain complete dungeons with new monsters and items that load seamlessly into existing game worlds. This is one of the reasons I'm using UUID's for keys, since they don't require multiple developers to coordinate or negotiate around potential collisions like vnums do.

So restarting the server cleans every floor of dropped loot and every dead NPC is alive again. This appeals to me because it's simple and troubleshooting scripts is easier from known states.


SQLite Database

Starting in Python 2.5, SQLite3 is part of the standard library (and a kick-ass little SQL database) which makes using it a non-brainer.

My early tables were pretty vanilla with columns; (name, race, level, guild, stat1, stat2, etc) and I was always changing them. Also, I was using a lot of sub-tables to hold information like skills, inventory, and character flags. These sub-tables all had nearly identical structure; (character_uuid, skill_name) and (character_uuid, inventory_slot).

At the same time, I was making structural changes to how I split player from their avatars (which is why the codebase is currently b0rked). I knew that the closer I handled avatars like NPCs the easier things would be down the road. Moving all the various traits into dictionaries was helpful and this transition to key-value thinking migrated to how I stored character data as well.

What I wanted was a way to load whole dictionaries at a time but update individual key-value pairs on the fly. When you log in, your character's profile dictionary is retrieved containing (name, race, guild, level, ...) and when you gain a level, your characters (profile + level) record is incremented by one and if the server crashes a moment later you, hopefully, didn't lose it.

One of the nuisances of SQL is you often have to do a SELECT before you can save data because sometimes you cannot be sure if you need to use the INSERT or the UPDATE command. SQLite supports a non-standard but convenient variation of this called INSERT OR UPDATE. The catch is the key has to be unique.

What I did was make the key = (Character UUID + Some Catergory Name + Key Name);

CREATE TABLE IF NOT EXISTS key_value
    (
    uuid TEXT,
    category TEXT,
    key TEXT,
    value TEXT,
    PRIMARY KEY(uuid, category, key)
    );

In the above example, the category name would be 'profile'.

Reading all the rows of a category would look like;

SELECT key, value
FROM key_value
WHERE uuid = ? AND category = ?;

And saving an individual key-value would look like;

INSERT OR REPLACE INTO key_value (uuid, category, key, value)
VALUES (?, ?, ?, ?);

Everything I need to know I learned from D&D

Clever presentation by a guy who compares software development to playing Dungeons and Dragons:

All I Need To Know About Life I learned From Dungeons and Dragons. An IgniteOKC Talk. from Chad Henderson on Vimeo.

MUD Game Programming

I believe the only book ever written as howto for creating a MUD was MUD Game Programming by Ron Pelton. Originally published in late 2003, it's been out of print for a while and I finally managed to snag a used copy for a reasonable price. It even came with an unopened CD.

I don't expect to glean a great deal from it having already written an asynchronous server, plus Pelton is using C++ where I'm doing everything in Python. Still, it seems like a quality book with a lot of example code. Books on game programming tend to suffer a poor signal to noise ratio. There's a ton of dreck out there, especially garbage written by industry insiders who have no clue about programming but feel like they can help you with their experience interviewing other non-programmers. Ugh. Even worse are the ones aimed at teens -- clearly targeting the well-intentioned but useless gift from Mom category.

Pardon the rant.

Anyways, I've read the first couple chapters on networking and socket programming and Pelton is delivering the goods. He is specific and writes in an friendly, engaging manner. He gets extra points for covering Linux in addition to Windows. Recommended (if you can find it).

Work Around

--start of edit

Well, it turned out that the approach listed below does not find methods from base classes. I revised it to use __getattribute__() instead which returns a bound instance of the class but that only lives for the scope of the cmd_driver() method:

def cmd_driver(self):
    ## call the driver method for current state
    self.__getattribute__(self.state)()

---end of edit


My last post detailed a garbage collection issue I was having when an instance of the User class referenced itself using a bound variable. I got around it changing my state property to a string variable holding the name of the method I wanted to call and then calling said method via some class introspection;

def cmd_driver(self):
    ## call the driver method for current state via class introspection
    self.__class__.__dict__[self.state](self)

That kind of introspection always feels like duct tape.

I should point out that Python's garbage collector probably would have deleted those User instances eventually and that I could have manually ended the connection by calling socket.close(). I prefer the immediate garbage collection closure because it's an indicator that everything is running smoothly.

Dynamic Methods and Garbage Collecting

I really enjoy the Python programming language. Even when it bites you, it takes you on an interesting journey.

In the MUD, I use the term Client to refer to the session with the user at the other end of the wire. The Client is passed to a User object that represents the player in some state or another. When a User gets deleted it gets chomped up by Python's garbage collector which should delete the client which in turn should delete the socket which fires the socket's close() method on the way out and gives the cat a fish, err, drops the user resulting in the famous 'Connection closed by foreign host..'

Only it wasn't.

Deleted clients and "disconnected" users were hanging around typing to unmonitored, uncaring sockets. After a bit of poking I found the problem was some cheap hackery I was using for the login process. I was using a property called "cmd_driver" like a state machine, changing it to point to the next method I wanted input to go;

self.cmd_driver = self.get_password

Turns out, this was setting cmd_driver to a bound method; one that is wrapped inside a reference to the an instance of the class, which in this case was itself. This caused the reference count to increase by one and avoid the garbage collector.Here's a demonstration snippet;

>>> class A(object):
...     def __del__(self):
...             print "__del__ called"
...     def foo(self):
...             pass
...
>>> a = A()
>>> del a
__del__ called
>>> a2 = A()
>>> a2.bar = a2.foo
>>> del a2
>>> 

So I need to work out a healthier version of the state machine.
Maybe one that calls unbound methods of the class or instance methods
via some introspection.

Random Name Generation

A few years ago I was playing around with a random name generator. My approach was to cobble together random letter combination like;

leading consonants + vowels + inner consonants + vowels + closing consonants

Basically, I was aiming for something pronounceable with commons letters weighted to appear more often. It produced output such as;

Votharn Eristacark Iplortidot Birtoil Udaeteahieb Aceastoherk Reloist Tharnog Wasterk Femewelav Ublyrrielic Cekird Owritothol Hoogoh Obloukajarriem Sleebont Niestart Pekev Lirtooth Efentoidagix Klyckas Yryfesat Klooton

Yeah ... that's really, really awful.

So I decided to give it another whack. This time I started with the premise, 'what sounds most like a name?'

Names do!

I found a couple files with over a thousand of the most common male and female first names on the US Census Bureau's web page and started playing. I wrote a Python script that used regular expressions to slice a batch of words into three lists;

List 1 = Zero or more vowels + One or more consonants at the start of the word
List 2 = One or more vowels + One or more consonants inside the word (not at the start or the end). We can get 0 or more of these patterns depending on the word.
List 3 = One or more vowels + Zero or more consonants at the end of the word.

Side note: If you haven't dug into regular expressions yet I highly recommend you check them out. I avoided them for years and now they're an essential part of my programmer tool box. Another big plus is their utility spans multiple languages.

I also tracked the frequency of each pattern, sorting by most common first and discarding the rares. Finally, I dumped the output formatted as Python lists that I could paste right into the source of the next script.

import re
import operator


_FILENAME = 'data/elves2.txt'
_CULL = 1

## Match 0 or more vowels + 1 or more consonants at the start of the word
_LEAD = re.compile(r'^[aeiouy]*(?:qu|[bcdfghjklmnpqrstvwxz])+')
## Match 1 or more vowels + 1 or more consonants inside a word (not start/end)
_INNER = re.compile(r'\B[aeiouy]+(?:qu|[bcdfghjklmnpqrstvwxz])+\B')
# Match 1 or more vowels + 0 or more consonats at the end of a word
_TRAIL = re.compile(r'[aeiouy]+(?:qu|[bcdfghjklmnpqrstvwxz])+$')


def token_lists(names):

   lead, inner, tail = {}, {}, {}

   ## Populate dictionaries; key=pattern, value=frequency
   for name in names:

       match = re.match(_LEAD, name)
       if match:
           pat = match.group(0)
           count = lead.get(pat,0)
           lead[pat] = count +1

       matches = re.findall(_INNER, name)
       for pat in matches:
           print pat,
           count = inner.get(pat,0)
           inner[pat] = count +1

       match = re.search(_TRAIL, name)
       if match:
           pat = match.group(0)
           count = tail.get(pat,0)
           tail[pat] = count +1


   ## Convert dicts to a list of tuples in the format (pattern, frequency)
   lead_srt  = sorted(lead.items(),key=operator.itemgetter(1),reverse=True)
   inner_srt  = sorted(inner.items(),key=operator.itemgetter(1),reverse=True)
   tail_srt  = sorted(tail.items(),key=operator.itemgetter(1),reverse=True)

   ## Build lists of patterns ordered most to least frequent and cull rares
   lead_list = [ x[0] for x in lead_srt if x[1] > _CULL ]
   inner_list = [ x[0] for x in inner_srt if x[1] > _CULL ]
   tail_list = [ x[0] for x in tail_srt if x[1] > _CULL ]

   return lead_list, inner_list, tail_list


if __name__ == '__main__':

   names = open(_FILENAME, 'rt').readlines()
   lead_list, inner_list, tail_list = token_lists(names)

   print '#', len(lead_list), len(inner_list), len(tail_list)
   print '_LEADS = ', lead_list
   print '_INNERS = ', inner_list
   print '_TAILS = ', tail_list

Next I used a script to assemble random names from these lists. Here's the one for male names:

import random


_LEADS =  ['d', 'j', 'm', 'r', 'l', 'w', 'c', 'h', 'g', 'b', 'br', 't', 'k',
   'n', 's', 'cl', 'fr', 'f', 'p', 'st', 'v', 'ch', 'sh', 'gr', 'tr']
_INNERS =  ['er', 'ar', 'el', 'or', 'an', 'ic', 'arr', 'am', 'ol', 'on',
   'al', 'en', 'ill', 'in', 'err', 'and', 'il', 'om', 'et', 'arl', 'ev',
   'ac', 'ust', 'av', 'ert', 'enn', 'ent', 'ath', 'onn', 'it', 'os', 'enc',
   'yr', 'em', 'ist', 'anc', 'arc', 'ich', 'as', 'est']
_TAILS =  ['o', 'on', 'e', 'y', 'ey', 'er', 'in', 'an', 'ie', 'io', 'en',
   'is', 'el', 'us', 'es', 'as', 'ian', 'ed']


def namegen():
   syllables = random.randint(0,1)
   if random.random() > .85:
       syllables += 1
   name = random.choice(_LEADS)
   for x in range(syllables):
       name += random.choice(_INNERS)
   name += random.choice(_TAILS)
   return name.title()


if __name__ == '__main__':

   for x in range(100):
       print namegen(),

Which gives output like;

Tyron Frian Grathan Charcan Tren Garrian Stasas Cholian Dites Ris Tie Non Lin Lon Wian Wio Ramas Lo Larus Pan Cio Wasie Mely Chie Levey Bustel Lillen Ben Ponny Panian Nyrus Bre Raso Stel Fasis Konandas Starrin Hian Starlio Rey Jathen Frander Ven Talamio Samin Ches Handey Sterian Nencin Sathon Ponel Citel Momen Wus Rencey Dence Rencan Bevon Jo Bre Tis Non Heved Ne Shestas Lales Ferrus Werrus Foso Standie Ver Gis Giner Ver Nin Res Veled Clon Geler Mustin Gio Bry Juste Shanan Cled Trely Chon Trin Binis Jinen Jacy Ralio Bo Metel De Sian Brie Fanan Donen Jed

Seeding with female names, I get;

Alianne Cler Cishel Kroller Namie Dah Kyn Niton Hie Trosancel Cher Na Angy Sestin Jah Dathe Pin Traton Teris Dishon Goren Sheny Frindis Chalin Frie Lolleen Ware Chrena Hy Veannon Dyn Padica Sher Chie Kon Brollis Cisi Gy Chey Kreni Dulani Gancis Shancey Satheen Chreanny Shy Clin Cessia Kriner Shestie Hancah Ner Trorer Ger Tren Ramerrer Freanne Kaurah Frolah Kritah Harley Kisten Eleen Trellyn Chiannel Trosi Alisin Genery Cla Mie Jate Shosacey De Angoron Kis Gianon Mandite Chryn Wer Shadalie Polon Gita Ner Fron Va Chrabica Vatia Clia Tacel Rareen Treen Angia Henen Angin Poreen Eleannel Ken Brer Branca Hesta

MiniBoa

Over on Mudbytes, Idealiad (aka Kooneiform) started a thread about creating a Python Socket MUD -- a very tiny socket library that Python coders could use as the basis for their MUD projects and experiments. There already were a few of these for other languages such as TeensyMud for Ruby but nothing for Python. I really liked this idea and, as I had a working Telnet server already, I decided to repackage my network modules as MiniBoa. Instead of the BogBoa's GPL licensing, I switched to the more permissive Apache 2.0 (one of the recommended licenses for maximum compatibility with the Python's own).

To be honest, the code was a lot messier to convert into a stand-alone library than I expected -- because this was the first code I wrote for BogBoa and I was hooking into it less than gracefully. Since I didn't want to maintain two copies of (mostly) the same modules, I decided to revamp BogBoa to use MiniBoa for networking.

You can find code and documentation at the project page and some discussion on this Mudbytes thread.