Desktop Linux

Linux has been my primary OS for about eight years. I am very fond of it and have often made the analogy that sitting down to a Windows box feels as clumsy as having your thumbs removed.

I used Redhat, then Fedora, then Ubuntu. It really annoyed me when they hid the kernel boot messages to be more noob friendly or stylish or whatever. My Linux boxes are workstations damn it; show me the bogomips.

Then Ubuntu gets stupid. Shuttleworth decides we need windows controls on the wrong side to prepare us for ... something. So I spend a year feeling like an American driving in Britain when using Linux and a Brit driving in the US when using Windows.

And that something turns out to be Unity.

And Unity sucks.

And my escape route is gone because my beloved Gnome had become Gnome 3 aka Gnome Shell aka Hunt the Wumpus but I give it a shot for a few weeks because it has the wonderful quality of not being Unity until it strikes me, this sucks too. So I spend an entire weekend trying different distros -- then going back to Ubuntu. Then spending the next weekend forcing XFCE into dressing up as Gnome 2. Which was close but what a weird the-opposite-of-forward step to make?

Pretty-please someone fork Gnome 2 and save the Linux desktop.

Remove the "close window?" prompt from Gnome-Terminal

$ gconftool-2 --type bool --set /apps/gnome-terminal/global/confirm_window_close "false"

#get_off_my_yard

The Web Client

Last weekend I got my WebSocket server for NetBoa working again (yay).

This weekend I've been laying out the HTML5 and CSS for the browser side. At the moment it looks like this;

It seems obvious now, but it felt like an epiphany when I realized that the project only required a single HTML page. Everything from the login screen forward will be elements dynamically create via Javascript (using the excellent JQuery lib) talking the server over a WebSocket. Chrome will happily maintain an open WebSocket for days with no activity (I tested it).

In John Gardner's book, Grendel, a dragon offers up the advice 'alternatives exclude'. I'm wrestling with some UI decisions -- screen dimensions vs. info glut. In the shot above, the left panel is somewhat like the main viewer from Star Trek;

When the right is like a console;

Originally, I had three columns where the far left one containing a kind of galactic IRC. But that was just too much.

I'm still trying to decide on a messaging protocol. JSON is great for interfacing with Javascript in the browser but that's precisely where I don't need speed. Given the short life of these message packets it seems rather wasteful to include an extra library just to generate Python objects to be cast into JSON strings a picosecond later. I'm 90% sure that I'll go with text commands and an option to send snippets of HTML.

Unpacking WebSocket Frames Cont.

Using WireShark, I captured some frames between my browser and WebSocket.org's Echo Server. This gave me some data to experiment with without needing a living server.

I mentioned that the protocol in version 13 was very different. Previously, you bookended your packets with \x00 and \xFF and shoved them down the wire. RFC 6455 adopted a more formal framing that includes op codes for the frame type (string vs binary && ping vs pong), a continuation flag, and three different data structures for describing payload size. Additionally, payloads from the Client to the Server are XOR'ed with a random 32 bit mask. This mask is included in the client frame. Frames from the Server to Client are sent in the clear. Plus some reserved fields.

Yes, it does seem a touch fiddly.

Since I'll own both ends of the wire, I should be fine with a server that only uses 'final fragment' frames (no need for messages to span data packets) using text frames. Most likely JSON. I don't plan to support the previous WS drafts since browser updates are darn near seamless today.

The output of the previous code is:

Final Frame: True
Masked: True
Opcode: 1
Payload Length: 28
Payload: Rock it with HTML5 WebSocket

Final Frame: True
Masked: False
Opcode: 1
Payload Length: 28
Payload: Rock it with HTML5 WebSocket

Final Frame: True
Masked: True
Opcode: 1
Payload Length: 165
Payload: This is a very long message that is longer than 125 bytes so it will be offet with a 16 bit extended payload length and therefore payload len should be equal to 126.

Final Frame: True
Masked: False
Opcode: 1
Payload Length: 165
Payload: This is a very long message that is longer than 125 bytes so it will be offet with a 16 bit extended payload length and therefore payload len should be equal to 126.

Unpacking a WebSocket Frame

I'm going to put my comments in the next post because Blogger likes to mangle my code snippets if I try to edit something...

#!/usr/bin/env python

import struct
import array 


##      Frames with a 7 bit payload length

CLIENT7 = (
    b'\x81\x9c\xb3\x8f\x67\x54\xe1\xe0\x04\x3f\x93\xe6\x13\x74\xc4\xe6'
    b'\x13\x3c\x93\xc7\x33\x19\xff\xba\x47\x03\xd6\xed\x34\x3b\xd0\xe4'
    b'\x02\x20'
    )

SERVER7 = (
    b'\x81\x1c\x52\x6f\x63\x6b\x20\x69\x74\x20\x77\x69\x74\x68\x20\x48'
    b'\x54\x4d\x4c\x35\x20\x57\x65\x62\x53\x6f\x63\x6b\x65\x74'
    )


##      Frames with a 16 bit payload length

CLIENT16 = (
    b'\x81\xfe\x00\xa5\x33\x92\xab\x19\x67\xfa\xc2\x6a\x13\xfb\xd8\x39'
    b'\x52\xb2\xdd\x7c\x41\xeb\x8b\x75\x5c\xfc\xcc\x39\x5e\xf7\xd8\x6a'
    b'\x52\xf5\xce\x39\x47\xfa\xca\x6d\x13\xfb\xd8\x39\x5f\xfd\xc5\x7e'
    b'\x56\xe0\x8b\x6d\x5b\xf3\xc5\x39\x02\xa0\x9e\x39\x51\xeb\xdf\x7c'
    b'\x40\xb2\xd8\x76\x13\xfb\xdf\x39\x44\xfb\xc7\x75\x13\xf0\xce\x39'
    b'\x5c\xf4\xcd\x7c\x47\xb2\xdc\x70\x47\xfa\x8b\x78\x13\xa3\x9d\x39'
    b'\x51\xfb\xdf\x39\x56\xea\xdf\x7c\x5d\xf6\xce\x7d\x13\xe2\xca\x60'
    b'\x5f\xfd\xca\x7d\x13\xfe\xce\x77\x54\xe6\xc3\x39\x52\xfc\xcf\x39'
    b'\x47\xfa\xce\x6b\x56\xf4\xc4\x6b\x56\xb2\xdb\x78\x4a\xfe\xc4\x78'
    b'\x57\xb2\xc7\x7c\x5d\xb2\xd8\x71\x5c\xe7\xc7\x7d\x13\xf0\xce\x39'
    b'\x56\xe3\xde\x78\x5f\xb2\xdf\x76\x13\xa3\x99\x2f\x1d'
    )

SERVER16 = (
    b'\x81\x7e\x00\xa5\x54\x68\x69\x73\x20\x69\x73\x20\x61\x20\x76\x65'
    b'\x72\x79\x20\x6c\x6f\x6e\x67\x20\x6d\x65\x73\x73\x61\x67\x65\x20'
    b'\x74\x68\x61\x74\x20\x69\x73\x20\x6c\x6f\x6e\x67\x65\x72\x20\x74'
    b'\x68\x61\x6e\x20\x31\x32\x35\x20\x62\x79\x74\x65\x73\x20\x73\x6f'
    b'\x20\x69\x74\x20\x77\x69\x6c\x6c\x20\x62\x65\x20\x6f\x66\x66\x65'
    b'\x74\x20\x77\x69\x74\x68\x20\x61\x20\x31\x36\x20\x62\x69\x74\x20'
    b'\x65\x78\x74\x65\x6e\x64\x65\x64\x20\x70\x61\x79\x6c\x6f\x61\x64'
    b'\x20\x6c\x65\x6e\x67\x74\x68\x20\x61\x6e\x64\x20\x74\x68\x65\x72'
    b'\x65\x66\x6f\x72\x65\x20\x70\x61\x79\x6c\x6f\x61\x64\x20\x6c\x65'
    b'\x6e\x20\x73\x68\x6f\x75\x6c\x64\x20\x62\x65\x20\x65\x71\x75\x61'
    b'\x6c\x20\x74\x6f\x20\x31\x32\x36\x2e'
    )

def unpack_frame(data):
    frame = {}
    byte1, byte2 = struct.unpack_from('!BB', data)
    frame['fin'] = (byte1 >> 7) & 1
    frame['opcode'] = byte1 & 0xf
    masked = (byte2 >> 7) & 1
    frame['masked'] = masked
    mask_offset = 4 if masked else 0
    payload_hint = byte2 & 0x7f
    if payload_hint < 126:
        payload_offset = 2
        payload_length = payload_hint               
    elif payload_hint == 126:
        payload_offset = 4
        payload_length = struct.unpack_from('!H',data,2)[0]
    elif payload_hint == 127:
        payload_offset = 8
        payload_length = struct.unpack_from('!Q',data,2)[0]
    frame['length'] = payload_length
    payload = array.array('B')
    payload.fromstring(data[payload_offset + mask_offset:])
    if masked:
        mask_bytes = struct.unpack_from('!BBBB',data,payload_offset)
        for i in range(len(payload)):
            payload[i] ^= mask_bytes[i % 4]
    frame['payload'] = payload.tostring()
    return frame

    
if __name__ == '__main__':

    frame = unpack_frame(CLIENT7)
    print '\nFinal Frame:', bool(frame['fin'])
    print 'Masked:', bool(frame['masked'])
    print 'Opcode:', frame['opcode']
    print 'Payload Length:', frame['length']
    print 'Payload:', frame['payload']

    frame = unpack_frame(SERVER7)
    print '\nFinal Frame:', bool(frame['fin'])
    print 'Masked:', bool(frame['masked'])
    print 'Opcode:', frame['opcode']
    print 'Payload Length:', frame['length']
    print 'Payload:', frame['payload']

    frame = unpack_frame(CLIENT16)
    print '\nFinal Frame:', bool(frame['fin'])
    print 'Masked:', bool(frame['masked'])
    print 'Opcode:', frame['opcode']
    print 'Payload Length:', frame['length']
    print 'Payload:', frame['payload']

    frame = unpack_frame(SERVER16)
    print '\nFinal Frame:', bool(frame['fin'])
    print 'Masked:', bool(frame['masked'])
    print 'Opcode:', frame['opcode']
    print 'Payload Length:', frame['length']
    print 'Payload:', frame['payload']

WebSocket RFC 6455 Handshake

I think I got the request/response bit figured out (it was actually simpler than the draft 76 method). This is my test using headers from the RFC document:

from hashlib import sha1
from base64 import b64encode

WS13_GUID = b'258EAFA5-E914-47DA-95CA-C5AB0DC85B11'

REQUEST = (
    b'GET /chat HTTP/1.1\r\n'
    b'Host: server.example.com\r\n'
    b'Upgrade: websocket\r\n'
    b'Connection: Upgrade\r\n'
    b'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n'
    b'Origin: http://example.com\r\n'
    b'Sec-WebSocket-Protocol: chat, superchat\r\n'
    b'Sec-WebSocket-Version: 13\r\n'
    b'\r\n'
    )

RESPONSE = (
    b'HTTP/1.1 101 Switching Protocols\r\n'
    b'Upgrade: websocket\r\n'
    b'Connection: Upgrade\r\n'
    b'Sec-WebSocket-Accept: {}\r\n'
    b'Sec-WebSocket-Protocol: chat\r\n'
    b'\r\n'
    )

CORRECT_ACCEPT = b's3pPLMBiTxaQ9kYGzzhZRbK+xOo='

def parse_request(request):
    req = {}
    lines = request.split(b'\r\n')
    line = lines.pop(0)
    items = line.split(b'\x20',2)
    if len(items) != 3:
        raise BaseException('Malformed WebSocket Request Method.')
    req[b'method'] = items[0]
    req[b'request_uri'] = items[1]
    req[b'http_version'] = items[2]
    for line in lines:
        if line:
            parts = line.split(b':\x20', 1)
            req[parts[0].lower()] = parts[1]
    return req

if __name__ == '__main__':
    req = parse_request(REQUEST)
    salted_key = req[b'sec-websocket-key'] + WS13_GUID
    sec_ws_accept = b64encode(sha1(salted_key).digest())
    assert(sec_ws_accept == CORRECT_ACCEPT)
    print RESPONSE.format(sec_ws_accept)

Step two is to get the VERY DIFFERENT message protocol working.

WebSockets

Naturally, the Overlords of the Internet have obsoleted my WebSocket code for the third time. Hopefully things will settle down now the current draft has been submitted as a proposed standard (one step away from an adopted standard).

Pointing my browser (Chromium) at http://websocketstest.com tells me that it's ready for RFC 6455, so I guess I need to get coding.