Oolite debug console TCP protocol

The Oolite debug console TCP protocol is used by testing and developer releases of Oolite to communicate with debugging consoles running as separate applications on the same computer, or on other computers over a network. It is supported in test releases since 1.70, and will be supported in the OXP developer version of the upcoming stable release.

Using the protocol, Oolite presents an interactive command prompt. Interaction is handled by a user-replaceable JavaScript script running in Oolite. The default script acts as an interactive JavaScript interpreter with a macro system for quick commands. Console applications are not required to know anything about JavaScript; they merely provide input and output, although additional features are possible.

This page is intended primarily for programmers implementing or maintaning console servers, that is, applications that provide the console user interface. The Oolite source repository contains implementations in C at trunk/tools/simpleDebugConsole and in Python at trunk/tools/pythonDebugConsole. The C implementation relies on CoreFoundation, which is difficult to use on non-Mac platforms, but the Python version is portable and separates the protocol handling from the interface.

Protocol overview

When a special enabling OXP is installed, Oolite acts as a client for the debug console protocol, and tries to connect to a server (console) over TCP on startup. By default, it will attempt to connect to 127.0.0.1:8563, but this is configurable.

The basic unit in the protocol is the XML property list.

Packet framing

Each property list is referred to as a packet, but this should not be confused with a TCP packet. At the TCP level, the protocol is strictly a stream. Dividing it into property list packets must be handled at a higher level. Each property list packet is preceded by a 32-bit size, in network-endian format, specifying the size of the packet data (the payload), excluding the size header, in eight-bit bytes. For instance, this property list:

{ message = "Hello, world!" }

would be expressed in XML as:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>message</key>
	<string>Hello, world!</string>
</dict>
</plist>

or the octet sequence:

3C3F786D 6C207665 7273696F 6E3D2231 2E302220 656E636F 64696E67 3D225554
462D3822 3F3E0A3C 21444F43 54595045 20706C69 73742050 55424C49 4320222D
2F2F4170 706C6520 436F6D70 75746572 2F2F4454 4420504C 49535420 312E302F
2F454E22 20226874 74703A2F 2F777777 2E617070 6C652E63 6F6D2F44 5444732F
50726F70 65727479 4C697374 2D312E30 2E647464 223E0A3C 706C6973 74207665
7273696F 6E3D2231 2E30223E 0A3C6469 63743E0A 093C6B65 793E6D65 73736167
653C2F6B 65793E0A 093C7374 72696E67 3E48656C 6C6F2C20 776F726C 64213C2F
73747269 6E673E0A 3C2F6469 63743E0A 3C2F706C 6973743E 0A

The length of this sequence is 249 octets, or 0xF9 in hexadecimal. The complete, framed packed data is thus:

000000F9 3C3F786D 6C207665 7273696F 6E3D2231 2E302220 656E636F 64696E67
3D225554 462D3822 3F3E0A3C 21444F43 54595045 20706C69 73742050 55424C49
4320222D 2F2F4170 706C6520 436F6D70 75746572 2F2F4454 4420504C 49535420
312E302F 2F454E22 20226874 74703A2F 2F777777 2E617070 6C652E63 6F6D2F44
5444732F 50726F70 65727479 4C697374 2D312E30 2E647464 223E0A3C 706C6973
74207665 7273696F 6E3D2231 2E30223E 0A3C6469 63743E0A 093C6B65 793E6D65
73736167 653C2F6B 65793E0A 093C7374 72696E67 3E48656C 6C6F2C20 776F726C
64213C2F 73747269 6E673E0A 3C2F6469 63743E0A 3C2F706C 6973743E 0A

This is obviously a rather verbose protocol; the sample packet could easily be expressed in fifteen bytes. However, since it is intended to be used on a single machine or machines on the same network, and the traffic level will be quite low, this is a secondary concern. Flexibility is a greater concern.

Why property lists?

It may seem like a somewhat perverse choice to base the communication protocol on property lists. Originally, I intended to use a binary protocol encoding. However, I wanted to ensure that the TCP protocol would be sufficient to replicate the functionality of the Mac-only integrated debug console which was already implemented. Part of this functionality is using configuration data set in property lists and modifiable via JavaScript for things like the font and colours of the console window. Since this configuration data is structured using property lists (and the similar concept of JavaScript property lists), I ended up with three options:

1. Design a new way of encoding the structured data.
2. Use some existing structured-data format, such as YAML or JSON, introducing a new dependency.
3. Use the format already being used throughout Oolite.

The latter was clearly the least complicated option. Having decided to use property lists in the protocol, it seemed simplest to make them ubiquitous, and use them for such things as specifying packet type. I initially intended to use binary plist format, but switched to XML because it’s potentially easier, or at least less scary, to implement in languages for which plist libraries do not exist.

Although the property list originates with NextStep and is therefore associated with Objective-C, various other implementations exist. For instance, the reference implementation for console servers is implemented in pure C, using the CoreFoundation libraries. There are also XML property list parsers for various scripting languages, such as Python, and of course it is not especially difficult to implement one on top of an XML parser. Oolite will always send XML format property lists, but will accept binary and OpenStep format as well.

Packet structure

Each property list packet is a dictionary. Each packet must contain the key “packet type” (kOOTCPPacketType), whose value is a string specifying the interpretation of the packet. Additional keys depend on the packet type.

Constants for packet keys and packet type values can be found in the Oolite source tree at trunk/src/Core/Debug/OODebugTCPConsoleProtocol.h.

Handshake

To establish a connection, Oolite connects to a console and sends a “Request Connection” packet (kOOTCPPacket_RequestConnection) packet. The console must reply with an “Approve Connection” (kOOTCPPacket_ApproveConnection) packet for the connection to be considered open. The console should send a “Reject Connection” (kOOTCPPacket_RejectConnection) packet if it cannot support a connection.

Pings

Both parties in an open connection may send a “Ping” (kOOTCPPacket_Ping) packet at any time. The other party must then return a “Pong” (kOOTCPPacket_Pong) packet within reasonable time for the connection to be considered open. Oolite does not currently send pings (although it does respond to them), but may start doing so in future.

Closing connections

Either party of an open connection may close it gracefully by sending a “Close Connection” (kOOTCPPacket_CloseConnection) packet with an optional explanatory message. Graceless connections are achieved by dropping the connection.

The configuration dictionary

A property list called the configuration dictionary is used to store key/value pairs of potential interest to the three parties in the connection – Oolite, the console script, and the console server. The contents of the configuration dictionary come from two sources: debugConfig.plist, which is merged from OXPs in the usual fashion, and an override dictionary which can be modified from JavaScript scripts and is saved in Oolite’s preferences.

When a connection is established, Oolite sends a “Note Configuration” (kOOTCPPacket_NoteConfiguration) packet containing the full contents of the configuration dictionary. Whenever the configuration dictionary is modified, Oolite sends one or more “Note Configuration Change” (kOOTCPPacket_NoteConfigurationChange) packets. The console server may also send “Note Configuration Change” packets to Oolite, in which case Oolite will respond with one or mor “Note Configuration Change” packets containing the same information. Additionally, the console server may send a “Request Configuration Value” (kOOTCPPacket_RequestConfigurationValue) packet to request “Note Configuration Change” packets for a certain key.

Console I/O

The central point of the protocol is to send and receive messages to and from the console script, which (by default) implements an interactive JavaScript console. A console server issues JavaScript commands, generally in response to user input, by sending “Perform Command” (kOOTCPPacket_PerformCommand) packets to Oolite. Oolite sends the console “Console Output” (kOOTCPPacket_ConsoleOutput) packets to write to the console. It can also send “Clear Console” (kOOTCPPacket_ClearConsole) packets to clear the console window (typically in response to the :clear macro), and “Show Console” (kOOTCPPacket_ShowConsole) packets to request that the console make itself visible and come to the front. (This behaviour can be configured in the configuration dictionary.)

Packet type reference

Approve Connection

Oolite ← Console
Symbolic constant: kOOTCPPacket_ApproveConnection

Sent by console server in response to “Request Connection” in order to open a connection.

Parameters:

• “console identity” (kOOTCPConsoleIdentity): string identifying console server. Optional.

Clear Console

Oolite → Console
Symbolic constant: kOOTCPPacket_ClearConsole

Indicates that the console should clear its output.

Parameters: none.

Close Connection

Oolite ↔ Console
Symbolic constant: kOOTCPPacket_CloseConnection

Sent by either party to end the session and close the connection. After a “Close Connection” packet is sent, no further communication should occur.

Parameters:

• “message” (kOOTCPMessage): short string describing the reason the connection is closing. Optional.

Console Output

Oolite → Console
Symbolic constant: kOOTCPPacket_ConsoleOutput

Sent by Oolite to print text to the console.

Parameters:

• “message” (kOOTCPMessage): the output text. Required.
• “color key” (kOOTCPPacket_ConsoleOutput): string identifying text colour/style to use for output. See OODebugTCPConsoleProtocol.h and debugConfig.plist for descriptions of how this is used. Required.
• “emphasis ranges” (kOOTCPEmphasisRanges): an array of numbers indicating parts of the output text that should be emphasized (bold in the screen shot below). Each range is specified as two integers, the starting offset (base 0) and the length of the range. Optional.

Note Configuration

Oolite → Console
Symbolic constant: kOOTCPPacket_NoteConfiguration

Sent by Oolite immediately after a connection is opened to inform the console server of the contents of the shared configuration dictionary.

Parameters:

• “configuration” (kOOTCPConfiguration): a dictionary. Required.

Note Configuration Change

Oolite ↔ Console
Symbolic constant: kOOTCPPacket_NoteConfigurationChange

Sent by Oolite to inform the console server of changes to the configuration dictionary, and by the console server to request changes to the configuration dictionary. Note: if the console server sends a “Note Configuration Change” packet, Oolite will typically respond with one or more “Note Configuration Change” packets. The console can only remove configuration overrides, not settings specified in “debugConfig.plist” files. Therefore, sending a packet to remove a configuration key may result in the key being set to a default value instead.

A “Note Configuration Change” packet must contain at least one of “configuration” or “removed configuration keys”.

Parameters:

• “configuration” (kOOTCPConfiguration): a dictionary of key/value pairs to change. Required if “removed configuration keys” is not included.
• “removed configuration keys” (kOOTCPRemovedConfigurationKeys): a dictionary of keys to remove. Required if “configuration” is not included.

Perform Command

Oolite ← Console
Symbolic constant: kOOTCPPacket_PerformCommand

Sent by the console server to execute a command, typically in response to user input. The command is handed off to the console script, which generally handles it by passing it to the JavaScript interpreter (unless it starts with a colon, in which case it’s treated as a macro).

Parameters:

• “message” (kOOTCPMessage): a string to send to the console script. Required.

Ping

Oolite ↔ Console
Symbolic constant: kOOTCPPacket_Ping

May be sent by either party to ensure that the other is still responding. The recipient must respond with a “Pong” packet. If the “message” parameter is specified, the same “message” must be sent with the “Pong” packet.

Parameters:

• “message” (kOOTCPMessage): a string which must be returned. Optional.

Pong

Oolite ↔ Console
Symbolic constant: kOOTCPPacket_Pong

Required response to “Ping packets.

Parameters:

• “message” (kOOTCPMessage): the same as the “message” parameter of the corresponding “Ping” message. Required if sent with the ping.

Reject Connection

Oolite ← Console
Symbolic constant: kOOTCPPacket_RejectConnection

Sent by console server in response to “Request Connection” in order to refuse a connection.

Parameters:

• “console identity” (kOOTCPConsoleIdentity): string specifying the reason the connection is being rejected. Optional.

Request Configuration Value

Oolite ← Console
Symbolic constant: kOOTCPPacket_RequestConfigurationValue

Sent by console server to request a “Note Configuration Change” packet for a specified key.

Parameters:

• “configuration key” (kOOTCPConfigurationKey): the key being queried. Required.

Request Connection

Oolite → Console
Symbolic constant: kOOTCPPacket_RequestConnection

Sent by Oolite to establish a connection. In order for a connection to be considered open, the console server must respond with a “Accept Connection” packet. The Request Connection packet contains a “protocol version” parameter, which the console server should check for compatibility. The version number is an integer with 24 relevant bits, encoding three eight-bit values. The highest-order of these is the “version format”. This specifies the basic framing format of the protocol; the XML property list format documented here is version format 1. The middle octet is the “major version”. This will change to indicate incompatible format changes. The lowest octet is the “minor version”, which will be incremented if new packet types or parameters have been added in a backwards-compatible way. The protocol version documented on this page is version format 1, major version 1, minor version 0, denoted 1:1.0.

Parameters:

• “Oolite version” (kOOTCPOoliteVersion): A string specifying the version of Oolite, such as “1.69.2”, for display purposes. Nothing is guaranteed about the format of this string. Required.
• “protocol version” (kOOTCPProtocolVersion): An integer encoding the version of the communication protocol being used, see above. Required.

Show Console

Oolite → Console
Symbolic constant: kOOTCPPacket_ShowConsole

Indicates that the console should make itself visible and frontmost.

Parameters: none.

Sample transcript

The following is a transcript of a session. (Similar transcripts can be acquired from the reference implementation by setting LOG_PACKETS to 1 in OOConsoleConnection.c.) It should result in console output similar to this screen shot:

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>oolite version</key>
	<string>1.69.2</string>
	<key>packet type</key>
	<string>Request Connection</string>
	<key>protocol version</key>
	<integer>65792</integer>
</dict>
</plist>

Console:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>console identity</key>
	<string>simpleDebugConsole</string>
	<key>packet type</key>
	<string>Approve Connection</string>
</dict>
</plist>

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>configuration</key>
	<dict>
		<key>command-background-color</key>
		<array>
			<real>0.87999999523162842</real>
			<real>0.87999999523162842</real>
			<real>1</real>
			<real>1</real>
		</array>
		<key>command-error-background-color</key>
		<array>
			<real>1</real>
			<real>0.87999999523162842</real>
			<real>0.87999999523162842</real>
			<real>1</real>
		</array>
		<key>command-exception-background-color</key>
		<array>
			<real>1</real>
			<real>0.75</real>
			<real>0.91666662693023682</real>
			<real>1</real>
		</array>
		<key>command-result-background-color</key>
		<array>
			<real>0.80000001192092896</real>
			<real>1</real>
			<real>0.80000001192092896</real>
			<real>1</real>
		</array>
		<key>console-host</key>
		<string>127.0.0.1</string>
		<key>error-background-color</key>
		<array>
			<real>1</real>
			<real>0.75</real>
			<real>0.75</real>
			<real>1</real>
		</array>
		<key>exception-background-color</key>
		<array>
			<real>1</real>
			<real>0.75</real>
			<real>0.91666662693023682</real>
			<real>1</real>
		</array>
		<key>font-face</key>
		<string>Courier</string>
		<key>font-size</key>
		<string>12</string>
		<key>general-background-color</key>
		<array>
			<real>1</real>
			<real>1</real>
			<real>1</real>
			<real>1</real>
		</array>
		<key>general-foreground-color</key>
		<array>
			<real>0.0</real>
			<real>0.0</real>
			<real>0.0</real>
			<real>1</real>
		</array>
		<key>macro-error-background-color</key>
		<array>
			<real>1</real>
			<real>0.87999999523162842</real>
			<real>0.87999999523162842</real>
			<real>1</real>
		</array>
		<key>macro-expansion-background-color</key>
		<array>
			<real>0.93999999761581421</real>
			<real>0.93999999761581421</real>
			<real>1</real>
			<real>1</real>
		</array>
		<key>macro-expansion-foreground-color</key>
		<array>
			<real>0.3333333432674408</real>
			<real>0.3333333432674408</real>
			<real>0.3333333432674408</real>
			<real>1</real>
		</array>
		<key>macro-info-background-color</key>
		<array>
			<real>0.80000001192092896</real>
			<real>1</real>
			<real>0.80000001192092896</real>
			<real>1</real>
		</array>
		<key>macro-warning-background-color</key>
		<array>
			<real>1</real>
			<real>1</real>
			<real>0.87999999523162842</real>
			<real>1</real>
		</array>
		<key>macros</key>
		<dict>
			<key>:</key>
			<string>player.call(PARAM)</string>
			<key>bgColor</key>
			<string>setColorFromString(PARAM, 'background')</string>
			<key>clear</key>
			<string>console.clearConsole()</string>
			<key>clr</key>
			<string>console.clearConsole()</string>
			<key>d</key>
			<string>dumpObjectLong(eval(PARAM))</string>
			<key>delM</key>
			<string>deleteMacro(PARAM)</string>
			<key>ds</key>
			<string>dumpObjectShort(eval(PARAM))</string>
			<key>fgColor</key>
			<string>setColorFromString(PARAM, 'foreground')</string>
			<key>listM</key>
			<string>for (let prop in macros) { ConsoleMessage('macro-list', ':' + prop) }</string>
			<key>resetM</key>
			<string>delete console.settings.macros; macros = console.settings.macros; undefined</string>
			<key>rmBgColor</key>
			<string>delete console.settings[PARAM + '-background-color']; undefined</string>
			<key>rmFgColor</key>
			<string>delete console.settings[PARAM + '-foreground-color']; undefined</string>
			<key>setM</key>
			<string>setMacro(PARAM)</string>
			<key>showM</key>
			<string>showMacro(PARAM)</string>
		</dict>
		<key>show-console-on-error</key>
		<true/>
		<key>show-console-on-log</key>
		<false/>
		<key>show-console-on-warning</key>
		<true/>
		<key>unknown-macro-background-color</key>
		<array>
			<real>1</real>
			<real>1</real>
			<real>0.87999999523162842</real>
			<real>1</real>
		</array>
		<key>warning-background-color</key>
		<array>
			<real>1</real>
			<real>1</real>
			<real>0.75</real>
			<real>1</real>
		</array>
	</dict>
	<key>packet type</key>
	<string>Note Configuration</string>
</dict>
</plist>

Console:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>message</key>
	<string>system.mainStation</string>
	<key>packet type</key>
	<string>Perform Command</string>
</dict>
</plist>

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>color key</key>
	<string>command</string>
	<key>emphasis ranges</key>
	<array>
		<integer>2</integer>
		<integer>18</integer>
	</array>
	<key>message</key>
	<string>&gt; system.mainStation</string>
	<key>packet type</key>
	<string>Console Output</string>
</dict>
</plist>

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>color key</key>
	<string>command-result</string>
	<key>message</key>
	<string>[Station "Coriolis Station" "Coriolis Station" ID: 105 position: (-49515.3, 60769.4, 427622) scanClass: CLASS_STATION status: STATUS_ACTIVE]</string>
	<key>packet type</key>
	<string>Console Output</string>
</dict>
</plist>

Console:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>message</key>
	<string>console.settings["show-console-on-log"]</string>
	<key>packet type</key>
	<string>Perform Command</string>
</dict>
</plist>

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>color key</key>
	<string>command</string>
	<key>emphasis ranges</key>
	<array>
		<integer>2</integer>
		<integer>39</integer>
	</array>
	<key>message</key>
	<string>&gt; console.settings["show-console-on-log"]</string>
	<key>packet type</key>
	<string>Console Output</string>
</dict>
</plist>

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>color key</key>
	<string>command-result</string>
	<key>message</key>
	<string>false</string>
	<key>packet type</key>
	<string>Console Output</string>
</dict>
</plist>

Console:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>message</key>
	<string>player.target.call("becomeExplosion")</string>
	<key>packet type</key>
	<string>Perform Command</string>
</dict>
</plist>

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>color key</key>
	<string>command</string>
	<key>emphasis ranges</key>
	<array>
		<integer>2</integer>
		<integer>37</integer>
	</array>
	<key>message</key>
	<string>&gt; player.target.call("becomeExplosion")</string>
	<key>packet type</key>
	<string>Console Output</string>
</dict>
</plist>

Oolite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>message</key>
	<string>Oolite is terminating.</string>
	<key>packet type</key>
	<string>Close Connection</string>
</dict>
</plist>