Artemis protocol hacking

Firstly: if you were looking for Ben's Mod go there, or my Artemis Cloud Server, AKA "Nebula", otherwise...

My Artemis Protocol Parser, and other useful stuff...

A totally unofficial, unsupported copy of some of my Artemis protocol-hacking tools, for decoding the binary TCP protocol used between Artemis clients and servers. My work is based heavily on the artemis-nerds protocol-docs, with many thanks to chrivers, rjwut, AwesomeAidenW, @Starry, @Slate, and many others

Artemis servers listen on TCP port 2010. The simplest way to capture a recording of an Artemis game USED TO BE with netcat:
```
  nc $server 2010 > file
```
Where $server is the name/address of your server (EG 127.0.0.1 if running on the same machine). This captured file will contain most but not quite all of the interesting server-to-client packets. Unfortunately recent Artemis clients send a "ClientHeartbeat" packet every 3 seconds, and the server will disconnect you after a while if you don't send it.
The protocol-docs (written mostly by rjwut and others, with some bug-fixes / contributions from me) describe the contents of the data in detail. See also chrivers' version which is built from a machine-readable protocol definition, and includes a few updates. Also PLEASE check the protocol-docs "issues" if you think you found something new, there's a good chance that we already know but it hasn't made it into the docs yet.
parser.pl is my parser, written in Perl. This is probably what you came here for? Given one of the above captures (or gzip/xzipped versions thereof) it will convert the binary into a somewhat more human-readable description of the contents. The output is, in fact, actually a Perl program. If you run this program it will (using parser.pl as a module) recreate the same binary data.
parser.pl accepts a whole bunch of different options - see parser.pl --help for a quick list or parser.pl --man for a full manual page. The most useful one is probably --noobject to not dump the noisy (and CPU-hungry) objectBitStream packets (but note you're then not going to be able to reproduce the complete original binary).
It should be noted that parser.pl changes often.
Most of the actual packet parsers look a bit like:
```
  $parsers{0x12345678}{0xab} = sub {
    # Parser for packet type 0x12345678 subtype 0xab:
    my ($a,$b,$c) = unpack('VVC', shift);
    print "FunctionName($a,$b,$c);\n";
  };
  sub FunctionName {
    # Code to recreate the binary for a (server to client) 0x12345678:0xab packet:
    s2c pack('VV VVC', 0x12345678,0xab, @_);
  }
```
It definitely helps to understand Perl's somewhat arcane pack() formats - they are mystic runes, but super-efficient at converting a list of values, of various types, to/from binary formats. The important ones are:
- f< - 32-bit "single-precision" float, little-endian
- V - 32-bit unsigned integer, little-endian
- v - 16-bit unsigned integer, little-endian
- C - 8-bit unsigned char
- V/v - 32-bit LE string length, followed by that many UTF16 chars (which is how Artemis stores strings)
- V XXXX V/v - as above except we want to keep the string length first. This actually means "read the u32, go back 4 bytes (XXXX), then re-read the u32 length and that many (/) u16s"
parser.pl now handles all Artemis 2.000000 to 2.7.5 packets cleanly
mainscreen.pl is a short program that asks to be a "main screen" console, useful because this receives a few more packets, including the very interesting GameOverReason and GameOverStats. mainscreen.pl can also/alternatively ask to be a fighter, it can also ask to be on ship 2/3/4/etc. This provides a useful improvement on the above netcat program - you can now do something like:
```
  touch gameover.txt    # create a dummy file - see below
  ./mainscreen.pl 2 main fighter heart gameover.txt | nc $server 2010 | tee capture.bin | ./parser.pl --noobj --nodmx --quit gameover.txt
  
```
... which will select ship 2, main screen and fighters, and send heartbeats whilst waiting for gameover.txt to be deleted. The netcat will send this to your $server (127.0.0.1 or whatever) and collect all the server2client packets. The tee will save a copy of these to capture.bin. The parser.pl will simultaneously dump the captured data in real-time, except the objectBitStream and DmxMessage packets (which were still saved to capture.bin) and when it sees a GameOver packet, will delete gameover.txt, signalling to mainscreen.pl to exit (which in turn will close the netcat, the tee, and parser.pl itself). The above example is so amazingly useful that I have wrapped it up into...
logger.sh. This is usually run like:
```
  ./logger.sh --heart --loop 1 server.addr.ess "some short description"
```
... which will connect to the server.addr.ess on ship 1 [optional/default], and sit in a loop logging each game to a separate file in a subdirectory "captures" with names like captures/artemis-YYYY-MM-DD-HHMM.some.short.description.01s.xz (it xzips them after the game ends). The "01s" fits with a different naming convention (see portwatch.pl below) but basically means this is the 01st server stream with that timestamp. logger.sh exits the --loop if/when it gets a 0-size capture, which is almost always a symptom of netcat failing to connect to an Artemis server that has gone offline.
I have an archive of over 1500 Artemis game captures, from USN, Canadian 1st Light Division, LAN games at home, Artemis "Nebula" servers in the cloud and a few very boring "test" games. This archive is available for other artemis-nerds to use to test their own parsers.
deadbeef2hex.pl will take artemis captures ("deadbeef" refers to the packet header - read the protocol-docs) collected via the above netcat techniques, or pcap (EG from tcpdump / wireshark) and spit out a hexdump, compatible with portwatch.pl --hex, parser.pl --hex and some of chrivers' tools.
totals.pl will accept one of those Perl programs created by parser.pl and collect some interesting stats - how many of each type of packet, how many of each type of object in the objectBitStreams, total number of packets parsed, etc etc. It can also take the output of previous totals.pl runs and total THOSE together too, EG:
```
  ./parser.pl captures/artemis-01s.xz | ./totals.pl > artemis-01s.stats
  ./parser.pl captures/artemis-02s.xz | ./totals.pl > artemis-02s.stats
  ./parser.pl captures/artemis-03s.xz | ./totals.pl > artemis-03s.stats
  ./totals.pl artemis-0[123]s.stats > TOTALS
```
tester.sh is used to test parser.pl against a binary capture file (EG from netcat / logger.sh). It takes it from binary, to Perl, back to binary again, makes sure the binary output of that is absolutely identical to the original, and/or dumps some stats. I call this binary-to-Perl-to-binary process "There And Back Again" because I'm not only an Artemis nerd but a Tolkien nerd as well. Essentially, every single capture I've collected (via portwatch.pl, logger.sh / netcat, or other means) is used as a unit-test for my parser.pl and any improvements therein.
Makefile is a GNU Makefile which which will take your collection of captures/artemis-YYYY-MM-DD-HHMM-*.xz, and feed them through tester.sh, declare "pass" or "fail", make it easier to unit-test the parser against my big game recording archive and collect "total stats for all games" etc.
tailer.sh is a 1-line wrapper (well, OK, it's 3, but could have been 1) to kinda "tail a bunch of capture files, parsing as you go", useful when you want to "catch Artemis in the act". It feeds data through parser.pl [...] --test which is good for making sure you have "tested all of the things" - it will periodically remind you which packets/objects it has NOT seen.
portwatch.pl is another way to capture both server and client traffic. Instead of making a separate connection, this one sits as a "Man in the Middle" proxy, acting like a fake server, accepting client connections and forwarding them on to a real server whilst recording traffic in both directions. This tools is not actually Artemis-specific, I have used it as a MitM for a number of other projects long before I even encountered Artemis. For Artemis-hacking, I usually run it like:
```
  portwatch.pl 2010 127.0.0.1 2020 \
    --in  captures/artemis-YYYY-MM-DD-HHMM.#c \
    --out captures/artemis-YYYY-MM-DD-HHMM.#s
```
This accepts connections on the usual port 2010, forwards to a server running on port 2020 (set networkPort=2020 in artemis.ini). The clients have the usual networkPort=2010 but are connecting via the proxy.
portwatch.pl also supports data It also supports data injection - if you want to insert extra fake/spoofed packets as if they came from the real client or server, you can use --inject file.in and/or --outject file.out, probably in conjunction with short code snippets like:
```
  #!/usr/bin/perl -w
  require 'parser.pl';
  Poof(87500,0,25000);
```
... and then ./test.pl >> file.out to pretend that packet came from the server, and see how the client reacts (in this case it renders the fuzzy green cloak/decloak "poof!" kinda thing at those coords)
JamCRC.pl implements the JamCRC checksum/hash, used by .net or C# or whatever Thom Robertson uses to write Artemis. The packet IDs, it turns out, make more sense if you realise they are packet names that have been JamCRC'd - thanks to chrivers for finding this out. If Thom adds any new fundamental packet types (EG recently 0x902f0b1a "bigMess" and 0xbe991309 idleText), you should be able to find the corresponding official name by running:
```
  strings -el Artemis.exe | ./JamCRC.pl
  # ... except sometimes the strings have other crap before them so...
  strings -el Artemis.exe \
  | sed -e 'P;  s/^.//;  P;  s/^.//;  P;  s/^.//' \
  | ./JamCRC.pl | grep ^0x902f0b1a # or whatever you packetID is
  # and in this example you'll find "0x902f0b1a bigMess"
```
marfez.pl is a VERY alpha-quality reverse-engineering tool (named after Zefram Cochrane in reverse, get it?). It takes the above hex format (or better still, a grep of a particular packet/object you're interested in) and tries to "help" reverse-engineer the packet formats for you. It looks at the collection of packets/objects of different lengths, and tries to spot which bytes look like strings, ints, floats, etc. It definitely can't tell you exactly how to decode a packet, but it might strongly imply that a particular packet seems to contain 2 integers and 3 floats, it might even tell you the approximate min/max of each, then leave you to work out that these correspond to an ObjectID, something, and X,Y,Z coords. It's a tool, it's not magic :-p
disco-server.pl D I S C O ! - a totally useless fake Artemis server which won't let you fly anywhere, but will let you P A R T A Y !
May happen to be available on noseynick.net right now if you want to connect your Artemis client to it :-)
I have a somewhat functional Artemis Cloud Server, AKA "Nebula", which builds an Artemis Server "from scratch" in AWS (or other clouds) in less than 5 mins. In AWS it costs about 17c/hr to run - rememeber how long 50c used to last down at the arcade? How would you like 3 hours of Artemis?
checkpoint.pm takes advantage of a new feature since v1.008 - parser.pl --hook checkpoint.pm captures/artemis-blah | perl will output perl which, instead of using parser.pl to output the original binary, will instead analyse all the objectBitstream packets, collect info about all objects "in game", and output them once per heartbeat.
starry-lisp.pm is essentially the same, except every heartbeat() it will output a lisp-formatted set of ObjPlayerShips and ObjNPCShips in a format that @Starry wanted. Use like: parser.pl --hook starry-lisp.pm captures/artemis-blah | perl
spawn.pm works in a similar way - you run parser.pl --hook spawn.pm captures/artemis-blah | perl and instead of recreating the original binary, it gives info when when objects are spawned or deleted, specifically what they were spawned or deleted NEAR. This should help answer questions like "who shot Torp 1234", or "drone 2345 hit who?", or "What was sucked into blackhole 3456?"
snt-dec.pl is a short script to decode Artemis *.snt files which describe the ship internals - roughly the floor plans / ship systems that Engineering sees and repairs.
snt.txt and snt.html are the latest output from this - floor plans for all the Player Ships!
files.pl scans a directory of Artemis files, telling you which ones are mentionned in Artemis.exe, vesselData.xml, or any of your mission scripts. files.html is the output for my Artemis 2.7.5 directory. It requires an HTML template like template.html
TSN-CA-Ships.pl is a script that uses parser.pl to set the names, ship types, and shuttle/fighter configurations of the TSN Canadian Fleet. To do this, you'll need to connect it to a server, EG:
```
  ./TSN-Artemis.ca.pl | nc $server 2010 | ./parser.pl
```
TSN-RP-Ships.pl - as above but for the TSN RolePlaying group fleet of ships
EFShips.pl - as above but for the Eastern Front fleet.
discord.sh is a short script to send stuff to a discord server. You'll also need a creds.sh containing your discord bot's credentials - this will NOT be shared here for obvious security reasons, but it's quite simple, and the format is described in the top of discord.sh itself. There's a parser.pl --discord and corresponding logger.sh --discord option that you may have spotted if you're really observant. These use discord.sh to send GameOverStats to the #scoring_and_games channel on the United Stellar Navy Discord server.
puppy-poll.sh is a Proof-of-Concept potential replacement for "Simple Poll ^BOT", but doesn't require "show website previews" enabled, and uses letters instead of numbers for the options - I believe A 3 B 1 C 5 should be less confusing than 1 3 2 1 3 5.
This should eventually be rolled into the Puppy bot... speaking of which...

Ideas for the future...

discord.sh (above) is good for just sending scores and snippets of stuff to a channel, but really USN discord needs a more interactive bot, whose initial purpose will be to watch for games starting, and kick off a corresponding logger.sh --discord $server_name_or_ip to watch your game(s) and report the game type, level, results stats. The bot became known as "@Artemis-Puppy", companion to the "@Artemis Kitten" AKA "@Artemis-Leaderboard" who logs scores for USN players.
The Artemis Cloud Server "Nebula" is about 70% automated. It would be nice to be 100% automated, and integrate with my bot/puppy so people can ask to start a server, Ben's mod, level 7 single-front, and 5 mins later have an address you can connect to, and some way to start once everyone is ready, shutdown after everyone is done, or after N hours if something seems to have gone wrong
It will also need an ability to say "please disconnect" if people are bandwidth-starved enough that 1 client is enough to break the camel's back.
I have used the above portwatch.pl, parser.pl, totals.pl, to examine exactly which packets are sent to exactly which consoles (Main Screen, Helm, Weapons, etc etc). There's honestly not much filtering going on, in particular it seems every station (or even just a quiet "netcat" that never joined any stations) get all the objectBitStream packets, and almost everything else. There are some packets only sent to Main Screen (GameOverReason/Stats), some only to fighters (fighter bay status / launch, possibly some others), some only to Eng, etc. I have a nice analysis I intend to get online ASAP, but for now, the nutshell is it doesn't matter much, the server doesn't filter much, each client is well within about 5-10% of any other client.
USN/TSN1D historical records archive! Take my archive, but add Stats! Maps! It should be possible to go back to any of those games, and at least see the game stats, but probably also draw "strategy maps" showing where ships flew throughout the game etc
Action replays - should also be possible to pick a game from the archive, and connect to a special server, where you could then be "Main Screen", "Observer", "Data", or any of the other non-interactive consoles, to re-watch any archived games. The server could pretty much just replay most/all of the packets in "real-time" or "double speed", should be fairly easy. Fast-Forward / Rewind / Seek to particular times harder. Would probably require controls in a browser because I can't add controls to a client.
Decode controls.ini into pretty HTML keyboard maps? See also HaydenBarca's controls.ini re-ordered to be more readable