Hacking EVE, Part 2 – Reading Market Data

Article summary

An Easy but Inaccurate Method
A Useful Pricing Model
The Implementation
Our Results

Welcome back EVE industrialists! Or anyone interested in using Ruby scripting and web APIs to do some data mining! This is the second post in my series on hacking EVE.

Our long-term goal is to use automation to determine which items will be most profitable for an EVE industrialist to build. In part 1 of this series, I talked about using Ruby, RSpec, ActiveRecord, and MySQL to read static data from a MySQL export file. This post will discuss using Nokogiri and Moneta to read dynamic market data and help us to determine accurate prices for our products.

An Easy but Inaccurate Method

While players have long requested that CCP provide official market data APIs… they do not. Fortunately we have some 3rd-party sites that do. I’ll be using EVE-Central’s API, although you could certainly use the EVE Marketdata’s API if you prefer.

EVE-Central provides two APIs that look promising. The first is the marketstat API. This API requires that you pass the typeId that you’re interested in like:

http://api.eve-central.com/api/marketstat?typeid=34

It then returns a document including statistics on the buy and sell orders for that item type:


	
		
			
				185093933839
				4.02
				6.32
				1.30
				0.74
				4.26
				4.78
			
			
				80623224020
				5.26
				16.00
				2.88
				0.95
				4.96
				4.29
			
			
				273991794319
				4.29
				12.50
				0.20
				1.07
				4.41
				1.30

Using Nokogiri, we can pretty easily download, parse, and extract the interesting bits of information here:

#!/usr/bin/ruby
require 'nokogiri'
require 'open-uri'

abort("Please pass an item id to be looked up.") unless ARGV.length > 0
typeID = ARGV[0]
api = "http://api.eve-central.com/api/marketstat?typeid=#{typeID}"
puts "Getting market data: #{typeID} ..."
marketData = Nokogiri::XML(open(api))
price = marketData.css("all avg")[0].content
puts "The average market price for type #{typeID} is #{price}"

And get output like the following:

$ ruby marketStat.rb 34
Getting market data: 34 ...
The average market price for type 34 is 4.29

If you’re looking for a ballpark number, say for a killboard, this may be all you need. I find that for modeling my actual build profits though, these numbers are fairly useless. There are a number of reasons for this, including:

I’m not looking for rough numbers — I’m looking for the most accurate model I can find. Five percent off is a big deal in terms of profit margins.
Players deliberately post wildly high and low market orders to throw off averages, so I want to filter out that noise.
There are only so many jumps I’m willing to make to gather build resources and take goods to market, so only some markets really matter to me.
I generally don’t want to wait 3 months for my items to sell, so I’m really only interested in the lower end of the sell orders.

To be fair to marketstat, you can specify a particular system or region that you want to see prices from and use the minimum sell order value as your price. Even using those changes however, I still tend to get prices that don’t match up particularly well to reality. What I want is to be able to:

Use data only from the markets in which I’m likely to buy or sell items. It’s not profitable to fly all over the universe to save an ISK or two; so for planning purposes, only the prices in the markets I use matter.
Average the low end of the sell orders, say the bottom five. I don’t like to count on selling products at prices much higher than that, as I prefer to move items quickly.
I am, however, likely to buy in the markets with the lowest prices and sell in the markets with the highest, so I want to take that into account.

A Useful Pricing Model

So what I’d like to do in code is this:

require './models/pricing/quick_look_data.rb'
require './models/pricing/low_sell_orders_pricing_model.rb'
require './models/pricing/composite_pricing_model.rb'
require './models/pricing/persistant_pricing_model.rb'

# First build a pricing model for each of the systems we care about.
# We'll do better than these magic system and type id numbers in later posts. 
jita = LowSellOrdersPricingModel.new(QuickLookData.new(usesystem: 30000142))
amarr = LowSellOrdersPricingModel.new(QuickLookData.new(usesystem: 30002187))

# Next we'll build a model that gives us the composite data from those systems.
markets = CompositePricingModel.new([jita, amarr])

# Persist it so we don't wear out the eve-central server.
pricing = PersistantPricingModel.new(markets)

# And now we should be able to get useful pricing data.
puts "Tritanium buy price: #{pricing.buy_price(34)}"
puts "Tritanium sell price: #{pricing.sell_price(34)}"

We’ll take the average of the lowest sell orders in each system we care about as the price of the item in that system, then have a composite pricing model that assumes we’ll buy in the cheapest system and sell in the most expensive one. Finally we’ll use a wrapper that persists the data we get back so that we can run our scripts more quickly. This won’t matter in our simple case but gets pretty important if you want to, say, find the profit margin of every buildable item in the EVE universe.

The Implementation

The full source of my tests and models is available on GitHub, but I’ll highlight the more interesting bits.

First, the code to build the quicklook api and open a stream to the eve-central server are pretty straightforward:

require 'uri/http'
require 'cgi'
require 'open-uri'

class QuickLookData

	def initialize(options = {})
		@query = options
	end

	def uri(extraQueryArgs = {})
		args = {}
		args[:host] = 'api.eve-central.com'
		args[:path] = '/api/quicklook'

		query = @query.merge(extraQueryArgs).map {|k,v| "#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}"}.	join("&")
		args[:query] = query unless query.empty? 

		URI::HTTP.build(args)
	end

	def data(args)
		open(uri(args))
	end
end

I’ll call data(:typeid => 34) to load a stream for the tritanium quicklook entries, and initialize my QuickLookData instances with (:usesystem => systemId) to only load the entries from systems I care about. The open-uri open(uri) method is all I need to fetch the feed.

Next I’ll use that data in my LowSellOrdersPricingModel.price method:

	def price(id)
		return @prices[id] if @prices.has_key?(id)
		print "Getting market data: #{id} ..."
		marketData = Nokogiri::XML(@dataSource.data(:typeid => id))
		puts " Done."
		prices = marketData.css("sell_orders order price").
			collect {|n| n.content.to_f}.
			sort.
			take(5)
		price = prices.size > 0 ?
			prices.inject(0.0) {|sum, el| sum + el} / prices.size :
			0
		@prices[id] = price
		price
	end

I parse the document using the Nokogiri library, grab the sell orders I’m interested in via the css lookup method, and average the lowest five prices.

Next I use some basic functional programming to implement the composite model:

class CompositePricingModel

	def initialize(models = [])
		@models = models
	end

	def buy_price(id)
		return 0 if @models.empty?
		@models.collect {|m| m.buy_price(id)}.min
	end	

	def sell_price(id)
		return 0 if @models.empty?
		@models.collect {|m| m.sell_price(id)}.max
	end	
end

And finally I use Moneta to persist the values so that I can be a good EVE-Central API citizen. The code defaults to updating prices that are more than a day old:

require 'moneta'

class PersistantPricingModel

	def initialize(model, baseDir = './data', expiration = 60 * 60 * 24)
		@model = model
		@buy_prices = Moneta.new(:PStore, :file => baseDir + "/buy.pstore", :expires => true)
		@sell_prices = Moneta.new(:PStore, :file => baseDir + "/sell.pstore", :expires => true)
		@expiration = expiration
	end

	def buy_price(id)
		return @buy_prices[id] if @buy_prices.key?(id)
		price = @model.buy_price(id)
		@buy_prices.store(id, price, :expires => @expiration)
		price
	end	

	def sell_price(id)
		return @sell_prices[id] if @sell_prices.key?(id)
		price = @model.sell_price(id)
		@sell_prices.store(id, price, :expires => @expiration)
		price
	end	
end

Our Results

Et voila, we have relevant pricing data for the markets we care about. At the time of writing, the output from our test script is:

$ ruby pricing.rb
Getting market data: 34 ... Done.
Getting market data: 34 ... Done.
Tritanium buy price: 4.63
Tritanium sell price: 4.774

The super-observant will notice that the actual price we can expect to buy tritanium at in our markets is almost 8% higher than the marketstat overall average and 12% lower than the marketstat sell average. This is actually a best case scenario as we’re using the most commonly bought and sold item in EVE – the margins for most items will be much worse.

Next time around, we’ll use the static data from Part 1 along with the dynamic pricing data we’ve found here to see what kind of profit we can expect to make on a specific item.