Windows Socket Functions in Ruby Extensions

If you want to use Windows socket functions (such as WSARecv, and WSASend) inside a Ruby extension on Windows, you may have your work cut out for you. Here’s a helpful work-around I discovered during a recent project.

Spotting the Problem

I was working on an improvement to the rENet library, which is a Ruby binding to the excellent ENet library. Previously, building the extension required preinstalling the ENet library on your system. But ENet is a nice, small library with no other dependencies, so I thought I’d try to build it into the extension. This would be much more convenient to users of the library, as the Ruby extension builder would build ENet automatically without requiring pre-installation.

My first attempt worked great on Linux and OSX systems, but to my bafflement, it failed in spectacular ways on Windows systems. After a little debugging, I found that most operations would work fine, but some would give:

WSAERROR 10038 : (WSAENOTSOCK): Socket operation on non socket.

What was especially strange is that, with no code changes, everything would work great if I just linked to an ENet library that was built externally—not by the Ruby extension builder. It appeared that the Ruby extension builder was messing up some, but not all, of my socket operations.

Some googling and the headers in the extension builder source hinted that Ruby was creating its own wrappers for socket functions. But I had to dig all the way down into the Ruby source code to figure out what was actually going on and how I could fix it.

Finding the Cause

Internally, Ruby wraps the Windows socket functions with its own functions. These wrappers replace the normal socket handles with a homemade “file descriptor,” which is managed internally by Ruby.

For an example of what I’m talking about, search for: rb_w32_socket here.

I’m assuming this is done to make the internal APIs more consistent across OSs. For example, it lets you use select() to block on either a file or socket IO event in the same call. This is a common technique on POSIX systems, but is normally not possible on Windows.

The problem arises when this fake socket handle gets mixed in with an unwrapped function that expects a real one, or vice versa. That’s what caused my WSAENOTSOCK error. The function was expecting a real socket, but got the fake handle instead.

Unfortunately, not only does Ruby wrap the socket functions; it also replaces the symbols for the wrapped functions at the linker level so it is nearly impossible to use the native Windows socket functions.

Choosing the Best Solution

Once you understand what’s going on, there are two potential solutions to this problem.

1. Make your own wrappers for the functions that are causing problems.
2. Use an ugly hack to go around Ruby and use the native functions.

If your extension needs to share socket handles with other Ruby code, you have no choice and must use Option 1. Unfortunately, this is the harder option. You’re pretty much going to have to dig into the Ruby source and try to copy what they are doing, but for your own wrapper. This might be a good place to start digging around for something close that you can copy.

If your extension does not need to share socket handles with other Ruby code and is going to manage the sockets completely internally, you can use either option. However, Option 2 is probably easier and less error-prone.

Here’s the code I used to get the actual Windows socket syscalls, instead of the wrapped ones provided by Ruby:

bind_ptr        = (LPFN_BIND)       GetProcAddress(GetModuleHandleA("ws2_32.dll"), "bind");
getsockname_ptr = (LPFN_GETSOCKNAME)GetProcAddress(GetModuleHandleA("ws2_32.dll"), "getsockname");
listen_ptr      = (LPFN_LISTEN)     GetProcAddress(GetModuleHandleA("ws2_32.dll"), "listen");
socket_ptr      = (LPFN_SOCKET)     GetProcAddress(GetModuleHandleA("ws2_32.dll"), "socket");
ioctlsocket_ptr = (LPFN_IOCTLSOCKET)GetProcAddress(GetModuleHandleA("ws2_32.dll"), "ioctlsocket");
setsockopt_ptr  = (LPFN_SETSOCKOPT) GetProcAddress(GetModuleHandleA("ws2_32.dll"), "setsockopt");
getsockopt_ptr  = (LPFN_GETSOCKOPT) GetProcAddress(GetModuleHandleA("ws2_32.dll"), "getsockopt");
connect_ptr     = (LPFN_CONNECT)    GetProcAddress(GetModuleHandleA("ws2_32.dll"), "connect");
accept_ptr      = (LPFN_ACCEPT)     GetProcAddress(GetModuleHandleA("ws2_32.dll"), "accept");
shutdown_ptr    = (LPFN_SHUTDOWN)   GetProcAddress(GetModuleHandleA("ws2_32.dll"), "shutdown");
closesocket_ptr = (LPFN_CLOSESOCKET)GetProcAddress(GetModuleHandleA("ws2_32.dll"), "closesocket");
select_ptr      = (LPFN_SELECT)     GetProcAddress(GetModuleHandleA("ws2_32.dll"), "select");

Then, simply replace all the calls to functions such as bind and select to calls on appropriate function pointer, such as bind_ptr and select_ptr.

In Ruby 2.0 or higher, you’ll probably need to add a +#define INCL_WINSOCK_API_TYPEDEFS 1 to gain visiblity to the pointer types. It’s a super-gross solution, but it happens to work quite well.