Article summary
Recently, I was met with a situation that you do not wish to run into as a software developer. I was playing a game of baseball, and due to a wild pitch, I broke my wrist. This resulted in being put into a cast (and for a moment a sling) that made it very difficult to type or use a keyboard. I had a moment of worry. I wondered how I would be able to continue to keep up my workload while not being able to produce code as effectively. This concern was worsened by the fact that around the time of the incident, I was transitioning from a larger project team to a project where I would be the one lone developer. I was unsure what I was going to do.
Luckily, following a conversation with a coworker, I was introduced to the helpful tooling accessible via Codex that would help compensate for my currently broken arm. Over the past several weeks, I have found Codex tools for improving accessibility very useful and thought I would share what I have learned.
Dictation
The first accessibility tool I used within Codex was dictation. Dictation also ended up being the tool I used the most amongst Codex’s accessibility features. The dictation tool allows you to set a hotkey that, when pressed, will open up your microphone to allow you to talk to Codex. Codex will then transcribe what you are saying into text wherever you have your cursor placed within your computer. I found this tool incredibly helpful for writing Slack messages, writing emails, and writing context to pass into Codex itself. I’m even using the dictation tool to type this blog post right now.
The dictation tool is useful for writing down what you would normally type, but it’s a lot quicker as you are only speaking at your computer instead of having to manually press the keys to get what you want to appear on screen. The dictation tool does have its limitations, though. There are a good number of times where I said something into my microphone and the dictation tool misinterpreted what I had said, either because I was too far away from the mic or did not enunciate enough. That meant I would have to go back and manually fix the mistakes (I cannot express how many times I had to correct the dictation tool when I wanted to say “throw” and it wrote “bro”).
To help account for this, like many dictation tools, the tool’s settings have a dictionary where you can define specific words. That way, when you speak those specific words to the dictation tool, it is more likely to correctly write what you intended. Overall, I found it an incredibly useful tool that I will likely continue to use even once my arm is completely healed.
Interfacing With Your Device
The next set of tools from Codex that I folded into my workflow were its integration tools. Integration tools are what allow ChatGPT and/or Codex to integrate with other applications on your device.
App Shots
The first of these tools that I commonly used was app shots. App shots are similar to screenshots, but provide further text information to feed into the agent outside of just the image. If you are app-shotting something, say like a web page, it’ll also collect content from that page that may be currently off-screen or not visible to you, but still have it accessible and visible to the agent for greater context.
I would often find myself using this feature in combination with dictation to provide an agent with an appshot of what I’m currently working on, and then use dictation to tell the agent what I am looking to update or change or investigate about said appshot. I found it very useful compared to copying and pasting code or text into the context of an agent, especially when being able to hit the copy and paste binds were not as simple anymore with a broken arm.
Computer Use
Besides appshots, I also found the Computer Use plugin for ChatGPT to be quite useful. The computer use plugin provides ChatGPT or Codex with a skill that allows it to access and use different things on your device. It can do this by taking screenshots, app shots, or screen sharing to ChatGPT, and using its own cursor to navigate and do things within an application that you request of it. A nice safety feature that I also noticed while using this feature is that when you request it to do something within an application, it will prompt you to allow it to access that application before doing anything, so that you don’t inadvertently try accessing applications you don’t want the AI to access.
In my situation, luckily I could still use my mouse fairly easily and without much complication. So this feature did not come into play much for me. However, I do see its use being very helpful for someone who is in a worse situation than me. They could continue to use their device without being inhibited by the fact that they would not be able to easily manage the input and output devices.
The Tools Are Getting Better
The tools I have discussed are just the tip of the iceberg of accessibility improvements and features within the Codex application. Especially with recent improvements within the ChatGPT’s voice functionality, which ties together the tools I had previously mentioned into a tool that feels almost like a personal assistant working your device for you. What started as a temporary workaround has become a meaningful addition to how I work. Even after my arm fully heals, I expect many of these tools to remain part of my workflow because they make development feel more flexible, approachable, and resilient.