Formatting AWS CloudFront and ELB Logs for Easy Review

Access logs from AWS CloudFront distributions and AWS Elastic Load Balancers can be essential to diagnosing problems with an AWS infrastructure. AWS provides the ability to store these logs in AWS S3 buckets.

However, the log files are often in very many small files which need to be combined in order to get a full picture of the traffic that they represent. In order to make this process easier, I wrote a few scripts which help me to quickly download and format the logs in a usable format for review. Read more on Formatting AWS CloudFront and ELB Logs for Easy Review…

Avoiding Incomplete/Corrupted Files During Processing Operations

Applications that work with files on disk can encounter incomplete or corrupted files if a target file is actively being written to disk by another process. Typically, this happens when two different systems or processes are interacting with the same file independently. Read more on Avoiding Incomplete/Corrupted Files During Processing Operations…

Five of My Favorite Command-Line Utilities

I spend a lot of time on the command line (generally, GNU/Linux), and often work on automating processes and tasks. My work often occurs on a remote machine to which I do not have access, and it generally must be headless (no GUI).

As a result, I have collected an arsenal of command-line interface (CLI) utilities that I always install when I’m setting up a new development machine for myself. There are often graphical analogues for these utilities, but I prefer these because of their CLI. Below is a sampling of my favorites. Read more on Five of My Favorite Command-Line Utilities…

A Script for Deploying Docker-Hosted Applications to AWS ECS

I have recently started relying more on AWS Elastic Container Service (ECS) to deploy applications. To assist with the process of building Docker images, pushing the images up to an AWS Elatic Container Repository (ECR), updating an existing task definition to make use of the new image, and updating an ECS cluster service to use that new task definition, I wrote a fairly simple script in Bash and Python to standardize and automate this process. Read more on A Script for Deploying Docker-Hosted Applications to AWS ECS…

Sharing an EFS Filesystem Across ECS Instances, Services, and Tasks

In application hosting scenarios, it is common to have data that must be available to all application instances. Often, this data is stored in a database (e.g. AWS RDS), or some other centralized repository (e.g. AWS S3). However, situations arise where third-party software expects data to be on the filesystem, or where application instances generate files which must be immediately available to all other application instances. Read more on Sharing an EFS Filesystem Across ECS Instances, Services, and Tasks…

Don’t Forget to Kick the Tires – On Automatic Monitoring & Human Intervention

With as much as we use modern technology to automatically monitor, observe, and report on so many different systems, I think it is important to manually check the viability of these same systems from time to time.

While the quality of sensors and metrics always seems to be improving, there are still loopholes and lapses which make human oversight and investigation important. I have experienced a few situations in the past few months where the electronic/computer-based monitoring solution for systems have failed in ways that could have been particularly costly; yet human intuition and manual intervention ensured that the situation was addressed appropriately to avoid real damage or loss. Read more on Don’t Forget to Kick the Tires – On Automatic Monitoring & Human Intervention…

Utility to Initiate the Rebuild of Indices for CouchDB Design Document Views

I recently worked on upgrading a CouchDB instance and migrating it to a new server. Because of the upgrade and migration path, I needed to dump all data from the existing CouchDB, and then load it into the new CouchDB.

Fortunately, the Python libraries for working with CouchDB provide a convenient set of utilities to snapshot a CouchDB instance to a MIME multipart file, and then load that MIME multipart file into a new CouchDB. Although the utility handles all documents, attachments, and design documents, it does not provide a way to initiate a rebuild of the indices for views associated with design documents. I wrote a simple utility to aid in this process. Read more on Utility to Initiate the Rebuild of Indices for CouchDB Design Document Views…

AWS IoT Help Button

We recently moved into our new building at 1034 Wealthy in Grand Rapids. The new building is much larger than our old one, and I find myself running around much more and stationing myself in different areas, depending on what I am actively doing.

To help ensure I can provide timely hands-on help—particularly for our printer (which is a common source of problems)—I procured one of the new AWS IoT buttons and programmed it to page me when pushed. Read more on AWS IoT Help Button…

Conference Room A/V Build-Out

We recently moved to our new building at 1034 Wealthy. We took the opportunity to update the A/V equipment for our conference rooms. Previously, we largely relied on projectors for presentation capabilities, an external USB microphone/speaker for audio, built-in webcams on laptops for video, and a table where we staged everything. This worked, but it was certainly not ideal. With the new building, I had the opportunity to standardize a new conference room A/V build-out that would be better suited to our needs. Read more on Conference Room A/V Build-Out…