Somewhere in your organization, someone has built or downloaded a piece of software you do not know about. It might be a browser extension, a small app that automates a report, or custom firmware on a device sitting on their desk. It was probably built with an AI tool, and nobody has checked the code, where its parts came from, or everything it does.
This is the age of software hyper-personalization. AI coding tools have made it easier than ever to change how software systems behave, reverse engineer software and device firmware, and build bespoke interfaces to meet unique wishes. It is both the tinkerer’s dream, and a software security nightmare.
Software personalization isn’t completely new. In 2005, I ran Linux as my laptop operating system at work. That involved a whole ecosystem of open-source software, including a window manager that I downloaded and compiled just to have something unique. I didn’t read all that code or do a deep analysis of the software supply chain. My trust was based on community sentiment and the relative rarity of visible security issues or supply chain attacks. But by 2008, my whole team had transitioned to company-managed Mac laptops to improve trust and consistency of our software ecosystem.
If that looked like a risk in 2005, the rapid pace of new software tools appearing across such a wide variety of distribution channels should absolutely have our attention. The speed of development has not been matched by systems that help us establish trust.
Niche efforts with big results
Modifying your own software, or using modified alternatives, has always been a minority pursuit. Desktop Linux, one notable home of that culture, was less than 1% of the market in 2009 and is around 8.8% today. I do not think AI has turned everyone into a builder, but the 1% or less of people who engage in building has multiplied, and each of them can now produce much larger volumes. Here are a couple of examples.
- A personal tool in one evening. Julie Zhuo, a designer and formerly Facebook’s VP of Product Design, was about to adopt a vendor’s app for managing her AI coding tools. She built her own instead. It worked in half an hour. Her summary: “I never want to use third-party software again.”
- Alternative firmware explosion in five weeks. A $70 pocket synthesizer went on sale this summer. Hobbyists worked out how to replace its built-in software. Five weeks after the first replacement firmware appeared, there were more than 130 custom firmwares. Many are built from existing open-source synthesizers adapted to fit the specific hardware and screen of the device, built rapidly with AI assistance.
For, debatably, the first time, this level of work is not limited to engineers. Anthropic studied about 400,000 sessions of its coding tool and found that “every major occupation succeeds at nearly the same rate as software engineers.” Your builders may be in finance, operations, or marketing.
Second, even these small efforts can have a large impact on a business. In ten days this October, four makers of music hardware, including Behringer, opened their products to outside developers or promised to. A fifth, Elektron, had gone the other way a month earlier and banned the sharing of modifications on its customer forum. A small group of hobbyists changed five companies’ plans in six weeks.
Much of it ships without review
The speed of shipping customized software has outpaced many creators’ appetite for disciplined review.
The website that distributes most of the previously mentioned synthesizer programs is run by one volunteer and says plainly that it “runs no tests or reviews.” Several were released without being tried on a real device. One community member pointed out that shared code “had not been checked for trojans.” Yikes!
The pattern holds at larger scale. A scan of about 5,600 publicly available AI-built apps found that roughly one in three had a serious security flaw. Code written with AI assistance leaks passwords and access keys at twice the usual rate.
Often the builder has not read the code either. The researcher Geoffrey Litt titled a recent essay “Understanding is the new bottleneck.” For a business, that is the risk: people can now build faster than they can understand what they built.
It is already inside your business
It arrives through three groups of people you trust.
Your staff. This is shadow IT, the tools staff adopt or build without asking, with a shorter build time. Every company eventually finds a spreadsheet or a small database that one employee built and the business now depends on. That used to take a month. It now takes an afternoon, and the result can be a browser extension with access to every system the employee is logged in to, or an AI assistant acting on their behalf across all of them. Researchers at the University of Washington tested seven AI-enabled browsers this year and found that the browser’s built-in security protections were “reduced to the strength of the agent’s defenses” against being tricked by a malicious web page.
Your customers. If you run a website or an app, some customers are no longer using it the way you designed it. A technology writer described building a personal extension for Instagram this spring. When the first version failed, her AI tool rebuilt it to talk to Instagram’s internal systems directly. She did not need permission and did not ask. Amazon is suing an AI company whose assistant kept shopping on customers’ behalf after Amazon tried to block it. That company’s position is that its assistant “has exactly the same permissions you have.” Hyper-personalization means your customers may be interacting with your systems through interfaces you did not build.
Your suppliers. Everything above is also happening inside the companies whose software and devices you buy, and inside the free components that software is built from. Your vendor’s product is only as examined as its least examined part.
Attacks through channels old and new
None of this needs to be common to cause a problem. It just takes one issue in the wrong place, at the wrong time, getting the wrong kind of attention.
- A quiet change of ownership. In April, a buyer acquired a set of popular website add-ons through an online marketplace, hid malicious code in about 30 of them, and later switched it on. The businesses running those add-ons had never asked who owned them.
- A patient insider. In 2024, a volunteer who had spent years earning the trust of a widely used open-source project slipped a backdoor into it. One engineer caught it by accident, because a login took half a second too long.
- A convincing disguise. A study of 5,551 AI-themed browser extensions found 341 were malicious. A helpful-looking AI tool is now a common disguise.
- Plain bad luck. Researchers showed that an AI browser assistant asked to summarize a web page would follow instructions hidden in that page. Nobody has to target you for that to go wrong.
Attackers have the same new speed. Last month, two flaws were fixed in PHP, software that runs a large share of the world’s websites. They had sat unnoticed for decades. More than 22 million servers were exposed, attacks began four days before the fix was available, and researchers credited AI with making the attack practical. In 2018, attackers took an average of 63 days to exploit a newly disclosed flaw. In 2025 the average was about a week before disclosure. Verizon’s 2026 breach report found that exploiting unpatched flaws started 31% of breaches.
Decisions ahead of you
In addition to bolstering traditional IT security practices and guidance for team members about the risks of installing or using unknown software, some new decisions are on your plate.
How valuable experiments get approval
You want your most curious and engaged people to experiment. If approving a tooling experiment takes six weeks, people will stop asking. The work may continue, but in the shadows. Give them a path to keep valuable experiments happening, in the open, and keep your builders on your side.
How customers may reach you
Some will arrive through software you did not write. You can welcome that, as Behringer did, or resist it, as Amazon is attempting to do. Your decision will involve weighing the risks, costs, and value that different degrees of openness will create.
How you will respond to unusual activity
If you’re monitoring software and digital infrastructure, I bet you’re seeing some weird stuff right now. How do you decide what is a curiosity vs. a problem? What action will you take to prevent or change unusual access patterns?
This is an interesting time for software. You have some agency in how you will respond in this age of hyper-personalization. Will it be an opportunity, or just a risk?