Challenges in Custom Software Development

Some challenges in building LLM-powered applications (including RAG systems) for large companies: 1. Hallucinations are very damaging to the brand. It only takes one for people to lose faith in the tool completely. Contrary to popular belief, RAG doesn't fix hallucinations. 2. Chunking a knowledge base is not straightforward. This leads to poor context retrieval, which leads to bad answers from a model powering a RAG system. 3. As information changes, you also need to change your chunks and embeddings. Depending on the complexity of the information, this can become a nightmare. 4. Models are black boxes. We only have access to modify their inputs (prompts), but it's hard to determine cause-effect when troubleshooting (e.g., Why is "Produce concise answers" working better than "Reply in short sentences"?) 5. Prompts are too brittle. Every new version of a model can cause your previous prompts to stop working. Unfortunately, you don't know why or how to fix them (see #4 above.) 6. It is not yet clear how to reliably evaluate production systems. 7. Costs and latency are still significant issues. The best models out there cost a lot of money and are very slow. Cheap and fast models have very limited applicability. 8. There are not enough qualified people to deal with these issues. I cannot highlight this problem enough. You may encounter one or more of these problems in a project at once. Depending on your requirements, some of these issues may be showstoppers (hallucinating direction instructions for a robot) or simple nuances (support agent hallucinating an incorrect product description.) There's still a lot of work to do until these systems mature to a point where they are viable for most use cases.

All levels of government have an oversight process for spending money, whether internal (e.g. business cases) or external (e.g. central IT project review). For big software projects, those are the processes that result in the present high rate of failure. These processes are often a source of the failures. Let's talk about what a good approval process would look like. Here's the underlying problem: big software projects fail most of the time. The more they cost, the greater their odds of failure. I've written extensively about the larger solution: the incremental delivery of software based on user needs uncovered through constant user research. First and foremost, scrutiny must increase with spending. More money, more risk. Encourage breaking up big projects into smaller components. If you make it very easy to spend $1 million, but very hard to spend $10 million, then program staff will decompose $10 million projects into $1 million projects. This is the right behavior to incentivize. Hell, let people think they're getting away with something. Second, the scrutiny has to be on the right stuff. If scrutiny is on nonsense like "are there enough pages of requirements" then you're rearranging the deck chairs on the Titanic. Verify that the underlying need is real. What problem are they trying to solve and how do you know it exists? Then verify is whether the proposed solution will actually solve the problem. How do they know that their planned spending will solve the problem? This is all verified through user research. At a small dollar value, it may be sufficient to simply ask those questions on an application form. But as the dollar value increases, it's correspondingly important to verify those claims, either by requiring that the applicant have conducted user research, or by having your own user research team that can spend a few weeks exploring this. The bigger the dollar value, the more extensive the research should be. Third, it's important to determine that the department or agency can actually oversee this project. That means that there is a named product owner who is set up for success. This also means that the project has a technical lead, a software developer with the experience, knowledge, authority, and time to review the software produced by the vendor to enforce the quality surveillance plan that will be part of the contract. Finally, it's good and right to prescribe the use of a specific solicitation template in order to get approval to spend money. For custom software projects, that's 18F's Agile RFP. This will make it far faster to approve RFPs, because they'll all be 90% identical, and the project will be using a known-good RFP template. These are straightforward steps that can be taken by executive-branch officials with approval authority over spending or procurement processes. They're easy to apply, range from free to inexpensive, and will help to ensure that projects have a fighting chance of succeeding.

Custom software is like adopting a puppy. I say this with love. My team has built custom applications used in thousands of healthcare environments, across providers, payers, life sciences, and med device companies. And every time, a non-technical executive is surprised. Not by the build cost, but by the ongoing upkeep costs. Custom software isn’t “set it and forget it.” It’s more like adopting a puppy. You need to care for it, feed it, train it, take it for walks, and patch it when it inevitably breaks. Neglect it and you'll feel the pain in downtime, compliance risk, or user drop-off. When we’re brought in to evaluate a project, the first question we always ask is 𝘴𝘩𝘰𝘶𝘭𝘥 𝘵𝘩𝘪𝘴 𝘦𝘷𝘦𝘯 𝘣𝘦 𝘤𝘶𝘴𝘵𝘰𝘮? We always start with a build vs. buy analysis. The answer is usually buy UNLESS it’s mission-critical to their workflow, off-the-shelf won’t cut it, and they're truly ready to become a tech-first company. That means more than hiring engineers. It requires committing to long-term product ownership. They'll need product managers, clear KPIs, defined objectives, and a strategy for how the software evolves over time to continue delivering business value. Too often, companies want the benefits of custom without fully grasping the tradeoffs. Custom software comes with control and flexibility but also with long-term responsibility. It only delivers real value when it’s treated as a living, evolving part of your business. Just like a puppy, with consistent care it becomes a trusted companion that fits perfectly into your world. 𝘗.𝘚: 𝘛𝘩𝘪𝘴 𝘴𝘢𝘺𝘪𝘯𝘨 𝘪𝘴 𝘭𝘰𝘷𝘪𝘯𝘨𝘭𝘺 𝘴𝘵𝘰𝘭𝘦𝘯 𝘧𝘳𝘰𝘮 𝘮𝘺 𝘝𝘗 𝘰𝘧 𝘌𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘪𝘯𝘨, Alex ‬Melton, 𝘸𝘩𝘰'𝘴 𝘧𝘢𝘳 𝘮𝘰𝘳𝘦 𝘤𝘭𝘦𝘷𝘦𝘳 𝘢𝘵 𝘢𝘯𝘢𝘭𝘰𝘨𝘪𝘦𝘴 𝘵𝘩𝘢𝘯 𝘐 𝘢𝘮. #healthtech #customsoftware #buildvsbuy #digitaltransformation #productstrategy