Recently I have been working on a project where I needed to extract data from vendor invoices and automatically create bills in the Accounting tool. Previously I had tried this task using some OCR (optical character recognition) tools. But it was not working out well; there were a lot of places where the task needed a cognitive ability, and it was not an easy problem to automate at that time.

Neither machine learning models nor a convolutional neural network was an easy solution at that time. Both of these paths needed tremendous time and effort, which I couldn’t afford as a solo developer while managing other day-to-day tasks.

A month ago I again started with a fresh set of eyes. Recently, a lot of dynamics have changed in the software development world. We have AI agents which code or program for us like any junior developer. We have open-source models which could accept image inputs and extract necessary information from them.

So the dots are connected now. I have both the tool to improve my productivity and a tool with cognitive ability. I meant to say that these transformer models have some form of cognitive ability which lets us pick up the desired values from the image without much effort. Finally, after working on this task for the last few weeks, a small prototype was ready.

The prototype for the task, which would have taken at least a quarter of the year, is now done in just less than a month. This was possible because of the help from the AI agent. But it’s not the only factor. I am trying to explain that in this blog. Before going any further, let me first explain my final pipeline for this task.

First, render each page of the invoice PDF using some PDF rendering library. This is a deterministic (which doesn’t depend on probabilistic deep neural network-kind models) part.
Extract information from each image using some open-source transformer model. When there are multiple pages, again use the model to merge them into a single JSON text.
Then we need to map this raw JSON text to a particular schema which is required by any bill-processing accounting tools (in my case I am using Zoho Books for this). This needs the cognitive ability of the model to map the data to the desired format. Sometimes fields might be in different names, so the model needs to figure out the mapping.
Resolve IDs (identifiers that belong to Zoho Books like Vendor ID, Account ID, Tax ID, Currency ID, etc.) to send in the bill creation payload. It is done deterministically now. But there is a story behind it which I will explain later in the blog.
Discard unused and unnecessary fields from the payload and send it to the server using the Zoho Books API.

There are a lot of small nuances which are not needed for this blog. But doesn’t this seem simple? But most of the time spent over the last few weeks has been spent on figuring out and refining this pipeline. To achieve this pipeline I have been using the following approach on a day-to-day basis.

Each time I would plan something to make the pipeline better.
Ask the AI agent to create an implementation plan. Review the plan and refine it.
Ask the AI agent to do the implementation, and review and make any necessary changes needed in the implementation by myself or by instructing the agent to do that.
Test it and repeat this cycle until I am satisfied with the end result.

So do you notice anything here? Any pattern? Any software development principle that is being used? It took me some time to notice it — this is not some new way of developing software. We have been advised to do this for a long time. It is called Pair Programming. In a Pair Programming model, two developers sit together at the same workstation; one will actually be doing the coding. This person is called the Driver. Another developer would review the code line by line, decide on which direction we should move next, and determine the bigger picture and end goals. This person is called the Navigator.

This is exactly what I have been doing for the last few months. The AI agent was doing the Driver part and I have been doing the Navigator part. This tremendously improved the productivity and innovation in the workflow. Now I have the privilege to try a lot of different approaches to solve the problem and leave all the small, redundant nuances to the AI agent. It can do that work much faster and more efficiently than I could ever do.

For example, at some stage, I needed the final desired schema for the Bill structure. For this I had to refer to the full Zoho Books API documentation, custom fields, and reporting tags configured in each of the Zoho Books orgs. It would have taken days, if not weeks, to do this on my own manually. In that time I might have totally lost the direction or interest in the problem itself. But now it is done in just a few minutes and I could focus on the actual task of automating and refining this pipeline.

Also, the AI agent is much cheaper than hiring a software developer when you know how to use them properly. That’s the reason I highlighted my workflow previously.

Plan → Refine → Implement → Review → Refine → Test → Repeat

It tremendously reduces the pressure put on AI agents while using their capability to our advantage. But does it mean we don’t need another human’s involvement anymore? Absolutely not. This is another reason I am writing this blog.

The plan which I have shared above is not the one I decided on the first day. The initial plan was just simple and straightforward. I would give the image to the AI model and directly ask it to extract the information in the desired format, together with resolved IDs, to be pushed into Zoho Books.

Then I split this task into multiple sub-problems. I separated all the critical interactions with Zoho Books as deterministic API calls, which would be done from the code rather than automatically triggered by the AI model.

Then I defined each sub-problem as either deterministic or non-deterministic. Deterministic ones will use normal mathematical functions and non-deterministic ones will use the probabilistic AI model. Even after that, for a long time I thought resolving IDs would be better put into the non-deterministic part of the pipeline.

This pipeline was finished, but it was taking a large amount of time to process even a single image. I had been brainstorming with a few AI chatbots to figure out this issue. I tried merging or splitting each step of the pipeline, but it didn’t improve. I thought about running the model with a different inference server, but Mistral-rs, which I was using, is already processing tokens at the best possible speed.

Finally, I thought the prompt for resolving IDs was huge, with more than a thousand lines of text. So I converted it into a static system prompt and enabled the prefix cache for the inference server. But still, it didn’t improve the timings.

But one fine evening I had a conversation with my manager about the current pipeline, and we were both not very satisfied with the time it took to process a single PDF. I was explaining to him that this ID-resolving part of the pipeline was actually taking up the huge part of the processing time.

The main culprit was determining IDs for Vendors and Accounts, which are huge in count. He asked, “Is it necessary to use a non-deterministic approach for this?” Then I thought about it and realised that it was actually not needed. By slightly modifying how we frame this sub-problem, it could be moved from a non-deterministic to a deterministic function. It would reduce the processing time of this step from minutes to milliseconds.

Now the whole pipeline has been reduced from more than five minutes to just under 2 minutes. There are still some parts I am figuring out, but it is reasonably good now for the locally running pipeline with the help of an open-source model.

So how we frame the problem is the key here. I needed not just someone with a bigger picture, but someone who is not already swimming in the problems like myself — someone who could stand on the shore and give me direction; a fresh set of eyes to see the problem from a different perspective.

Then I realised the AI agent and I are actually Pair Programmers, but not to the full extent yet. In Pair Programming, people often switch their Navigator and Driver roles, which gives different perspectives on the same problem, but here it is not fully possible.

Even though the AI agent helps me solve problems efficiently and quickly, we still need to figure out which problem to focus on next and which direction we should move to find a solution. Sometimes the problem itself could be omitted, like in the above case. As a navigator, we need to guide it through by brainstorming either with AI agents or other people.

This had happened to me a lot of times even before this AI era was a thing. When I was stuck somewhere, I always used to have small discussions with my manager, and then I would restart the solution with a fresh perspective.

Now this AI agent has taken that Driver part of the Pair Programming, and I am doing the Navigator part. But it doesn’t mean I don’t need a fresh set of eyes anymore. I would always need someone like my manager or co-worker to explore the unexplored directions of the problem.