Great article. I've been trying to achieve something similar with a Revit. It's an old CAD application for Windows which means there's a few additional hurdles in exposing a cli interface that allows the LLM to drive it. However, once that is done, the loop of "write code, take a screenshot, repeat" works pretty well.
I'm developing a Revit add-in that and using the LLM to help write the code. I use AI in the development process but no AI is used at runtime.
The Revit API is a challenge to build against. The documentation isn't great and often API calls will fail for mysterious reasons.
The old school, non-AI workflow for developing an add-in for me was: read the docs, write some code, run Revit which loads your add-in, do some actions within Revit to test your code, figure out what failed and repeat until you get the desired result. There was a lot trial and error involved and the loop was slow. Revit takes long time to start even on high-spec machines and the user actions required to test take some time.
I'm trying to use LLMs to speed up this development process by allowing it to run this loop. Things that are required to enable the loop without human intervention:
- a cli tool that allows Revit to be started, running a single addin. There's a couple of ways to do this: using a master add-in that dynamically loads child add-ins or manipulating the .addin files.
- a way to capture logs from within the running Revit addin and feed this back to the LLM. This is just logs in a standard location.
- a way to capture screenshots from within the running Revit addin. Revit has a image export functionality that can be triggered from within the code.
The loop is still slow but once setup it allows you to prompt, set it to run and come back later with some hope that it has achieved your goals.
Modelling is part of what I'm doing but it's also trying to get figure out how to work with extrusions, dimensions, views, etc, etc.
I'm going to try to get it to generate extrusions in Revit based on images of floor plans. I've tried doing this in bunch of models without success so far.
You might want to give it some guidance based on edge centers? It'll have a hard time thinking of wall thickness and have it draw points if you're trying to copy floor plans.
for clarity now that I'm rereading: it understands vectors a lot better than areas. Encoding it like that seems to work better for me.
Note that the tilt only has one level (flat and tilted), the animation might make you think otherwise. I use it in tilted mode.
It is highly customizable using the Logitech options software which I am using on both macOS and Windows. For example you can assign actions to all the buttons that are dependent on the running program, and even involve custom key combinations.
The only downside of the trackball is that you have to clean it once in a while, when the pointer appears to be moving very slowly - simply eject the ball by poking a screwdriver into the hole at the bottom of the mouse, clean out the accumulated dust and push the ball back.
If you like le Carre, British TV Series "The Sandbaggers" is in a similar vein. I've watched a bunch of episodes on youtube and they are great with a focus on dialog and schemes as opposed to action.
Some of my best ideas that ended up being company changing started as a conversation walking to lunch with colleagues.
Initially these ideas were just undeveloped thoughts and I would never dream about booking a meeting to present them. Having a chance to develop them in a casual conversation might have been the difference between successfully building the thing and not doing it at all.
I love working from home but miss the unstructured collaboration.
> Initially these ideas were just undeveloped thoughts and I would never dream about booking a meeting to present them.
Your company doesn't have informal communications channels for this sort of thing? Team/Slack/whatever? In every place I've worked for the last 15-20 years (it was different before that), such informal "watercooler" talks have never happened in person, whether everyone was in the office or not. It was always over electronic communications.
The reason for this is that it's less disruptive to other productive work.
