One smartass way to answer that for the general case is: you don't. And I mean that in the same sense as "you don't write example based tests using a PBT framework". You supply independent actions through generators and have it apply them "chaotically", not strictly in sensible sequences.
That said, for things like auth, you need to do some fixed steps to get anywhere at all. I haven't added support for this yet, but as I'm envisioning it, you'd either:
a) authenticate the browser using headers or cookies that you inject into the Bombadil browser, or
b) use a "custom action" (essentially a stringifed JS function) that'd perform your particular steps
Or what you could do now: add preconditions to your actions so that they'd necessarily fill out the form before anything else could be done.
Another thing we might need in Bombadil is secrets, but for now I think it's fine to inject test credentials that are plain visible text.
Please let me know if you try it out and if you need any help!
> I think his jolly chaotic good energy, along with how he's uncorruptible and sort of outside the normal world, fits a fuzzer in a spiritual sense. But it might all just be a good excuse to use one of my favorite names from my favorite book.
It's also my favorite part of the book, and definitely not something I'd skip. I know some people would object, but I think of it as an homage rather than stealing.
Yes, and in fact, capturing "prior" values with bindings and closing over them in temporal operator thunks is how you talk about some relation between s and s' in a Bombadil formula (not having that particular syntax though). It's a deliberate way of embedding this LTL flavor in JS/TS in the most natural and ergonomic way I could think of. I didn't want a deep EDSL or even a new bespoke spec language that people (and LLMs) would have to learn, and to have to write tools for. Now you can write Bombadil specs with a good LSP and be able to import packages off of npm or whathaveyou. Most web devs will probably be comfy with JS or TS, so that's why I chose that style.
Yes, I understand why you made these design decisions. And I also agree that sticking to JS/TS keeps things simple (for humans, and LLMs). I generally default to the s and s' way of specifying things (in a C# property-based testing framework I'm working on) but looking at how you approached things here gives me another angle to think about.
Hey, yeah, so actions are indeed independent and you can't express fixed sequences of them (e.g. fill out thus form by doing X, then Y, then Z). If you want to enforce certain sequentiality you'd need to precondition your generators. This is admittedly limited right now. I do want to add QoL things like `filter` on the generators to make that more ergonomic.
Also, as you note, you can't implement custom actions. And that's just something that I haven't gotten to yet. It'd be quite straightforward to add a plain function (toStringed) as one of the variants of the Action type and send that to the browser for eval. (Btw, we're taking contributions!)
Weighting is also important right now. There's no smart exploration in Bombadil yet, only blind random, so manual weighting becomes pretty crucial to have more effective tests (i.e. unlikely to run in circles). I'd like to both make the Bombadil "fuzzer" better at this, but eventually you might want to run Bombadil inside Antithesis instead to get a much better exploration, even in a single campaign.
The Until operator is also one of those things that I haven't gotten to yet. I actually didn't expect someone to hit this as a major limitation of the tool so quickly, which is why I've focused on other things. Surprising!
To add some more context, Bombadil is an OSS project with one developer (but we're hiring!) and it's 4 months old and marked experimental. I'm sorry you were disappointed by its current state, but it's very early days, so expect a lot of these things to improve. And your feedback will be taken into account. Thanks!
Not yet but I definitely will record one soon. I've been working at the debugging UI for Bombadil (some hints at https://x.com/owickstrom/status/2034925537706020978) that will make for a nice cohesive demo.
"Bombadil is property-based testing for web UIs, autonomously exploring and validating correctness properties, finding harder bugs earlier."
I can't make heads or tails of this. What does it do? "Bombadil checks each property as it explores your system in its chaotic ways, reporting back any violations." what properties? checks how? does it read application state? "recording violations" what violations? "select an action and performs it" what actions?
This needs some code example, or a video, or something to explain what this tool is
The page "Language features > Properties" starts to explain what this is.
Go for it! I've been meaning to do an "architecture of Bombadil" blog post that'd likely answer this question. It's not super advanced by any means, but it's a mindset shift to how you might think about browser testing coming from the mainstream frameworks like Playwright.
I'd say it does! Bombadil is very new but its predecessor has found complicated and very real bugs in my work projects, and in the paper we wrote about it, we found bugs in more than half of the TodoMVC example apps: https://arxiv.org/abs/2203.11532
