The first AFAIK customer owned self-hosting as a service company. You own the hardware, the software and your data without the hassle of needing to configure or maintain any of it. https://skarabox.com/
You might be interested in checking out my project SelfHostBlocks which allows you to declaratively setup quite a few services with declarative LDAP and SSO integration with LLDAP and Authelia. Even if you don’t end up using it, it might inspire you. Also, all integrations are tested with NixOS VM tests using playwright to ensure no breakage.
Cool, I'll definitely take a look! I do have a preference for container-oriented setups and do have an elaborate set of plumbing on kuberenetes at the moment.
That being said, I procrastinated on getting postgres backups working and ended up causing self-inflicted corruption, so it is nice to see you've got that setup and have thought of pretty much everything!
The idea is a contract is defined saying which options exist and what they mean. For backups, you’d get the Unix user doing the backup, what folders to backup and what patterns to exclude. But also what script can be run to create a backup and restore from a backup.
Then you’d get a contract consumer, the application to be backup, which declares what folders to backup either which users.
On the other side you have a contract provider, like Restic or Borgbackup which understand this contract and know thanks to it how to backup the application.
As the user, your role is just to plug-in a contract provider with a consumer. To choose which application backs up which application.
This can be applied to LDAP, SSO, secrets and more!
What about self-hosting as a service? You get a server in your home which you own with your open source software and data in it. And you pay a subscription to have a remote sysadmin take care of maintenance for you and can train you on the software? What happens if you don’t pay anymore is you keep everything. But like a good insurance, you’d keep the subscription because of top notch customer service.
I agree. Keeping your data private is just not a big enough motivation. For me though the big issue is making sure one keeps access to their data forever. It’s so easy these days to use everything from one vendor and then get access shut off with no recourse. That is IMO the biggest fear everyone should have these days.
Yes, the only solution is self-hosting and yes it requires being your own sysadmin and it’s hard and not convenient. That’s why I’m building https://github.com/ibizaman/selfhostblocks. It’s a NixOS collection of modules that sets up services that fit well together and have declarative setup for LDAP and SSO. They have integrated backups, https and other features required for self-hosting. Also, the LDAP and SSO setup is tested with e2e NixOS VM tests that use playwright to make sure users can login if they have access.
I’m hoping to lower the bar to self-hosting significantly.
Hey, I remember seeing this like two years ago when researching Nixarr. Glad to see others focusing on NixOS for self-hosting stuff. I need to look into it again properly when I have time.
Thanks for the feedback! I've added a full screenshot gallery to the README showing the dashboard, client/contact/communication management, forms, user administration, and dark mode.
I was going through the list, nodding as I read all the reasons, agreeing with them all. But also thinking each time I’ll get one anyway. I guess I agree with the last sentence of the post haha. If anything, I’ll justify this by saying I want to choose with my money and encourage this further. But that’s really just to give me good conscience.
Thanks for sharing! Your article illustrates well the benefits of this approach.
One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones. This means that bugs are much more likely, they're more difficult to maintain, etc. You do mention that it's a lot of code, but I wonder if the complexity is worth it in the long run. I suppose that since testing these scenarios any other way would be even more tedious and error-prone, the answer is "yes". But it's something to keep in mind.
> One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones.
I don’t think that’s true, I just think the complexity is more explicit (in code) rather than implicit (in the process of coming up with examples). Example-based testing usually involves defining conditions and properties to be tested, then involves constructing sets of examples to test them and which attempt to anticipated edge cases from the description of the requirements (black box) or from knowledge of how the code is implemented (white box).
Property based testing involves defining the conditiosn and properties, writing code that generates the conditions and for each property writing a bit of code that can refute it by passing if and only if it is true of the subject under test for a particular set of inputs.
With a library like Hypothesis which both has good generators for basic types and good abstractions for combining and adapting generators, the latter seems to be less complex overall, as well as moving the complexity into a form where it is explicit and easy to maintain/adapt, whereas adapting example-based tests to requirements changes involves either throwing out examples and starting over or revalidating and updating examples individually.
> Property based testing involves defining the conditiosn and properties, writing code that generates the conditions and for each property writing a bit of code that can refute it by passing if and only if it is true of the subject under test for a particular set of inputs.
You're downplaying the amount of code required to properly setup a property-based test. In the linked article, the author implemented a state machine to accurately model the SUT. While this is not the most complex of systems, it is far from trivial, and certainly not a "bit of code". In my many years of example-based unit/integration/E2E testing, I've never had to implement something like that. The author admits that the team was reluctant to adopt PBT partly because of the amount of code.
This isn't to say that example-based tests are simple. There can be a lot of setup, mocking, stubbing, and helper code to support the test, but this is usually a smell that something is not right. Whereas with PBT it seems inevitable in some situations.
But then again, I can see how such tests can be invaluable, very difficult and likely more complex to implement otherwise. So, as with many things, it's a tradeoff. I think PBT doesn't replace EBT, nor vice-versa, but they complement eachother.
You’re right it’s always a trade off. One unexpected but very welcomed side effect of having those stateful property tests is we could use them to design high fidelity stubs. I wrote a follow-up blog post about it https://blog.tiserbox.com/posts/2024-07-08-make-good-stubs-w...
My experience is that PBT tests are mostly hard in devising the generators, not in the testing itself.
Since it came up in another thread (yes, it's trivial), a function `add` is no easier or harder to test with examples than with PBT, here are some of the tests as both PBT-style and example-based style:
@given(st.integers())
def test_left_identity_pbt(a):
assert add(a, 0) == a
def test_left_identity():
assert add(10, 0) == 10
@given(st.integers(), st.integers())
def test_commutative(a, b):
assert add(a, b) == add(b, a)
@parametrize("a,b", examples)
def test_commutative():
assert add(a, b) == add(b, a)
They're the same test, but one is more comprehensive than the other. And you can use them together. Supposing you do find an error, you add it to your example-based tests to build out your regression test suite. This is how I try to get people into PBT in the first place, just take your existing example-based tests and build a generator. If they start failing, that means your examples weren't sufficiently comprehensive (not surprising). Because PBT systems like Hypothesis run so many tests, though, you may need to either restrict the number of generated examples for performance reason or breakup complex tests into a set of smaller, but faster running, tests to get the benefit.
Other things become much simpler, or at least simpler to test comprehensively, like stateful and end-to-end tests (assuming you have a way to programmatically control your system). Real-world, I used Hypothesis to drive an application by sending a series of commands/queries and seeing how it behaved. There are so many possible sequences that manually developing a useful set of end-to-end tests is non-trivial. However, with Hypothesis it just generated sequences of interactions for me and found errors in the system. After each command (which may or may not change the application state) it issued queries in the invariant checks and verified the results against the model. Like with example-based testing, these can be turned into hard-coded examples in your regression test suite.
For sure, the hardest part is to create meaningful generators for the problem at hand which can test interesting cases in a finite amount of time. That’s where the combinatory explosion takes place in my experience.
I wanted to highlight one unexpected but very welcomed side effect of having those stateful property tests is we could use them to design high fidelity stubs. I wrote a follow-up blog post about it https://blog.tiserbox.com/posts/2024-07-08-make-good-stubs-w...
> Since it came up in another thread (yes, it's trivial), a function `add` is no easier or harder to test with examples than with PBT
Come on, that example is practically useless for comparing both approaches.
Take a look at the article linked above. The amount of non-trivial code required to setup a PBT should raise an eyebrow, at the very least.
It's quite possible that the value of such a test outweighs the complexity overhead, and that implementing all the test variations with EBT would be infeasible, but choosing one strategy over the other should be a conscious decision made by the team.
So as much as you're painting PBT in a positive light, I don't see it that clearly. I think that PBT covers certain scenarios better than EBT, while EBT can be sufficient for a wide variety of tests, and be simpler overall.
But again, I haven't actually written PBTs myself. I'm just going by the docs and articles mentioned here.
> Come on, that example is practically useless for comparing both approaches.
Come on, I admitted it was trivial. It was a quick example that fit into a comment block. Did you expect a dissertation?
> that implementing all the test variations with EBT would be infeasible
That's kind of the point to my previous comment. PBTs will generate many more examples than you would create by hand. If you have EBTs already, you're one step away from PBTs (the generators, I never said this was trivial just to preempt another annoying "Come on"). And then you'll have more comprehensive testing than you would have had sticking to just your carefully handcrafted examples. This isn't the end of property-based testing, but it's a really good start and the easiest way to bring it into an existing project because you can mostly reuse the existing tests.
Extending this, once you get used to it, to stateful testing (which many PBT libraries support, including Hypothesis) you can generate a lot of very useful end-to-end tests that would be even harder to come up with by hand. And again, if you have any example-based end-to-end tests or integration tests, you need to come up with generators and you can start converting them into property-based tests.
> but choosing one strategy over the other should be a conscious decision made by the team.
Ok. What prompted this? I never said otherwise. It's also not an either/or situation, which you seem to want to make it. As I wrote in that previous comment, you can use both and use the property-based tests to bolster the example-based tests, and convert counterexamples into more example-based tests for your regression suite.
The technology is using my open source project https://github.com/ibizaman/selfhostblocks
reply