Yeah it confuses me when I meet backend developers who can't deploy their app for example.
If you can't:
a) Tell me what libs your app requires & how to install them in a resonable way
b) The privledges required to run said app
c) The security implications
d) How to scale your app, particularly what I need to do to get it to behave over multiple servers
e) be resonably sure your code is working
f) Write some code to automate the deploy of this.
Then you're probably just not that amazing of a developer.
Plus I think there's a big difference in respect a full-stack team / tech leader garners over someone who doesn't really understand the technical challenges of half the team.
Experience with messaging platforms. Keep in mind that Twitters platform is not particularly high volume for messaging systems. It is reasonably high volume for messaging between people. But people tweet rarely enough for Twitters overall volume to be all that special.
And Twitters core message exchange functionality is conceptually extremely simple.
You need to be able to take a message, and put it into the set of targets, T.
Problem is T can be "unreasonably big". E.g. millions of followers. Solution? If something is too big, see if you can divide the problem. Split T into smaller chunks t1..tn, and distribute them. If those chunks are still too large, split them again.
This reduces your problem to figuring out how to insert and remove and balance a tree, and how to process two kinds of message deliveries: Insert into timeline or reflect message to specified set of targets (each of which can be another reflector, or a final target timeline)
We all know of and use a system that does this at massive scale: E-mail w/forwarding and mailing lists. If you really wanted to (though it'd be extremely inefficient) you can do this with off the shelf mail-servers and other e-mail software. Using mail servers as messaging middleware might sound weird (and there are certainly better alternatives today), but it works just fine.
At Twitter scale there's certainly enough savings to be made from figuring various obvious and not so obvious optimizations (off the top of my head: for some of the users with the really extremely follower counts, it may make sense not to push their messages, but to push them to caches and weave them in dynamically) to justify the salaries of a few software developers to cut down on hosting and ops costs, but the basic problem is not hard.
He's actually dead on. After that it's a 'mere matter of programming', so there's plenty of work to be done but that's the exact way how you tackle this particular problem.
The "handwaving" here are simple, well understood problems both on the development side and operational side. As I've mentioned elsewhere: Twitters message volume or messaging patterns are nothing special. And it's a problem that is trivial to subdivide into manageable chunks.
Clojure seems too high level for SICP. Scheme was a perfect fit because of how simple it is and how few language features or data structures it contained. You had to build everything yourself, which was kind of the point.
They pretty much are as without sequencing you can only do the most trivial thing (a single IO action). Most interesting programmes are going to need at least two IO actions.
Getting a good idea is usually harder than getting the tech right. Being able to quickly make prototypes is the most important thing in a start up.
FYI I'm not a "brogrammer". I work at a large established company and one of my main jobs is making systems work at large scale. But that's not what you do when starting a company. If you make it you can sort out those issues later.
If you lay things out well that it has a much better chance of succeeding than if you lay things out poorly. And knowledge helps with that.
If you optimize more than you need to then that is premature optimization. But if it takes 5 seconds to load a page (or more) then any level of success will kill you and that's bad for business. You need to scale just ahead of the needs of your users. Too much and you're going down the premature optimization rabbit hole, too little and you'll end up with angry users and competitors will likely eat you.
All loops encode simple logic? I don't think that's true.
Either way, a very good reason for not using loops is to make code more readable. If I see a loop I have to run it inside my head to figure out what the intent is, and if there's multiple things going on, that can take time. If I see a 'map' I think "right, this is tranforming every element in this list in this way"; if I see a filter, I think "right, this is removing elements in this list that don't satisfy this property", if I see a groupBy, I think "right, this is grouping elements in this list by this property", etc., etc.
Code isn't only more succinct, it's more human friendly.
Expanding a map to a for loop is very simple. Yes it removes a shortcut available when writing code, but when reading code I can consume that simple chunk of logic in one mental bite just as easily as I can a call to map.
Sure ... but if a loop is effectively doing a map, a filter, and a bunch of other operations all at once? It's a lot quicker to figure out what's going on if it's been written with combinators (once you're familiar with them) than if it's the vanilla loop.
If we assume the operations we're talking about take time linear in proportion to the list, you've just gone from a * n time to b * n time, where b > a. Both of these are still O(n) in Big O notation, which drops constants, because constants unless they're very large or n is extremely large, tend to have relatively little effect on the running time of an algorithm.
Choosing to write more verbose, difficult to decipher, difficult to maintain code, under the claim that it will perform better, is not a good thing: "premature optimisation is the root of all evil".
In practice, if this becomes an issue (which you find out through benchmarking once you know there is a perf issue), most modern languages offer an easy way to swap out your data-structure for a lazily evaluated one, which would then perform the operations in one pass. Languages like Haskell or Clojure are lazy to begin with, so do this by default.
My comment was carefully worded in order to denote that it is not true in all cases.
Your Big O analysis is correct. However, in a real-world case the list could be an iterator over a log file on disk. Then you really don't want to use chained combinators, repeatedly returning to the disk and iterating over gigabytes of data on a spinning plate.
And yeah, you could benchmark to figure that out, or you could use the right tool for the job in the first place.
Or you could use a library explicitly designed with these considerations in mind, that offers the same powerful, familiar combinators, with resource safety and speed. e.g.,
I've no doubt an equivalent exists for Clojure, too, although I'm not familiar enough with the language to point you in the right direction.
One of the most amazing things about writing the IO parts of your program using libraries like these is how easy they become to test. You don't have to do any dependency injection nonsense, as your combinators work the same regardless of whether they're actually connected to a network socket, a file, or whether they're just consuming a data structure you've constructed in your program. So writing unit tests is basically the same as testing any pure function - you just feed a fixture in and test that what comes out is what you expect.
I found this really useful when writing a football goal push notifications service for a newspaper I work for. I realised that what the service was essentially doing was consuming an event stream from one service, converting it into a different kind of stream, and then sinking it into our service that actually sent the notification to APNS & Google. The program and its tests ended up being much more succinct than what I would normally write, and the whole thing was fun, and took hardly any time.
This started as a conversation about for loops vs. chained combinators and wound up with specialized libraries for IO. You're not wrong, but that's a lot more work than a for loop to parse a log file efficiently.
By that logic why bother with for? You can implement any control structure with goto, after all.
The value of using map is precisely that it can't do everything a for loop can. It can only do one simple thing, which makes it easy for the reader to understand what it's doing.
> You can implement any control structure with goto, after all.
Because a for loop is the most appropriate construct in the given language. That's different than comparing hypothetical constructs that a language doesn't have.
Go has higher-order functions, and you can write all the maps funcs you want, it just doesn't have the ability to create generic combinators with parametric polymorphism. That's a trade-off I accept with Go to get things done. Like a lot of people, Go for me hits a sweet spot of simplicity, productivity, and performance; simply put the benefits outweigh the drawbacks. Whatever, it's a programming language; a tool.
It's like complaining that a functional language doesn't have an easy way to write mutating procedural code. You're always working within the confines of some language, and unless you're directly working on that language's implementation where you can change things, I'd rather be working with the language than fighting it.
But we're talking about language design. Of course it's harder to use the feature the language doesn't have than the feature the language does have. The whole point is that the language should have that feature, because if it did have that feature then code using that feature would be better than code using the current features.
> It's like complaining that a functional language doesn't have an easy way to write mutating procedural code.
I do complain about such languages. That's why I use Scala.
