I write a unit testing framework in every new language I learn. I find it's a great workout because making it usable for yourself is immediately assessable, and it often forces you into deeper areas of the language, including reflection and meta-programming.
You might want to have a look at QuickCheck. Trying to port it's techniques to other languages will also give you some insight (and will result in valuable tools).
And make you miss the ability to overload functions just on return type. Same thing happens if you use Haskell's regex library or try t port monads to another language.
Indeed. Overloading on return type is one of the few things that you can't do in dynamic languages by design (and in most statically typed ones, neither, but that's an accident).
Please do! It's very interesting; I'm not good enough in PLT to try and find out the answer for myself (in such a short amount of free time I have lately) but I'm very curious if it's true that you cannot have overloading on return type in unityped languages by principle.
This is not hard to understand: under usual semantics for function calls it is clearly impossible to do return type overloading in a dynamically typed language. But, as I pointed out above, it is perfectly practical to cheat and provide an indication of the desired return type as an extra hidden argument to every function call (this is what Perl does with wantarray: it is a "builtin function" but the effect is really that of an extra boolean parameter to every function call that specifies whether the result should be an array or not).
Ok, this much I understand, but I wouldn't call this overloading on return type... It's - clearly - overloading on input type, either implicit (wantarray - I checked the docs, it seems to be a variable set automaticaly depending on context by the interpreter) or explicit, but input type nevertheless.
So, the short answer is - as I thought previously - that no, you can't have implicit dispatch on return type in dynamically typed language.
Contracts are probably getting close, but you still need a version of apply that will get expected return type and a bunch of functions to introspect (or a macro that would do the same, but let's not wander there, as lispy macro systems are equivalent with static type systems anyway).
And you can generalize from wantarray's Boolean to a more general parameter indicating required return type.
It gets harder and harder though, to provide information about more and more distant parts of the program in a dynamically typed (or untyped) environment.
From what I've heard, Racket's contract system is another interesting attempt to provide something that's typically done in a static setting---the benefits of dependent typing---in a dynamic one.
For those who want a very clear, concise set of userland code to try translating for practice, consider using 9base: http://tools.suckless.org/9base
It's not quite Unix, but it's still quite lovely and it's rather succinct:
"It also contains the Plan 9 libc, libbio, libregexp, libfmt and libutf. The overall SLOC is about 66kSLOC, so this userland + all libs is much smaller than, e.g. bash (duh!)."
This works well as a learning method, most probably because UNIX tools are well documented, and there are readily available binaries. Many introductory programming courses I took and taught back in the day were using UNIX tools as programming assignments.
I believe, any well documented coding problem accompanied with a sample binary implementation should have the effective educational value.
Sites like interviewstreet.com should take this as a guide, even though their audience is not beginners. If you look at sample problems there, they are usually poorly described, and sample outputs are trivial and don't contribute to the textual description.
My favourite way to learn new languages & platforms was implementing a du-like ('disk usage') tool. It was also the exercise I proposed to my students. It doesn't require complex algorithms but touches a lot of the basics: recursion, filesystems, command line parsing, output formatting, etc.
I am taking Operating Systems at Auburn University, and my professor had us work on the Linux kernel source, learn how to define and implement our own system calls using the SYSCALL_DEFINE[0-6] macro and then implement our own memory snapshot tool. It is similar to /proc/<pid>/statm. We had to get other info on major and minor page faults as well. I learn a lot doing things like this.
Rodrigo, I thought you should know that when browsing your site on an iPhone 4, the header overlapped the index. That wasn't a great description, so here is a screenshot: http://db.tt/7grTPHsL
You also cannot zoom in, which made it very hard to read.
It is a good approach and back say 20 years ago the favorite was to redo the du command and add a conversion to MB (now handled by the -h option for human in the du and ls command).
Cat is also a good example and not just to learn a programming language but the OS. Ask somebody to come up with 20 ways to display a file, you can do it with not just the cat command. Now I'm not saying there are twenty ways, but it is one area which some delving and approach will allow people to try and find them.
Examples are for me the best way to learn a programming language and again with the simple Unix commands you have a common base which people are more likely to ifnd an example. Can't recall but great site on the net which has "Hello World" in about every programming language around.
After a while, you will start to add option to your redeveloped commands, then add entirely new commands and from there you can think about writting your own shell. No graphics or the like to distract you too much. Graphics as a rule in programming languages I have found to be like learning an after thought and also you are more controlled in the mentality behind the API. So I do advise not even looking at graphics for a while until you have learned how to flex the language without the distractions graphics and the way they are handled add a overhead to your code. I'm sure somebody will list a programming language that is easier to do graphicly as apposed to text output, though its a safe rule. SO just because you have windows, don't mean you have to jump into GUI's from day one, if even at all.
I remember that hello world site, I think it fell off the internet but there are still many of those types of sites. One of my favs was the 99 bottles song in many languages. There are also larger projects like PLEAC and Rosetta code that are similar.
Earlier this year, I did 21 different implementations of a hex dump program in C (starting here: http://boston.conman.org/2012/01/09.1). I also included implementations in a few other languages just for comparison. Why? It was a neat break from the programming I do for work.
Shouldn't this be doable with simple library wrapping console framebuffer, that you Linux guys have and I suffer lack of on FreeBSD? (I can be very wrong - I'm just a bit jealous, not jealous enough to dive into fb implementation :))
When learning a new language, my first project is usually an IRC bot. It gives me a good feel of the language and I need to learn most core structures.
I learned both perl and python by using them to replace my sysinit scripts on a Centos box. If you can get through this, I'd say you're "conversational" in a language.
You'll also know Linux better than most ever will.
I feel like this approach might get boring after a couple of languages.
I think most people here could skim through the language intro, and then just start working on a new project in that language (picking up more of the language as they go).
This is the approach I usually use, and it seems more fun than porting the same tools. Just my opinion though :)
For a purely functional language like Haskell, this would not be a very good advice. Any kind of I/O would involve monads and other imperative constructs. Better implement an algorithm involving trees or graphs to better appreciate a functional language.
Not at all. You can start doing I/O in Haskell without knowing anything at all about monads by just treating the do-syntax as an imperative DSL.
In fact, Bryan O'Sullivan (who wrote Real World Haskell) just held a tutorial session on Haskell a couple of weeks ago where people completely new to the language implemented simple Unix tools like "wc". I don't think monads were mentioned at all.
This is a slightly crude implementation of cat in Haskell. As you can see, it is replete with scary monad plumbing that gets in the way of the actual functionality (copying stdin to stdout).
I would explain further and implement more than what the blog post suggested but there's already a book (Real World Haskell) that does this with a number of other Unix commands.
On the contrary, any interesting program is going to have to do some I/O. So if you're want to do useful things with a language, I think it's a good thing to learn the basics of opening, reading and writing files.
Also, you should take a look at the top comment in this thread.
Your ruby version of cat implements none of the command line switches.
I learned C by going through the FreeBSD code and helping with POSIX compliance. For fun I would implement a lot of the commands in Python. You get the most out of learning both the language and UNIX by implementing all the command line options.
"Your ruby version of cat implements none of the command line switches."
I don't think he was trying to show off what a feature complete cat command he wrote. Just trying to illustrate his idea. No need to be rude about it.
And you get a better grounding in the Unix philosophy if you omit the command line switches and make small utilities to handle those cases, because as the paper said, cat -v is harmful (http://harmful.cat-v.org/cat-v/unix_prog_design.pdf).
The Unix philosophy is somewhat of a myth. http://cm.bell-labs.com/cm/cs/who/dmr/man12.pdf shows that that same first edition already had a -t option on ls. Surely, that should have been some | sort, but AFAICT, they did not bother to write that until some later time.
Sticking to a philosophy like a zealot makes it a religion. Keeping it as a philosophy means it encourages a certain approach, not that it makes it an absolute unbreakable rule, plus it would only results in bikeshed/flamewar arguments.
as PEP20 says:
Special cases aren't special enough to break the rules.
Although practicality beats purity.
I said "somewhat" for a reason, but I think there are just too many special cases to keep calling them exceptions. I use awk and sed, pipe things into bzip2, but I more often use tar with a z or j option to compress stuff an in a pipeline. We also do not have separate tools for parsing python/ruby/perl and executing it. If code reuse using pipes were successful, wouldn't we have a separate 'exec' tool that is backend to all scripting languages? Looking at gcc with its zillions of options, pipes sometimes look more like an implementation detail than as the primary way to compose tools.
Also, other kinds of reuse of the things mentioned as examples of the Unix philosophy are rare. What languages on your system use lex and yacc? 'but thise tools are outdated' is not a good revuttal; if they are, how has that come about? I think the Unix philosophy is perfect for prototyping, but 'real' stuff tends to need polish that 'the unix philosophy' cannot provide. As a final example, consider git. As I understand , it started life as a set of scripts, but eventually became a C program.
I generally use the -j option to tar primarily because it's less typing. I think the ideal option is for these flags to be provided, but implemented using pipes under the hood so tar -cj folder is just syntactic sugar for tar -c | bzip2. Similarly, tar -cf folder.tar folder/ could be implemented as tar -c folder/ > folder.tar. I think this way you get the conciseness of the flags approach, but also each item of functionality is only implemented once. Just a random thought.
Great advice. I've written a simple version of the `column` tool as an entry to the International Obfuscated Ruby Code Contest: https://github.com/saizai/iorcc/pull/1
Just a bit of fun, the contest doesn't seem to be very 'official'.
A few years back I remember submitting the implementation of a couple of Unix tools in Perl. The idea was to implement as many of the standard tools as possible, in perl.
The project seems to be sleeping, although there are many contributions:
The advantage and disadvantage of this approach: you have to already know Unix commands. Inevitably all programmers learn Unix commands - but probably a rough approach for beginners. Although, I'm guessing this advice isn't really aimed at beginners anyway.
The example code for cat is only half of cat, which reminds me of Rob Pike's criticism of "cat -v"---Unix programs are often more complicated than they arguably should be, are "reimplementing" Unix programs often means implementing a small subset of the features.
Or, look up the manpage for your locally installed "tree" and see how many non-standard options and features have been bolted on.
I think it would be helpful when reimplementing Unix to try to work in more core features of the language, like its object or module system, which are more important than parsing argv or touching the filesystem.
Even if you ignore all of the bolted on options, his simple implementation of cat doesn't even handle "cat -" so I would say less than half. I think this is still a good exercise if you don't dig deep into the plumbing of all the tools you're reimplementing, but it's even better if you do.
Unix, as an environment, uses a handful of primitive paradigms. You will not learn much if you're using an advanced language.
If you were to learn Oz[1] in this manner, none of the its powerful features will be called for to implement simple filter programs, transforming text, and crashing when confused by unexpected input.
For languages that support advanced features, you're better off modeling "advanced" systems. Say, in the case of Oz, you might be better off modeling a secure microkernel or a VM; not dumb text processors.
Another good thing about learning UNIX (or using Linux) is the ability to write code snippets that could automate daily chores (you can do in Windows too, but I am not going to speak about it here).
Some of the code I enjoyed writing and using is, code to remember directory paths I visited (I visit lot of them, and it is a pain to type lengthy ones). A wrapper for "ssh" to remember hosts and list them (again I had to visit bunch of them daily, and pain to type fully qualified host names), just like PuTTy-saved sessions.
These tools really boost the productivity and joy to use.
I agree with you, having the ability to script everyday tasks is awesome, though however in the case of the ssh wrapper it might be a little overkill, given that you can achieve the same (and more) with a ssh_config file [1] [2]. I would particularly recommend the ControlMaster option.
One of the simplifying assumptions that is built into this idea is that the input and output to the program are simple streams ( a file, a terminal ). Not surprisingly that makes things a lot less complex so you can focus on the programming.
Since a Raspberry Pi is only $35 or $50 with enough stuff to program it, that is one way to get started. Of course taking an older tossed of PC and installing Linux on it works too (which can often be done for free if there are businesses around)
You can download complete Linux VM images that just work from the net. VirtualBox is free and is installed in 5 minutes. It's hardly frustrating. Certainly less so than getting a RPi up and running, not to mention you end up with an odd, underpowered device secondary to your regular machine.
That's not to say the RPi isn't an interesting, rewarding experience on its own merits -- that's why I got one --, but I wouldn't recommend it to anybody interested primarily in getting a *nix environment for experimentation or programming.
As far as the Windows and Mac comment goes, there is no reason why you couldn't do this on either. In fact Mac OS X supports the same commands as linux. In Windows case, I see no reason why you couldn't do one of two things:
1. The first would be implementing your own linux commands so they work on Windows. You would probably want something like cygwin, a linux vm, or an actual linux box to test this stuff on though.
2. Pick Windows commands instead and implement them instead. It’s the same idea roughly. I haven't used a Windows machine in almost a decade, so I'm not sure what all would be involved, but seems possible that you could get them to work roughly similar.
As far as specific languages go, there’s no reason that almost any language couldn’t be used as long as it can take arguments from the command line.
The point is to copy the commands, you can copy them on any operating system. It just happens that Mac & Linux have them built in while Windows doesn't.
As for which languages, this actually seems like it would be useful for learning any language that has reasonable ways of interacting with IO streams. Off the top of my head:
I think, almost by definition, for a language to have any interest, it must be able to deal with I/O streams. I can't think of a language that wouldn't work (I'd be very interested if somebody knows one and why it wouldn't work).
What the scripting languages give you is simplicity. No need to muss with compilers.
On Windows you can install Linux in a virtual machine, or you can install Cygwin (which is basically an almost complete POSIX system on Windows).
You can use your preferred programming language to implement any of these tools: Ruby, Python, Perl, Scheme, Common Lisp, C, C++ ... The original Unix implementation of these tools was done in C.
Windows doesn't have many of the basic Unix tools, but you can still implement them using your programming language of choice (how great). No, you're not limited to these three languages. I only meant in the article that these three are probably the easier to copy Unix with, for their scripting nature.
This is a brilliant idea, and may I suggest that one takes it slightly further than just implementing the core functionality, and aiming for a complete clone (optional parameters and all) with a few enhancements.
Indeed, for a further ego boost, why not also benchmark the performance of your versions against the performance of the native utilities? You never know, your new clone might end up being the new 'less' to the old 'more'!
Yes. I think I nice approach would be just cloning core functionality of the program initially in your language of choice, then dealing with parameters and more advanced options as you learn that language better.
This is kind of funny since I taught myself a lot of go by porting a bunch of the coreutils. It still amazes me how much there is to learn from unix. Many many times I find myself going back to some programs src (cron, find, etc.) when I need the design/algorithm/feature in my own programs.
I agree with this and have done it in the past myself. Its a great learning exercise for new languages. The problems are easily defined and its apparent when you've successfully completed. Once you've done with the easier ones try implementing grep and then diff.
He's right. I have versions of "cat" in Java, Perl, Python, and C, all source. Reading from stdin and writing to stdout is the skeleton of a lot of tools, so starting from "cat" source is often the best way to get going on a new tool.
I think this advice only really makes sense if you understand and have an appreciation for UNIX tools. Otherwise, I think it would be rather burdensome to come to understand how seemingly abstruse UNIX tools function.
Congrats for being 18 and making it to the top of hacker news. You have proven that simple and concise advice can be greater in value than the most complex systems (or technically advanced articles that frequent HN).
tl;dr - this does work to a point, but won't necessarily teach you idiomatic and community practices that come with experience, but it is surprisingly sticky
I had the great pleasure, year ago in my undergrad Operating Systems class, for the class assignment to be "write an OS in Java"...which of course was handed out to a group of students who had never seen Java. By the end of the semester we had written the core guts of a multi-tasking OS, a couple shells and the display systems to handle even displaying things like a unix-like console, a sane piping system, all the major user land utilities (sans some of the compiler things, but things like ls, cat, ps, etc.) a simple text editor and intra-system messaging system, etc. etc. etc.
It was a great curriculum and really was the first time we, as CS students, had the chance to really spend time understanding the subject matter without spending time focusing on stupid language tricks like we had in our various C and C++. The code we wrote was fairly straight forward (we were learning the language as we went, so kept to the KISS method) and focused instead of the material. It was probably among the hardest, and best class I've ever had on any subject.
Did I know Java at the end of it?
To a point -- I knew the pidgin dialect we wrote the OS in. A few semesters later I took a fluff software engineering course and had to hack out some various java server bits and had a roughshod time of it as I ran head first into the now common overengineeringitis that plagues modern Java development. I found the syntax and most of the standard library familiar, but the idiomatic ways of writing the code, community practices, the shibboleths, nearly impenetrable without years buried in an enterprise software house.
I swore off Java and never looked back...moving on to Perl and Python for a spell (incidentally my standard "learn a new language" project is to write a simple non-lexical phrase extractor, it touches I/O, data structures, database connectivity, program flow, and if I get daring, multi-threading and a few other odds and ends and usually gives me a pretty good idea how a language works.
Now years later, taking a look at Android dev, I'm finding that writing code for the platform, even though it's Java, to be like writing code for our old OS. It's pretty simple, there's great library support, and I don't have to wrap simple method calls in hundreds of lines of framework boilerplate nonsense. It's actually pretty fun.
But I've definitely been drawing heavily on that pidgin dialect of Java that I learned way back when -- it's kinda like riding a bicycle, except a few bits have changed here and there. So yeah, I think I did "learn" the language, and it's been amazing how much of it I can recall since it's been a decade since I did any coding in it.
(this method also handily solves the "I need a project, a goal, to learn the language, otherwise I'm just twiddling bits" problem).
It blows my mind how often this happens to me, even though I understand how and why. Especially since it's usually just a few minutes after the original comment is written. Google is awesome.
It's a fun way to find sequential tokens (words) that have a high probability of being the names of people.
- Take a lexicon (list) of single words in a given language (English for example).
- Take a block of text and think of it as an ordered sequence of tokens, news articles work really well for this approach, books not as much.
Example (from http://www.cnn.com/2012/11/02/showbiz/movies/flight-review-c...): With its spectacular plane crash -- I would rate it fractionally behind the air disasters director Robert Zemeckis staged in "Cast Away" and Joe Carnahan in "The Grey," but still more than gut-wrenching enough to make you think about taking the train, next time -- "Flight" immediately raises the stakes on your typical addiction drama. But that's essentially what it is -- with a courtroom finish for extra lift.
- Stream through the text, any word that is in your lexicon, throw away.
- treat each group of remaining tokens as separate objects, in this case we have 2
- write these non-lexical (not in your original lexicon) sequences out:
Robert Zemeckis
Joe Carnahan
Boom, you just made an entity extractor that plucks names out of text without having to model the English Language too rigorously. And it generally works in most languages that have a low intersection between name-part tokens and lexicon tokens. And it can be brutally fast.
Where this gets interesting is in suppressing junk and tweaking the algorithm around things like parenthesis, apostrophese, and sentence boundaries. There's lots of little edge cases like this that you have to be mindful of - numbers in the text for example are never parts of names but aren't in your lexicon so you have to figure out what to do with those. And then you can use other heuristics to improve the results, suppose another sentence just had "Zemeckis" in it, that's a name, but then suppose another sentence had a token not in your lexicon like "Samoflange"...do you count that as a name? What about lexical tokens that are names like "Bush"? So you can try things like only counting sequences of tokens that have more than 2 tokens (like "Robert Zemeckis") and ignoring ones that have only 1.
And it goes on and on -- endless tweaks to improve the quality of the names you get and suppress non-name sequences.
To make the project more interesting, try storing your lexicon in a database or some kind of index so you can search it quickly, I like to use SQLite files with indexes on the lexicon table myself, but it's a fun assignment to try different things like in-memory TRIEs.
If you want to try threading, you can try playing around with searching the text at different start and end points (thread 1 searches the first 25% of the text, thread 2 the second 25%, etc.) or have different threads search different articles.
You can try all kinds of different things to keep it interesting and as you start abstracting the problem you can play with all kinds of different control and data structures to accomplish the task. Trying to make this as fast as possible (with all the heuristics turned on) can also be a fun challenge.
My standard "learn a new language" project usually is a calculator... touches many aspects but I have grown bored of it. A phrase extrator is simple but can be much more complex, might try it next language!
I always implement "cat" as a first toy application when learning a new langauge - it's a great introduction tool as it deals with dealing with files, dealing with streams (stdin) and so on.
Heh. I've been doing something similar. In my free time, I've been actually re-implementing the `tree` command (with most of the switches as well) in Python.
How many people learning python for the first time have a good knowledge of unix? This is like saying you should become a stronger first time weightlifter by juggling barbells.
My first usable program in python was an implementation of 'wget' and seriously, implementation of such unix based commands do teach you an awesome lot.
I disagree. I think writing code to solve whatever small problem you're having right now is better. No point in recreating the wheel ,which is boring, and will result in something you'll never actually use because, well, cat already exists on your system.
But if you use it to start something new, or to add to your repretoire of utilities, then that is something you're more motivated to complete and more useful expenditure of time to boot.
