I agree with his explanation, I disagree with his conclusion, and here I am treading on thin ice. I don't really take that many photographs, not anymore anyways. Besides the only reason I know anything about photography is because of my dad, who's a better photographer than I ever will be. I however like cameras, old, heavy ones which have that loud shutter releases.

Old cameras with satisfying sounds do not produce pictures which are easily digitized, and hence do not support Tim's idea of Lightroom composition. But this is not my main reason for disagreeing with Tim. Digital cameras are great, it costs little or nothing to shoot a picture, and shooting a bad one is almost free, and free is a great thing when you are either experimenting or learning.

That said, 15 years ago, one essentially had a copy of nearly every photograph one took, and you ended up looking at both the good and the bad, because deleting a photo was too expensive. I once felt that being able to get rid of photos as easily as one now can somehow took away from what was recorded, its somehow bad that one can choose what one saves. I have gotten over that feeling (film is surprisingly expensive when you are a student and your parents aren't paying for it :)). Composing post fact seems worse somehow. My dad and I used to take photographs in the jungles in India. The animals aren't usually tame, and they don't always tend to show up at photogenic locations with perfect lighting. Worse still, you don't always have time to compose, and as a result when you find something interesting enough you shoot as many photos as possible hoping that some of them are composed reasonably well.

In cases where composing photos is easier (flowers for instance, since that is what Tim talks of), it seems like there is artistic value to actually composing things, besides maybe you end up with more relevant pixels once you are done. I don't know, somehow the idea of lightroom composing rubs me the wrong way. But then again I don't really know that much about this.

Before I begin bitching, there are a couple of things that need to be put out. Yes I am aware of the existence such things as s11n, and the Boost Serialization library. I have even spent a few days looking at such things as Thrift, and the promise of writing something like an IDL, passing it through a compiler, producing C++ code which then serializes itself. Really this is a great idea, and if I was worried about writing serialization code for myself I'd have picked one of those. Overall, I would probably have gone with s11n for things I was adapting to be serializable, and Thrift for everything else. But then again, I could also use Python, Java, C# or any of the dozen languages which provide nice serialization interfaces.

The thing is I am not writing code for myself, well I am, but not really. I am trying to see if this class I am TAing can be made doable in C++. I don't like all of those solutions for various reasons, listed forthwith.

1. s11n, boost: I think any serialization depending on me completely specifying what needs to be serialized in code has issues. In short, there are times when data fields are added to classes, and people forget to change code. This is somewhat panful to debug (think of forgetting to initialize stuff in a constructor and such), adding another place where you need to worry about this seems like a poor decission.
2. Thrift, ...: Learning another language, merely for the privilege of serialization seems a little counterproductive. Besides why pick Thrift's IDL over any of the other serializable language. After all managed code is supposed to become as fast as native code, give or take a little

No sir, I do not like either of those choices, and for once in my life, I find myself wanting something lots of languages have, in C++, because pig headedly this is the Java alternate some of us backed.

s11n's author aptly states that serialization is easy, it is deserialization that really sucks. Assuming we are not going down blind alleys (as I once was), packing data in a known format, with know boundaries, is a fairly reasonable strategy of assuring oneself that the data seen on the receiving end did indeed originate from a compatible library. You ask what of data which has boundaries and properties identical to what you're expecting, but is actually corrupt. Well no matter how much you worry about solving problems involving arbitrary data, one must always remember that provably showing equivalence between two generators of data is unsolvable.

Now the only real reason deserialization is so hard, is because in some cases you want people to be able to encapsulate arbitrary data in objects belonging to your class. Arbitrary data is somewhat hard to dynamically recreate. And by somewhat I mean a lot.

Let us assume then that we neither want to learn an IDL, nor do we want to face the problems I feel exist with using libraries like s11n. However, we also recognize the fact that s11n or boost is probably much better done than anything we can produce, and perfect in a reasonable amount of time (a few hours in particular). Now if we had reflection, or what some fancy sites seem to call meta-object programming, we could go through the process of producing code which automatically serializes all data fields in an object. Well C++ doesn't really have reflection, and C++'s version of RTTI is not very useful for these purposes.

But we are stubborn, pig headedness always comes with stubbornness. Well if we had multi-pass compilers, we could use C++'s macro system (which is arguably not very powerful), to build ourselves a reflection framework, and then use said reflection framework in the way mentioned above. After all, we know that C/C++ macros are mere text replacement tools, and hence it makes perfect sense to write #defines which produce other #defines. Of course the C++ compiler is single pass, probably for many many reasons, including the avoidance of infinite loops. Ah the avoidance of infinite loops, always a worthy cause.

Well I am lost. I have ideas, none of them seem to work. Writing serialization code by hand seems to be very very painful, and is one of those things I know is going to come back to haunt me later this semester. We need better alternates to Java. And other things...

Panda

Since it is clear that I write something only when classics essays are bothering me, this entry is going to be updates about life. Clearly this needs to be done, and who knows, writing scathingly harsh satire might actually be fun.

Updates then, I will be in Delhi from the 22nd of December through the 19th of January. Hopefully some of this time will be spent doing work for 138, the class I am TAing next semester.

I will be at Microsoft starting in August next year. There was this one group at Microsoft working on stuff that was too interesting to pass on. Can't say too much more than that.

Panda

But now I am back, and I need to write three essays, revise one, and make sure one makes sense. By tomorrow. While also working on type inference. And writing code for research. It is interesting how all seems to go well until I hit one of these points, and titter between giving up and just handing in some junk. And junk is truly how I'd describe some of these essays. Except that they seem to do well when handed in, my last one supposedly analyzed the topic well.

I managed to read most of a Murakami on my flight to San Francisco. It was a very nice book, well written and all, but put me in a funk. This seems to be the general effect of reading Murakami on planes for me. Which is strange, though being in a funk has some good effects. I got more work done on my second flight, than I have in the last couple of days. I also discovered this pen I had lying around which writes really well. I have not used many of my nicer pens for a while, and increasingly hole ridden pockets are making pens hard to carry around.

We had an argument about the ANSI C standard earlier this week. It was not an argument, more of a question, who's entire basis was knowing some subtle details in the standard. I don't even know the standard. What is amusing is that this came a day after a discussion about what time was. I got to meet a couple of friends I hadn't seen in a while, it was nice, we went to this Indian restaurant in the bay area I like a lot.

I have been working on these Classics essays for most of the day. Some of them have even been interesting. But I need to do other things, and this is pretty tiring. A lot of this is just about expressing my opinions on issues I don't have clearly defined opinions on. Or on things I don't wish to express an opinion about. I want to not give up on this class though. This is all very strange, I think there's a time when I wouldn't have done this. Someone, somewhere in the past, had a thing about how education is supposed to change us, since not changing is not learning. Or some such weird new age philosophy. It was a company, so one can be fairly sure it was printed in multiple colors on card stock. So strange, and all.

In general, like almost any of the past OS X iterations, it seems to have changed the UI in not-so subtle ways. I could complain about things I find weird, but three months from now, the UI changes are going to be unimportant. Mail is better behaved with IMAP, which is sort of nice. I don't really care that much about the newly added RSS support in Mail, or the changes in iChat. Spaces is cool, I haven't gotten used to it, and am hence using exactly one Space, though it had potential. I don't think I am going to be using the coverflow Finder look much, it does strange things with multiple selections. Stacks are cool, but having the choice of having stacks and pre-Leopard style folders on the Dock would have been better. As would have been the, promised, creating a stack from an arbitrary collection of items feature. I like the new spotlight indexing feature, it is finally usable for what I do, but really, Quicksilver did this pretty well. And besides, I am now stuck without Quicksilver.

All this might make one wonder why I installed Leopard. It wasn't easy (trust me on this, my MacBook Pro had issues with the install). I formatted my hard disk, after backing up most of my data onto the external, the format did wonders for my space problem, and got rid of most of the accumulated junk. I am actually excited about the prospect of using DTrace, even though too much work, and a weirdly busy travel schedule makes this a bad time to learn D. And at some level, not having Leopard would have bothered me, since I'd want to do all the cool things everyone else was talking about, and in some ways OS X upgrades are strange, you're damned if you do upgrade, and damned if you don't. And well there's Quick Look, which is nice for most file types. Really it's a good upgrade, if you are me.

I am going to be away for the next few weeks, leading this weird busy schedule involving travel. Or watching other people travel. Should be exciting, maybe I'll have more fun stories.

Panda

I was walking around New York a couple of days ago, one of those short visits where one gets to enjoy expensive hotels and rain. I went to an Afghani restaurant, supposedly owned by someone who was a judge in Kabul in the late 70s. Funny considering his leaving the country would neatly coincide with the rise of the communists, and such things usually have interesting stories. The restaurant itself didn't have a story, it wasn't even exceptionally good. At least I didn't really pay for the food. The rain did actually suck.

I am not sure why I wrote this, I didn't have that much to say. Observations for the night: scheme's cool, coding is fun, writing papers is less fun than it used to be.

Panda

Following the publication of a weird paper about MapReduce and ML, Itay (you should go to his blog, he has finally updated it) and I worked on an implementation of MapFold in Erlang. We actually intended on making MapReduce in Erlang, but forgot to read the original paper, instead taking our cue from what Functional Programming defines map and fold to be. Well they are equivalent anyways, except for the part where distributing fold with our semantics is hard (think about how you'd distribute folding product over matrices). Doing this in Erlang is surprisingly easy, neither of us had ever used Erlang before, and yet within a night we got a fairly substantial part of this done. This led us to some questions about why Erlang receive clauses work the way they do, turns out scala and most other languages have similar semantics, but it would seem that with very little overhead at compile time, one could make it so that in some cases message receiving is fast. Specifically, people in the past have complained about how Erlang's slow when you're looking for one specific, static, message in a queue full of other messages you aren't interested in. We did not want to change the preexisting receive construct too much, since the entire pattern matching thing is fairly intuitive, and changing the structure doesn't get us much gains in the worse case. The compiler tricks simple, but people with more PL experience than us seem to agree that it cannot really be done easily without looking at the program text, and hence outside the compiler, which is really quite sad.

My PL class just started using Scheme macros for performing Fischer transforms into CPS, and I am a little freaked out by them. It is hard to say why, they are powerful, make sense, and are well liked, I am just not convinced that using macros isn't cheating in some sense of that word.

I seem to be traveling far too much in the next month, this should be interesting.

Panda

 
  std::vector<int> something[numelts];
  something[7].push_back(2);
  int j = something[7][0];
 

and

 
  std::vector<int> something[numelts];
  something[7].push_back(2);
  int j = (something[7])[0];
 

.
The later is correct, the first is wrong, C++ seems to be treating unparenthisized things as normal array lookups, while the parenthisized thing leads to it using operator[] from vector.

panda

I have been looking for a good reference to point people I argue with to, mostly because I do this a lot, and do it on both sides of the world (partially retro mostly mechanical things are weird, though friends in India are more likely to agree about really expensive cars). Well I think this pretty accurately reflects my feelings on liking weirdly expensive things.

Now back to work.

panda

Since I can't compile and run said code on my home machine for lack of expensive libraries, and I'd rather not code in the department for a general lack of productivity, using Hg seemed like a better idea than just scping code around. Now this in general presents a problem on Mac OS X. For reasons I haven't yet discovered (I have not yet taken a look at the hg code), mercurial while remotely cloning from Linux seems to ignore the PATH variable and will explicitly look for hg in /usr/bin, which is not where Mercurial resides for a whole lot of OS X users. Bad in terms of design, and probably a fairly simple fix, but hardly the end of the world seeing as a simple ln -s solves the problem for now. Either ways once I was done with this, I was ready to commit all my past work to a repository, which is when I discovered a fairly large roadblock which has little to do with mercurial, and this got me thinking about a few other backup strategies I had been playing with in my mind...

At some point in time, I had gone through the effort of creating a few hundred thousand test cases for this program I am writing, they are all pretty small (around 1K each), but there are something like 180000 of these. For reasons which involve everything from a slow hard disk, to perhaps the general lay out of the HFS+ filesystem, doing much of anything with these files, even lsing them, takes a while. Creating hashes and metadata for these files probably did not make Hg very happy, and at some point in the middle of trying to commit all these changes I was struck with the realization that just listing every one of these 180000 files in a log file will take Hg a while.

Now of course this realization hit me after it had already been running for a while, and had probably been churning away at creating and writing all those hashes and metadata. Either ways I ended up trying to cancel the commit by hitting ctrl-c. For future reference, don't hit ctrl-c too many times when canceling a commit, yes it takes a while to respond to interrupts when processing this big a commit, but it really is trying to rollback the log and leave your system in a known state. Lots of ctrl-cs might save you time with canceling, but you will spend as much, if not more time, biting your nails as you run through hg recover. Anyhow this worked out, but it got me thinking...

Now obviously the sane way to deal with so many files is tar them so you usually have to deal with no more than one file, in my case I need to use them individually, but even that is a bad response. Most people don't really tend to use a 180000 files in a single directory, bad structuring and problems with finding information usually prevent this.

However, and this is the fun part, there is at least one situation where the specs of a specific system make it hard to ensure this limit. Amazon's S3 is described as a scalable, reliable, low latency storage solution designed using the same storage infrastructure used by Amazon itself. More importantly, S3 is fairly well priced, and I am running out of space (though there's all sorts of debates there, is 15 cents a month per GB really less expensive than spending even 300 dollars for a terabyte of storage), and while working on something for my father I did actually consider the possibility of using S3 as a place to back stuff up, or even as a backing storage for everything I have on my disk. In essence you could get your disk to act as a really big cache for S3, since networks aren't that fast I think it is safe to assume that real disks are much lower latency than S3, besides I don't like diskless nodes.

Now security is an obvious concern for this, but there is already a single company controlling vast amounts of my information, and information for a vast majority of the people I know, and swapping one company for another is not the worse thing. More importantly people are already working on the security problem. However, and this is where the entire mercurial discussion ties in with this, S3 is a flat file format. You could perhaps try and get more than one buckets, but buckets are fairly hard to get, and buckets merely contain keys linking to single objects which can be upto 5 GBs of information. So let's see, one could create a real file system on top of these buckets and use these 0kb-5gig blocks as actual file system blocks, however for any reasonably recent file system which does versioning, you'd rather rapidly reach a point where you're using enough blocks that attempting to list everything in your bucket will take a long time just in terms of transferring data over the network. Fortunately if you intelligently designed this, listing blocks should be a fairly rare process, perhaps something coming into play only every time you loose your cache (disks are unreliable, things happen). However I don't actually know how S3 stores this index, and how access to object maps out, and I can't really find out how they do this, I am just not sure how Amazon handles the case of an overly full bucket, it doesn't seem like a hard place to get to, hourly backups of entire 5 gb files might do this rather fast, and I am sort of curious about how this works out. Of course seeing as MySql now has a S3 backend, and other people have already made various backup things for S3, and of course seeing as networks are still rare to come across when traveling, I am not really expecting to see too much of this entire S3 backing disk idea come about anytime soon. If someone's working on something similar, it'd be nice to know.

Panda