EPISTOLARY PURGATORY

This semester I am taking a class on Image and Video Processing and it has opened me up to the wildly precarious scaffolding that all of our visual media works on. Thus far we have covered a good amount of predictive coding, optimization and image compression.

Time and again, though, I have been straying off my usual homework to poke around standard algorithms and find interesting ways in which their behavior may reveal itself to the user. Needless to say, Rosa Menkman has been a great source of inspiration during my Sunday-night detours like these.

Wavelets

Wavelets have been kicking around the dusty attics of my academic world for a couple of years now, but I had never quite explored them until last week. The overall concept is to pick a wavelet (which is a small piece of a wave, so to speak) and find a way to layer various sizes of them on top of each other to form an image.

By doing a Wavelet transform of an image, what we get is a big set of weights with which one could combine each repetitive wavelet-layer(imagine transparent tiles of various sizes but the same pattern on each of them) and then add them all up to form an image. An interesting point to note here is that the smaller wavelets correspond to finer aspects of the picture (such as little inscriptions, patterns, brushstrokes and such) while larger wavelets correspond to overall dynamics (bright center+dim corners maybe?) Hence, every image has its own unique weighted sum of wavelet functions and coefficients.

Many image compression algorithms use wavelet transforms to separate finer details in a picture from the prominent ones, and exercise discretion on which parts can be ignored to reduce its bit size. A major chunk of research in Electrical Engineering has led to people coming up with optimum weights for these coefficients so as to cut down on details without the viewer noticing it — hence allowing the algorithm to maintain its transparency with respect to the end user. The JPEG2000 algorithm is the result of some that work.

Since a lot of this is heavily abstract and mathematical in nature, I set out to break it so as to obtain a visualization of how wavelet coefficients contribute to even form an image. An early step was me trying to randomize some coefficients to see them misbehave on screen. Here is one:

While there is little or no coherence in the image, it is worthwhile to notice repeated occurrence of L shaped formations all over. This was because I had left the coefficients controlling vertical and horizontal differences and randomized the diagonal differences.

Rabbit Hole

After some more of tinkering and the frenzy of homework submission, I had quite a bit of working code that I could easily use outside their intended function. Seeing the potential for aesthetics using wavelet artifacts, I tried deliberately breaking images by weighting their wavelet coefficients differently, sometimes switching their places. I picked an image of Archeological Reminiscence by Salvador Dali off the web for this. Here are a few outputs:

(Original Image)

The kind of directional coefficients that I played with become even more apparent after Wordpress uses its compression before displaying it on a webpage. Having done this for a bit, I asked myself another question — What if I used differential coefficients from an image over the approximation coefficients of another image?

I went ahead and did that. The results were pretty sweet. Here are a couple of them.

In the first one, the dynamics of Persistence of Time juxtapose with the objects in Archeological Reminiscence in very interesting ways. In the second case, King of Limbs develops a dotted canvas-like texture when combined with Hands Cannot Erase.

End Note

Clearly, Wavelet deformation has great potential for aesthetics, and it would be interesting to see how these would appear if combined with other image/video coding techniques related to prediction and compression. I’m not sure how deep I will dig into this rabbit hole in the days to come, but if I do, I will certainly write about it.

Meanwhile, feel free to look around and play with some of the reusable code I put together on MATLAB to do this.