Simpy Chichimichi

Uh, all this talk about del.icio.us is too much, but this is a note for those still migrating to Simpy from del.icio.us:

NOTE: Syncing with del.icio.us is currently kaput because del.icio.us is apparently down ("del.icio.us is down for maintenance. we'll be back in one hour."). Please sync later today or whenever del.icio.us comes back to life. The sync is here. Over and out.

Ari Paparo wrote a nice post about del.icio.us vs. Blink. Reminder: Blink used to be a bookmarking service. Not quite like Simpy or del.icio.us, but similar in the sense that it dealt with people's links. Ari's post is not so much about one vs. the other and which was/is better, but about the different approaches taken by two services, and their consequences. As I used to work at Blink back in 2001/2002, I often wondered what other "Blinkers" think about what is happening with Social Bookmarks now, 3-4 years after Blink. I also wondered whether people doing bookmarks back in 1999, like Ari, would argue that it's all the same old stuff, just branded as "woho, social, Web 2.0, so it must be good". I am very happy to see that Ari doesn't equal the two approaches to solving the same or very similar problem, and compares them rather objectively.

I didn't join Blink early enough to participate in decisions about the company's direction, but in Blink's defense, one thing that all services, companies, and individuals now have at their disposal is that large echo-chamber - "the Blogosphere". Back then, in the Blink era, there were link exchanges and revenue sharing models, but there was no real Network Effect infrastructure that exists today. The Blogosphere is the megaphone that insures that, if you yell loud enough, and you yell about interesting stuff, you will be heard! What, you didn't hear me, you say?

What's a "splog"? Don't know? spam + blog = splog.

But what about "sprefs"? (freshly coined, yes) I haven't seen anyone talking about sprefs, despite seeing a lot of chatter about splogs, comment spam, wiki spam, and such. Here is an example of what I'm caling a "spref":

218.22.246.34 - - [13/Dec/2005:03:11:21 -0500] "GET / HTTP/1.1" 200 44403 "http://bontril.threethreethree.us/" "Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021)

The geekier portion of the readership will recognize this is a line from the web server log file. See that bolded URL? That's "spref", a fake referer [sic] that is supposed to make you think that somebody from that page came to your site, and intrigue you enough that you go and visit that page. Of course, this referral is completely artificial, there is really no link to your site on that page, and you end up at some v1@gr@ site. So now we have: spam + refer(r)er = spref.

I think I'll just turn this blog into a Web 2.0 M&A charting service. I fall for graphs and charts, can't help it. Here are some recent acquisitions, via Alexa:

What is interesting about these two services, from the technical point of view, is that both of them are built on Perl. Remember Perl? The Web 1.0 web programming language of choice, remember? While some people thing that Web 2.0 equals Ruby on Rails + Ajax + Tags, companies using completely different technologies are being built and sold. What about Flickr? PHP, apparently - check this.

Funny, I just posted about Slashdot the other day, and today heaps of Slashdot readers decided to join Simpy after finding out Yahoo! bought del.icio.us. I don't think there is a reason to panic, though - Yahoo! seems like a good company.

Regardless, welcome to Simpy and make yourself at home, and if you want to migrate over from del.icio.us, you can do it here (after you log in, of course).

Short and geekly sweet this post will be. I've talked about crawlers and aggregators and how they affect web site traffic patterns twice in the past: first in Web 2.0 Tagging Study, Part 4: Nocturnal Activity Pattern, and then in Web 1.0 vs. Web 2.0 Web Activity Pattern.

In order to get a better understanding of Simpy's usage, I wanted to weed out the numerous crawlers and aggregators that constantly hit Simpy, crawl it, fetch its web pages and suck its feeds. As Simpy uses Apache web server, this was quite simple to do, using mod_setenvif Apache module. The following is a snippet from Simpy's web server configuration file:

CustomLog /log/apache/access_log combined env=!DoNotLog
CustomLog /log/apache/robot_log combined env=ROBOT

<IfModule mod_setenvif.c>
  ## you need to make all this be a single line
  BrowserMatchNoCase "Feed|Google Desktop|Bloglines|Teoma|NaverBot|msnbot|Googlebot|
  Slurp|crawler|Crawler|rojonet|NewsGator|MagpieRSS|PubSub-RSS|AppleSyndication|
  ZyBorg|ia_archiver|kinjabot|Yahoo-Blogs|BecomeBot|Blogslive|Blogpulse|SBIder|
  NetNewsWire|ping.blo.gs|RssBandit|Baiduspider" ROBOT=1 DoNotLog=1
</IfModule>

Clearly, the trick is to list all user agents that you want to exclude, set an environment variable, and then use that variable to do conditional logging of web requests. You will notice that my configuration is a bit redundant, as I could really accomplish the same with only 1 variable.

This is obviously not rocket science, but I find it useful to log human and non-human requests separately, and I thought I'd share this, in case anyone else wants to do the same.

While everyone is busy writing and thinking about Web 2.0, Ajax, Tag Clouds, Ruby and Ruby on Rails, landing pages with just an email field, etc., nobody is talking about Slashdot any more. However, check out what is happening to Slashdot, that old Web 1.0 site without Ajax and Tag Clouds, written in Perl:

Attention? No, just the basic, known, tractable, human addiction!

In the original Simpify for Blogs I provided several HTML snippets that you can stick in your blogs and let your blog readers save and tag your posts in Simpy. Unfortunately, I managed to make a mistake in the snippet for WordPress, as well as Movable Type and TypePad (that's about 50% error, I really have to pat myself on the back for this one). The snippets are now corrected. If you use WordPress, get the correct snippet here, and if you use Movable Type or TypePad, the correct HTML snippet is here.

Another Simpy Chichimichi reader, Kossatsch, also provided the following HTML snippet for users of Textpattern blogging software:

Finally, note that all HTML snippets have extra spaces in them, and you have to remove these spaces when pasting the HTML in your blog template. I had to put the spaces there in order to force the snippets to wrap inside the blog post page. The spaces are in front of '?href' and in front of '&title'.

The regular Simpy Chichimichi readers may remember that my Web 2.0 Study series included the analysis of web usage activity observed at Simpy. I have recently done some more work in that area, and will present relevant charts and geek tips in one of the future posts.

Remember "Web 1.0"? Remember what your web log reports looked like? I do! They look something like this (look at the very first picture). In this post, I'll show you several charts that show how different the web usage patterns are in the "new Web 2.0 world". In the old world, say 5 years ago, you'd analyze your web logs and see your typical "during-work-makes-your-wonder-usage-hump" - an increased web usage during the working hours, at least if you were in the U.S. With Web 2.0 this has changed. What's Web 2.0? Ha ha ha...well, I won't get into that now, others have already beaten that dead horse enough, but I think we can all agree that with what we are calling Web 2.0 RSS and its younger cousin Atom climbed the tech charts (hm, no pun intended, really, but here is a pretty little chart). They climbed the charts despite not being new, just like....what else? (hint: shares the name with a detergent I used for washing a bathtub when I was a kid) How about another trend chart. Did I mention I love trends and their visualizations?

Back to the main topic... Because feeds gained so much in popularity, web sites nowadays see a whole lot of new type of activity, and this activity happens in non-work hours.

The following heatmap-like picture represents the recent Simpy web usage. Lighter colors represents increased web activity. To get this chart I excluded various robots/spiders/crawlers, as well as news/feed aggregators/readers, in order to derive usage by only (us) humans. Can you see that "during-work-makes-your-wonder-usage-hump"? Of course you can, I marked it up! But you can also see some increased activity right after midnight. What's that? That's me not excluding quite all non-human activity.

Here is another display of the same activity, but as a more familiar bar chart, with the same day periods marked up.

Clearly, Monday to Wednesday, and 07^h-17^h EST are peak usage periods. This is the type of activity we were used to seeing in pre-feed, pre-Web 2.0 era. What about now? Well, it's a very different story. The following charts show the web usage of this particular blog.

As you can see, the bulk of activity occurs right after midnight. It's not that no human reads this blog, and it's not that people 2.0 changed their reading schedules, it's just that now we no longer have only the web search engine robots/spiders/crawlers visiting us. The web user agent crowd was expanded by numerous feed/news aggregators that seem to prefer the 00:00-01:00 EST period to any other. Why? Oh, and have you noticed Argus visiting your site? Argus is a busy bot, I'll introduce him some other time.