If you do any file processing, you will inevitably find yourself writing recursive code to traverse a directory structure. If you have a directory structure that is broad and deep with files/folders numbering in the thousands, recursion is inefficient as it consumes stack resources. In some cases you will run out of stack space and cause the code to throw an error.

You can do this more efficiently by using a seldom-used class of the System.Collections namespace: Stack. The Stack class implements a LIFO data structure (Last In First Out), as opposed to Queue, which implements FIFO (First In First Out). Instead of deferring to the run-time to implement the stack (which is essentially what recursion does, but in an inefficient way), you can optimize the process and also prevent your code from breaking, by managing the stack yourself.

Here is a simple example that will traverse and print-out all the files in a directory structure without using recursion:

private void TraverseFolder(DirectoryInfo dir)
{
	Stack stack = new Stack();
	stack.Push(dir);
	while (stack.Count > 0)
		{
			DirectoryInfo dirInfo = (DirectoryInfo) stack.Pop();
			foreach(FileInfo file in dirInfo.GetFiles())
			{
				System.Web.HttpContext.Current.Response.Write(file.FullName + "<br>");
			}					   
			DirectoryInfo[] subDirs = dirInfo.GetDirectories();
			for (int index = subDirs.Length - 1; index >= 0; index -= 1)
				stack.Push(subDirs[index]);
		}
}

Looks like Lee Sykes beat me to the DotNetNuke lens on Squidoo. Interestingly enough, the ASPnet lens was available for the taking. I also picked up WebDAV, Tagging and Logos.

Umair could not have made a more timely post about Yahoo’s lack of innovation. He discusses Yahoo’s “hyperimitation” and how it never really improves anything in any meaningful way.

I say timely because I have now spent some time engineering a better “forums” application because I have given up on forum software available today. Many of us use web-based discussion forums everyday, and they are all the same. From the common seed of BBS many years ago, there evolved a design for web discussions — forum groups, forums, threads and posts. It all worked quite well until everybody had a computer, a browser, an Internet connection and an opinion. This has resulted in three major problems:

– It’s hard to keep up with the volume of information. With hundreds and sometimes thousands of posts in different forums, across different sites, how does one track information of interest and more importantly relevance. Email digests and email subscriptions don’t cut it. They just move the information to different client software. If you track even 3–5 busy forums, you will be innundated with information.

– It’s hard to find relevant information. Forums are notoriously hard to search because there is no meaningful way to perform contextual searches. Users freely post anything (even in moderated forums) and there is no guarantee that threads will stay focused on a topic. Keyword searches yield tons of results and out of frustration, users will often just post a question which has already been answered umpteen times.

– It’s hard to tell which user’s posts are worth reading. Just because a particular user has a high post count does not make them interesting, relevant or for that matter, credible. For instance, there are many users on the ASP.Net forums with huge post counts that have very little substance in their posts.

As some of you may have noticed, I have not been very active blogging or on the forums lately. And yes, I am also behind on releasing updates to some of Speerio products. This is because I have been hard at work doing a fair amount of re-architechting of existing products, while also working on a Speerio product suite called “Community Studio” (more on that next week). One of the apps in the suite is a discussion application. In working out the design for this app, I was tempted to take the interaction metaphor used by most most forums today and implement it, perhaps with some U.I. goodies to make things interesting. But the more I thought about it, the more I disliked this idea. There is no point in replicating a bad concept with a good U.I. — all it does is give you more of the same problems, albeit with better user interaction. A good example of this is the recent upgrades made to the ASP.net forums. Many client-side U.I. enhancements, but none that substantially help with usability and easily finding relevant information.

I decided that I would take a fresh approach to designing a web-based discussion application (I’m not going to call it a forum any more). Starting with a clean slate, I prioritized some key requirements —

1) The application should enable a user to instantly know what topics are discussed the most.

2) The application should enable a user to instantly find all discussions on a topic, prioritized by relevance, chronology or amount of participation for all time or a user-specified period of time.

3) The application should enable the user to be notified when more discussions about topics of interest to her/him emerge.

4) The application should provide the user with hints about contextually relevant topics that have been discussed when he/she searches for a topic.

5) The application should be incredibly easy to use, very responsive and should be optimized to take advantage of the one trait that is common to all web-based discussions — 99.99% of the time all content is write-once/read-many.

Looking at those requirements, it should be obvious that the group – forum – thread metaphor isn’t going to cut it. The more I thought about it, the more I became convinced that web-discussion applications have been over-engineered and the solution is not to add, but to subtract. Doing just that, I arrived at a design that is quite different from today’s discussion software, but which I think meets all the above requirements. I’ll go into detail in a later blog post, but let me share some high-level ideas here:

1) Information navigation is done using a folksonomy-generated tag cloud.

2) There are no groups or sub-groups or forums…only a flat collection of discussions (100 or 1,000,000…doesn’t matter)

3) Each discussion is natively stored as an RSS feed. A discussion is the originating post and all responses to that post. There is no hierarchical thread, just a linear series of responses.

4) Each user who posts a response, must tag her/his response with tags that are relevant to the content of her/his post. This is key because contextual relevancy is best determined by humans and not computers (at least given today’s technology).

5) Tag clusters are automatically created for contextual relevancy.

This, in a nutshell, is a design that meets all the requirements. Of course, there is a lot of detail in the implementation and the U.I., but from a 20,000–foot perspective, the design is very, very simple. While technical gurus are busy arguing the merits of a folksonomy over structured taxonomies, ordinary people are enjoying their ease of use, both for content creation and retrieval. This makes me confident that even though my folksonomy-based discussion app will take some getting used to, once users see how it makes web-based discussions easier to track, digest and retrieve, they will love it.

More design details and information on Community Studio to follow.

Great to know that Ray Ozzie is blogging again.

Google is now providing what looks to be a pretty cool service for anaylzing web traffic. Google Analytics is free upto 5M page views for most sites and unlimited for sites using AdWords. Nice.