It took me a few months before I decide “All My Files” is a useless feature for Mac OS Lion.
I was irked that it is the default view when you create new Finder window. And I was even more irked that the default arrangement by Kind shows my Adium chat log right at the top..
So I decided to change it.
Finder > Preferences > General > New Finder windows show: Downloads
Finder > Preferences > Sidebar > Uncheck as Favorites
If I still want to access “All My Files”, I could still use the handy shortcut Shift+Cmd+F.