0x0Lab Blog Just another damn blog


DirBuster Dictionary Populator

Dirbuster and dirb are in the toolset of all web application security fans. Both tools are excellent (although I prefer dirb due to it being command line and not Java), but their results obviously depends on how good the wordlist you are using is. I often find myself editing the wordlist file to add directories/files relevant to the site I'm running dirbuster against. So one day I though: hey why not automate this as much as I can.
Here comes dict_populator.pl.
The tool will crawl and discover URL's in the site received in its input and output the following:
- A file with all discovered URL's (site.name-urls.txt)
- dict.txt (Will append any directory/file names found while crawling and not found in the dictionary to this file)
- ext.txt (Will append any extensions found while crawling the site and not found in the file)

The script also supports using cookies for better crawling of sites that require authentication. The cookies can be added in the first few lines of the script (no command line switch yet).

Just as a test I used the common.txt wordlist found in dirb to scan one of my sites. The scan did not include any extensions and discovered 25 directories. I run dict_populator against the site making it append the discovered names in the above wordlist and then run dirb again. This time 34 directories where discovered.

Download: dict_populator.pl
Requires: LWP::UserAgent, HTTP::Cookies and HTML::LinkExtor.

Updated to version 0.2, which fixes a bug that multiple image URL's appear in the output.

Posted by cirrus

Comments (2) Trackbacks (1)
  1. Is it better than Wikto? Please explain.

  2. @soroush:
    Essentially it is very similar to what Wikto does.
    The main differences (excluding the added functionality wikto has) are:
    – Wikto adds full directories spidered to the backend scanner. It will add “/2010/06/dirbuster-dictionary-populator/” but not “2010”, “06” and “dirbuster-dictionary-populator” in the wordlist and will only add it in the Directories wordlist not the files wordlist (feel free to correct me if I’m wrong, I’ve only tried Wikto a few times).
    – Wikto runs in Windows. I usually don’t run Windows and prefer command line utilities (like dirb).

    Again this is not intended to be run by itself, but rather as an addon to the excellent DirBuster and dirb.