WYAE - Thekla v12.8.29

When performing a web application security tests, you need a list of URLs and forms to test. For this a web pen tester needs a list of all forms and dynamic URLs which (s)he should check.

Thelka is a web spider/crawler designed to extract paramter-containing URLs and forms. The result are text and CSV files, listing the URLs of dynamic web pages and their referers.

Requirements

Usage

You need to configure target specifications at the top of the script file.

First all starting URLs from which the spider should start:

	all_urls = [ 'http://www.wyae.de/' ]
Then the domains the spider must not leave (as many as you like):
	valid_urls = ['http://www.wyae.de/docs','http://www.wyae.de/wyae']
and domains the spider must not visit (as many as you like):
	forbidden_urls = ['http://www.wyae.de/software','http://www.wyae.de/lists']
Then you can call the phython script which will spider the web site, showing URLs found, downloaded and to-be-worked-on. All results are logged into configurable CSV files.

You can configure a proxy in the top section of the file. Default is direct connection

	
	proxies = None
	# proxies = {'http': 'http://localhost:8080'}

Known Bugs/Limitations:

Evaluation

Thekla only does the numbing web site enumeration. It will not perform any security checks or vulnerability settings. It only extracts a number of URLs a web application tester should check as they are found to contain parameters.

Beware:
Depending on web application architecture Thekla can run into endless loops. Have an eye on it while it is running and abort if that seems to be the case.

If you need a security consultant to check your network, systems or architecture, or to help you with a security problem/incident or (a better approach) check your security architecture and risks, simply contact me.
;-)

Downloads

License

For details on licensing and support please see there.

This software is published under a "Modified BSD" License", an open source license.