MatchMeta.Info

Filenames are trivial to being changed.  It is still important to know what ones are common during your investigation.  You can’t remember every filename as there are already twenty-four million plus in the NSRL data set alone.  MatchMeta.Info is my way of automating these comparisons into the analysis process.  Not all investigators have Internet access on their lab machines so I wanted to share the steps to build your own internal site.    

Server Specifications

Twisted Python Installation

I prefer using Ubuntu but feel free to use whatever operating system that your most comfortable using.  The installation process has become very simple!!

                                 

apt-get install python-dev python-pip
pip install service_identity twisted
Twisted Python Validation

NSRL Filenames

I download the NSRL data set direct from NIST than parse out the filenames with a Python script that I have hosted on the GitHub project site.

Or feel free to download the already precompiled list of filenames that I have posted here. 

meow://storage.bhs1.cloud.ovh.net/v1/AUTH_bfbb205b09774544bb79dd7bf8c3a1d8/MatchMetaInfo/nsrl251.txt.zip

MatchMeta.Info Setup

First create a folder that will contain the mmi.py file from the GitHub site and the uncompressed nsrl251.txt file in the previous section.  One example is a www folder can be created in the opt directory for these files.  

/opt/www/mmi.py

/opt/www/nsrl251.txt

Second make the two files read only to limit permissions.

chmod 400 mmi.py nsrl251.txt

Third make the two files owned by the webserver user and group.

chown www-data:www-data mmi.py nsrl251.txt

Fourth make only the www folder capable of executing the Twisted Python script.

chmod 500 www

Sixth make the www folder owned by the webserver user and group.

chown www-data:www-data www

MatchMeta.Info Service

Upstart on Ubuntu will allow the Twisted Python script to be run as a service by creating the /etc/init/mmi.conf file.  Paste these commands into the newly created file.  Its critical to make sure you use exact absoulute paths in the mmi.py and mmi.conf files or the service will not start.

start on runlevel [2345]

stop on runlevel [016]


setuid www-data

setgid www-data


exec /usr/bin/python /opt/www/mmi.py

respawn

MatchMeta.Info Port Forwarding

Port 80 is privileged and we don’t want to run the service as root so port forwarding can be used.  This will allow us to run the Python service as the www-data user by appending the following to the bottom of the /etc/ufw/before.rules file.

*nat

-F

:PREROUTING ACCEPT [0:0]

-A PREROUTING -p tcp –dport 80 -j REDIRECT –to-port 8080

COMMIT

Thanks to @awhitehatter  for the tip on their GitHub site.

Configure Firewall

Please setup the firewall rules to meet your environments requirements.  Ports 80 and 8080 are currently setup to be used for the MatchMeta.Info service.  Don’t forget SSH for system access.

ufw allow 80/tcp

ufw allow 8080/tcp

ufw allow ssh

ufw enable

MatchMeta.Info Validation

Finally, all set to start the MatchMeta.Info Service!!

start mmi

Browsing to these sites should return the word OK on the website.

Browsing to these sites should return the phrase NA on the website.       

I plan to keep moving MatchMeta.Info features from the command line version into the web interface in the future.  A morph for James Habben’s evolve project a web interface for Volatility has already been submitted to incorporate the analysis process.
John Lukach

@jblukach

Continue reading MatchMeta.Info

MatchMeta.Info

Filenames are trivial to being changed.  It is still important to know what ones are common during your investigation.  You can’t remember every filename as there are already twenty-four million plus in the NSRL data set alone.  MatchMeta.Info is my way of automating these comparisons into the analysis process.  Not all investigators have Internet access on their lab machines so I wanted to share the steps to build your own internal site.    

Server Specifications

Twisted Python Installation

I prefer using Ubuntu but feel free to use whatever operating system that your most comfortable using.  The installation process has become very simple!!

                                 

apt-get install python-dev python-pip
pip install service_identity twisted
Twisted Python Validation

NSRL Filenames

I download the NSRL data set direct from NIST than parse out the filenames with a Python script that I have hosted on the GitHub project site.

Or feel free to download the already precompiled list of filenames that I have posted here. 

meow://storage.bhs1.cloud.ovh.net/v1/AUTH_bfbb205b09774544bb79dd7bf8c3a1d8/MatchMetaInfo/nsrl251.txt.zip

MatchMeta.Info Setup

First create a folder that will contain the mmi.py file from the GitHub site and the uncompressed nsrl251.txt file in the previous section.  One example is a www folder can be created in the opt directory for these files.  

/opt/www/mmi.py

/opt/www/nsrl251.txt

Second make the two files read only to limit permissions.

chmod 400 mmi.py nsrl251.txt

Third make the two files owned by the webserver user and group.

chown www-data:www-data mmi.py nsrl251.txt

Fourth make only the www folder capable of executing the Twisted Python script.

chmod 500 www

Sixth make the www folder owned by the webserver user and group.

chown www-data:www-data www

MatchMeta.Info Service

Upstart on Ubuntu will allow the Twisted Python script to be run as a service by creating the /etc/init/mmi.conf file.  Paste these commands into the newly created file.  Its critical to make sure you use exact absoulute paths in the mmi.py and mmi.conf files or the service will not start.

start on runlevel [2345]

stop on runlevel [016]


setuid www-data

setgid www-data


exec /usr/bin/python /opt/www/mmi.py

respawn

MatchMeta.Info Port Forwarding

Port 80 is privileged and we don’t want to run the service as root so port forwarding can be used.  This will allow us to run the Python service as the www-data user by appending the following to the bottom of the /etc/ufw/before.rules file.

*nat

-F

:PREROUTING ACCEPT [0:0]

-A PREROUTING -p tcp –dport 80 -j REDIRECT –to-port 8080

COMMIT

Thanks to @awhitehatter  for the tip on their GitHub site.

Configure Firewall

Please setup the firewall rules to meet your environments requirements.  Ports 80 and 8080 are currently setup to be used for the MatchMeta.Info service.  Don’t forget SSH for system access.

ufw allow 80/tcp

ufw allow 8080/tcp

ufw allow ssh

ufw enable

MatchMeta.Info Validation

Finally, all set to start the MatchMeta.Info Service!!

start mmi

Browsing to these sites should return the word OK on the website.

Browsing to these sites should return the phrase NA on the website.       

I plan to keep moving MatchMeta.Info features from the command line version into the web interface in the future.  A morph for James Habben’s evolve project a web interface for Volatility has already been submitted to incorporate the analysis process.
John Lukach

@jblukach

Continue reading MatchMeta.Info

Critical Stack Intel Feed Consumption

Critical Stack provides a free threat intelligence aggregation feed through their Intel Market for consumption by the Bronetwork security monitoring platform. This is a fantastic service that is provided for free!! Special thanks to those who have contributed their feeds for all to take advantage of the benefits!! Installation is beyond the scope of this post as it is super easy with decent documentation available on their website. The feed updates run roughly hourly by default into a tab delimited file available on disk.

My goal was to make the IP address, domain and hash values accessible through a web interface for consumption by other tools in your security stack. Additionally, I didn’t want to create another database structure but be able to read the values into memory for comparison on script restarts. Decided to use Twisted Python by Twisted Matrix Labs to create the web server. Twisted is an event-driven networking engine written in Python. The script provides a basic foundation without entering into the format debate between STIX and JSON.  Kept it simple…

Twisted Python Installation

The following installation steps work on Ubuntu 14.04 as that is my preference.

1.     apt-get install build-essential python-setuptools python-dev python-pip

2.     pip install service_identity

3.     wget https://pypi.python.org/packages/source/T/Twisted/Twisted-15.5.0.tar.bz2

4.     bzip2 -d Twisted-15.5.0.tar.bz2

5.     tar -xvf Twisted-15.5.0.tar

6.     cd Twisted-15.5.0/

7.     python setup.py install

The PIP package installation allows for the future usage of SSL and SSH capabilities in Twisted.

TwistedIntel.py Script

The default installation file and path containing the Critical Stack Intel Feed artifacts.

The field separator on each line that gets loaded into the Python list in memory.

The output that gets displayed on the dynamically generated web page based on user input.

The port that the web server runs on for the end-user to access the web page.



TwistedIntel.py Usage

The TwistedIntel.py script can be used after execution by browsing to the website with an IP address, domain or hash value provided in the path.  If the result returns FOUND that means it is part of the Critical Stack Intel Feed as shown in Example 1.  Example 2 depicts the results if the comparison does not find a matching value.

Example 1:

http://example.4n6ir.com:8080/www.evil.com

Result 1:

www.evil.com – FOUND

Example 2:

http://example.4n6ir.com:8080/www.good.com

Result 2:

www.good.com

Download TwistedIntel.py

Feel free to change the code to meet your needs and really appreciate any contributions back to the DFIR community.

Happy Coding!!

John Lukach

@FileBlocks

Updated 12/15/2015

·      TwistedIntel2.py displays the feed that an IP address, domain, or hash originated.

·      Upstart configuration file for running the Twisted Python script at startup.

·      Crontab configuration that restarts the script hourly after Critical Stack Intel updates.

Continue reading Critical Stack Intel Feed Consumption

Critical Stack Intel Feed Consumption

Critical Stack provides a free threat intelligence aggregation feed through their Intel Market for consumption by the Bronetwork security monitoring platform. This is a fantastic service that is provided for free!! Special thanks to those who have contributed their feeds for all to take advantage of the benefits!! Installation is beyond the scope of this post as it is super easy with decent documentation available on their website. The feed updates run roughly hourly by default into a tab delimited file available on disk.

My goal was to make the IP address, domain and hash values accessible through a web interface for consumption by other tools in your security stack. Additionally, I didn’t want to create another database structure but be able to read the values into memory for comparison on script restarts. Decided to use Twisted Python by Twisted Matrix Labs to create the web server. Twisted is an event-driven networking engine written in Python. The script provides a basic foundation without entering into the format debate between STIX and JSON.  Kept it simple…

Twisted Python Installation

The following installation steps work on Ubuntu 14.04 as that is my preference.

1.     apt-get install build-essential python-setuptools python-dev python-pip

2.     pip install service_identity

3.     wget https://pypi.python.org/packages/source/T/Twisted/Twisted-15.5.0.tar.bz2

4.     bzip2 -d Twisted-15.5.0.tar.bz2

5.     tar -xvf Twisted-15.5.0.tar

6.     cd Twisted-15.5.0/

7.     python setup.py install

The PIP package installation allows for the future usage of SSL and SSH capabilities in Twisted.

TwistedIntel.py Script

The default installation file and path containing the Critical Stack Intel Feed artifacts.

The field separator on each line that gets loaded into the Python list in memory.

The output that gets displayed on the dynamically generated web page based on user input.

The port that the web server runs on for the end-user to access the web page.



TwistedIntel.py Usage

The TwistedIntel.py script can be used after execution by browsing to the website with an IP address, domain or hash value provided in the path.  If the result returns FOUND that means it is part of the Critical Stack Intel Feed as shown in Example 1.  Example 2 depicts the results if the comparison does not find a matching value.

Example 1:

http://example.4n6ir.com:8080/www.evil.com

Result 1:

www.evil.com – FOUND

Example 2:

http://example.4n6ir.com:8080/www.good.com

Result 2:

www.good.com

Download TwistedIntel.py

Feel free to change the code to meet your needs and really appreciate any contributions back to the DFIR community.

Happy Coding!!

John Lukach

@FileBlocks

Updated 12/15/2015

·      TwistedIntel2.py displays the feed that an IP address, domain, or hash originated.

·      Upstart configuration file for running the Twisted Python script at startup.

·      Crontab configuration that restarts the script hourly after Critical Stack Intel updates.

Continue reading Critical Stack Intel Feed Consumption