i MISP – Malware Information Sharing Platform – All things in moderation

MISP – Malware Information Sharing Platform

Hi guys!
Today, I will introduce you a great platform. It is MISP – Malware Information Sharing Platform. Now let’s go!

What is MISP?

MISP, Malware Information Sharing Platform and Threat Sharing, is an open source software solution for collecting, storing, distributing and sharing cyber security indicators and threat about cyber security incidents analysis and malware analysis. MISP is designed by and for incident analysts, security and ICT professionals or malware reverser to support their day-to-day operations to share structured informations efficiently.

The objective of MISP is to foster the sharing of structured information within the security community and abroad. MISP provides functionalities to support the exchange of information but also the consumption of the information by Network Intrusion Detection System (NIDS), LIDS but also log analysis tools, SIEMs.

MISP, Malware Information Sharing Platform and Threat Sharing, core functionalities are:
– An efficient IOC and indicators database allowing to store technical and non-technical information about malware samples, incidents, attackers and intelligence.
– Automatic correlation finding relationships between attributes and indicators from malware, attacks campaigns or analysis.
– A flexible data model where complex objects can be expressed and linked together to express threat intelligence, incidents or connected elements.
– Built-in sharing functionality to ease data sharing using different model of distributions. MISP can synchronize automatically events and attributes among different MISP. Advanced filtering functionalities can be used to meet each organization sharing policy including a flexible sharing group capacity and an attribute level distribution mechanisms.
– An intuitive user-interface for end-users to create, update and collaborate on events and attributes/indicators. A graphical interface to navigate seamlessly between events and their correlations. Advanced filtering functionalities and warning list to help the analysts to contribute events and attributes.
storing data in a structured format (allowing automated use of the database for various purposes) with an extensive support of cyber security indicators along fraud indicators as in the financial sector.
export: generating IDS, OpenIOC, plain text, CSV, MISP XML or JSON output to integrate with other systems (network IDS, host IDS, custom tools), STIX (XML and JSON), NIDS export (Suricata, Snort and Bro) or RPZ zone. Many other formats easily added via the misp-modules.
import: bulk-import, batch-import, import from OpenIOC, GFI sandbox, ThreatConnect CSV. Many other formats easily added via the misp-modules.
– Flexible free text import tool to ease the integration of unstructured reports into MISP.
– A gentle system to collaborate on events and attributes allowing MISP users to propose changes or updates to attributes/indicators.
data-sharing: automatically exchange and synchronization with other parties and trust-groups using MISP.
delegating of sharing: allows a simple pseudo-anonymous mechanism to delegate publication of event/indicators to another organization.
– Flexible API to integrate MISP with your own solutions. MISP is bundled with PyMISP which is a flexible Python Library to fetch, add or update events attributes, handle malware samples or search for attributes.
Adjustable taxonomy to classify and tag events following your own classification schemes or existing classification. The taxonomy can be local to your MISP but also shareable among MISP instances.
Intelligence vocabularies called MISP galaxy and bundled with existing threat actors, malware, RAT, ransomware or MITRE ATT&CK which can be easily linked with events in MISP.
Expansion modules in Python to expand MISP with your own services or activate already available misp-modules.
Sighting support to get observations from organizations concerning shared indicators and attributes. Sighting can be contributed via MISP user-interface, API as MISP document or STIX sighting documents.
STIX support: export data in the STIX format (XML and JSON). Additional STIX import and export is supported by MISP-STIX-Converter or MISP-Taxii-Server.
Integrated encryption and signing of the notifications via PGP and/or S/MIME depending of the user preferences.

How to install MISP?
Environment: Ubuntu server 16-04.

1. Minimal Ubuntu install

Make sure your system is up2date:

sudo apt-get update
sudo apt-get upgrade

Install postfix, there will be some questions.

sudo apt-get install postfix

Postfix Configuration: Satellite system
Change the relay server later with:

sudo postconf -e 'relayhost = example.com'
sudo postfix reload

2. Install LAMP & dependencies

Once the system is installed you can perform the following steps:

Install the dependencies: (some might already be installed)

sudo apt-get install curl gcc git gnupg-agent make python openssl redis-server sudo vim zip

Install MariaDB (a MySQL fork/alternative)

sudo apt-get install mariadb-client mariadb-server

Secure the MariaDB installation (especially by setting a strong root password)

sudo mysql_secure_installation

Install Apache2

sudo apt-get install apache2 apache2-doc apache2-utils

Enable modules, settings, and default of SSL in Apache

sudo a2dismod status
sudo a2enmod ssl
sudo a2enmod rewrite
sudo a2dissite 000-default
sudo a2ensite default-ssl

Install PHP and dependencies

sudo apt-get install libapache2-mod-php php php-cli php-crypt-gpg php-dev php-json php-mysql php-opcache php-readline php-redis php-xml

Apply all changes

sudo systemctl restart apache2

3. MISP code
Download MISP using git in the /var/www/ directory.

sudo mkdir /var/www/MISP
sudo chown www-data:www-data /var/www/MISP
cd /var/www/MISP
sudo -u www-data git clone https://github.com/MISP/MISP.git /var/www/MISP
sudo -u www-data git checkout tags/$(git describe --tags `git rev-list --tags --max-count=1`)

if the last shortcut doesn’t work, specify the latest version manually

example: git checkout tags/v2.4.XY
the message regarding a "detached HEAD state" is expected behaviour
(you only have to create a new branch, if you want to change stuff and do a pull request for example)

Make git ignore filesystem permission differences

sudo -u www-data git config core.filemode false

install Mitre’s STIX and its dependencies by running the following commands:

sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev python-setuptools
cd /var/www/MISP/app/files/scripts
sudo -u www-data git clone https://github.com/CybOXProject/python-cybox.git
sudo -u www-data git clone https://github.com/STIXProject/python-stix.git
cd /var/www/MISP/app/files/scripts/python-cybox
sudo -u www-data git checkout v2.1.0.12
sudo python setup.py install
cd /var/www/MISP/app/files/scripts/python-stix
sudo -u www-data git checkout v1.1.1.4
sudo python setup.py install

install mixbox to accomodate the new STIX dependencies:

cd /var/www/MISP/app/files/scripts/
sudo -u www-data git clone https://github.com/CybOXProject/mixbox.git
cd /var/www/MISP/app/files/scripts/mixbox
sudo -u www-data git checkout v1.0.2
sudo python setup.py install

4. CakePHP
CakePHP is included as a submodule of MISP, execute the following commands to let git fetch it:

cd /var/www/MISP
sudo -u www-data git submodule init
sudo -u www-data git submodule update

Once done, install CakeResque along with its dependencies if you intend to use the built in background jobs:

cd /var/www/MISP/app
sudo -u www-data php composer.phar require kamisama/cake-resque:4.1.2
sudo -u www-data php composer.phar config vendor-dir Vendor
sudo -u www-data php composer.phar install

Enable CakeResque with php-redis

sudo phpenmod redis

To use the scheduler worker for scheduled tasks, do the following:

sudo -u www-data cp -fa /var/www/MISP/INSTALL/setup/config.php /var/www/MISP/app/Plugin/CakeResque/Config/config.php

5. Set the permissions

Check if the permissions are set correctly using the following commands:

sudo chown -R www-data:www-data /var/www/MISP
sudo chmod -R 750 /var/www/MISP
sudo chmod -R g+ws /var/www/MISP/app/tmp
sudo chmod -R g+ws /var/www/MISP/app/files
sudo chmod -R g+ws /var/www/MISP/app/files/scripts/tmp

6. Create a database and user
Enter the mysql shell

sudo mysql -u root -p
MariaDB [(none)]> create database misp;
MariaDB [(none)]> grant usage on *.* to [email protected] identified by 'XXXXdbpasswordhereXXXXX';
MariaDB [(none)]> grant all privileges on misp.* to [email protected];
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> exit

Import the empty MISP database from MYSQL.sql

sudo -u www-data sh -c "mysql -u misp -p misp </var/www/MISP/INSTALL/MYSQL.sql"

Enter the password you set previously

7. Apache configuration

Now configure your Apache webserver with the DocumentRoot /var/www/MISP/app/webroot/
If the apache version is 2.2:

sudo cp /var/www/MISP/INSTALL/apache.22.misp.ssl /etc/apache2/sites-available/misp-ssl.conf

If the apache version is 2.4:

sudo cp /var/www/MISP/INSTALL/apache.24.misp.ssl /etc/apache2/sites-available/misp-ssl.conf

Be aware that the configuration files for apache 2.4 and up have changed.
The configuration file has to have the .conf extension in the sites-available directory
For more information, visit http://httpd.apache.org/docs/2.4/upgrading.html

If a valid SSL certificate is not already created for the server, create a self-signed certificate:

sudo openssl req -newkey rsa:4096 -days 365 -nodes -x509 \
-subj "/C=/ST=/L=/O=/OU=/CN=/[email protected]" \
-keyout /etc/ssl/private/misp.local.key -out /etc/ssl/private/misp.local.crt

Otherwise, copy the SSLCertificateFile, SSLCertificateKeyFile, and SSLCertificateChainFile to /etc/ssl/private/. (Modify path and config to fit your environment)

============================================= Begin sample working SSL config for MISP
<VirtualHost :80>
        ServerName 

        Redirect permanent / https://

        LogLevel warn
        ErrorLog /var/log/apache2/misp.local_error.log
        CustomLog /var/log/apache2/misp.local_access.log combined
        ServerSignature Off


<VirtualHost :443>
        ServerAdmin [email protected]
        ServerName 
        DocumentRoot /var/www/MISP/app/webroot
        
                Options -Indexes
                AllowOverride all
                Order allow,deny
                allow from all
        

        SSLEngine On
        SSLCertificateFile /etc/ssl/private/misp.local.crt
        SSLCertificateKeyFile /etc/ssl/private/misp.local.key
#        SSLCertificateChainFile /etc/ssl/private/misp-chain.crt

        LogLevel warn
        ErrorLog /var/log/apache2/misp.local_error.log
        CustomLog /var/log/apache2/misp.local_access.log combined
        ServerSignature Off

============================================= End sample working SSL config for MISP

Activate new vhost

sudo a2dissite default-ssl
sudo a2ensite misp-ssl

Restart apache

sudo systemctl restart apache2

8. Log rotation
MISP saves the stdout and stderr of its workers in /var/www/MISP/app/tmp/logs
To rotate these logs install the supplied logrotate script:

sudo cp /var/www/MISP/INSTALL/misp.logrotate /etc/logrotate.d/misp

9. MISP configuration
There are 4 sample configuration files in /var/www/MISP/app/Config that need to be copied

sudo -u www-data cp -a /var/www/MISP/app/Config/bootstrap.default.php /var/www/MISP/app/Config/bootstrap.php
sudo -u www-data cp -a /var/www/MISP/app/Config/database.default.php /var/www/MISP/app/Config/database.php
sudo -u www-data cp -a /var/www/MISP/app/Config/core.default.php /var/www/MISP/app/Config/core.php
sudo -u www-data cp -a /var/www/MISP/app/Config/config.default.php /var/www/MISP/app/Config/config.php

Configure the fields in the newly created files:

sudo -u www-data vim /var/www/MISP/app/Config/database.php
# DATABASE_CONFIG has to be filled
# With the default values provided in section 6, this would look like:
# class DATABASE_CONFIG {
#   public $default = array(
#       'datasource' => 'Database/Mysql',
#       'persistent' => false,
#       'host' => 'localhost',
#       'login' => 'misp', // grant usage on *.* to [email protected]
#       'port' => 3306,
#       'password' => 'XXXXdbpasswordhereXXXXX', // identified by 'XXXXdbpasswordhereXXXXX';
#       'database' => 'misp', // create database misp;
#       'prefix' => '',
#       'encoding' => 'utf8',
#   );
#}

Important! Change the salt key in /var/www/MISP/app/Config/config.php
The salt key must be a string at least 32 bytes long.
The admin user account will be generated on the first login, make sure that the salt is changed before you create that user
If you forget to do this step, and you are still dealing with a fresh installation, just alter the salt,
delete the user from mysql and log in again using the default admin credentials ([email protected] / admin)

Change base url in config.php

sudo -u www-data vim /var/www/MISP/app/Config/config.php

Example: ‘baseurl’ => ‘https://’,
Alternatively, you can leave this field empty if you would like to use relative pathing in MISP
‘baseurl’ => ”,

And make sure the file permissions are still OK

sudo chown -R www-data:www-data /var/www/MISP/app/Config
sudo chmod -R 750 /var/www/MISP/app/Config

Generate a GPG encryption key.

sudo -u www-data mkdir /var/www/MISP/.gnupg
sudo chmod 700 /var/www/MISP/.gnupg
sudo -u www-data gpg --homedir /var/www/MISP/.gnupg --gen-key

The email address should match the one set in the config.php / set in the configuration menu in the administration menu configuration file
And export the public key to the webroot

sudo -u www-data sh -c "gpg --homedir /var/www/MISP/.gnupg --export --armor YOUR-KEYS-EMAIL-HERE > /var/www/MISP/app/webroot/gpg.asc"

To make the background workers start on boot

sudo chmod +x /var/www/MISP/app/Console/worker/start.sh
sudo vim /etc/rc.local

Add the following line before the last line (exit 0). Make sure that you replace www-data with your apache user:

sudo -u www-data bash /var/www/MISP/app/Console/worker/start.sh

Now log in using the webinterface:
The default user/pass = [email protected]/admin

If any of the directories that MISP uses to store files is not writeable to the apache user, change the permissions you can do this by running the following commands:

sudo chmod -R 750 /var/www/MISP/
sudo chown -R www-data:www-data /var/www/MISP/

Make sure that the STIX libraries and GnuPG work as intended, if not, refer to INSTALL.txt’s paragraphs dealing with these two items
If anything goes wrong, make sure that you check MISP’s logs for errors:

/var/www/MISP/app/tmp/logs/error.log
/var/www/MISP/app/tmp/logs/resque-worker-error.log
/var/www/MISP/app/tmp/logs/resque-scheduler-error.log
/var/www/MISP/app/tmp/logs/resque-2015-01-01.log // where the actual date is the current date

Optional features
MISP has a new pub/sub feature, using ZeroMQ. To enable it, simply run the following command

sudo pip install pyzmq

ZeroMQ depends on the Python client for Redis

sudo pip install redis

MISP interface
Login interface MISP:
The default user/pass = [email protected]/admin

Main interface:

View event:

You can comment in event:

View Correlation graph:

Export event:
You can export with format Json, XML, Snort,….

And more. MISP is a platform with a lot of features. Learn about those features to better understand MISP. You can read more about the MISP manual here.

References: https://github.com/MISP/MISP

Leave a Reply