shadow-plugin-tor wiki

This wiki explains how to set up and use the Shadow plug-in that runs Tor. It can be used to run a private Tor anonymity network on a single machine using the Shadow discrete-event network simulator. For more information about Shadow, see https://shadow.github.io and https://github.com/shadow.

This plug-in is most useful in conjunction with the traffic generator plug-in included with Shadow to transfer data across private anonymity networks and measure performance characteristics.

Dependencies

After installing Shadow and its dependencies:

YUM (Fedora):

sudo yum install -y gcc automake autoconf zlib zlib-devel

APT (Ubuntu):

sudo apt-get -y install gcc automake autoconf zlib1g-dev

Setup and Install

Get shadow-plugin-tor

git clone https://github.com/shadow/shadow-plugin-tor.git -b release

Quick auto setup

We need to setup openssl and libevent with custom configuration options in order to run Tor in Shadow. The setup script should handle this for you (run ./setup --help for more details).

cd shadow-plugin-tor
./setup dependencies
./setup build
./setup install

Important build options (see ./setup build --help for full list):

• --tor-prefix to build with your local custom Tor distribution (instead of downloading one from torproject.org).
• --openssl-prefix and --libevent-prefix to specify custom library locations

Detailed manual setup

The manual setup process is not necessary if the auto setup completed successfully.

Build openssl and libevent

Download the libraries and run:

cd openssl
./config --prefix=/home/${USER}/.shadow shared threads enable-ec_nistp_64_gcc_128 -fPIC
make
make install_sw
cd ../libevent
./configure --prefix=/home/${USER}/.shadow --enable-shared CFLAGS="-fPIC -I/home/${USER}/.shadow" LDFLAGS="-L/home/${USER}/.shadow"
make
make install

NOTE: if you want to use valgrind and don't want it to report spurious openssl errors, add this to the end of the openssl configure line: -g -pg -DPURIFY -Bsymbolic

Get and configure tor

We need to get the tor source so we can compile it into our Shadow plug-in, and then configure it to obtain a proper orconfig.h file.

cd shadow-plugin-tor
mkdir build
cd build
git clone --branch release-0.2.7 https://git.torproject.org/tor.git
cd tor
./autogen.sh
./configure --disable-transparent --disable-asciidoc CFLAGS="-fPIC -fno-inline" --with-libevent-dir=`readlink -f ~`/.shadow --with-openssl-dir=`readlink -f ~`/.shadow
make -jN
cd ..

Replace N with the number of cores you want to use for a parallel build.

Note that --with-openssl-dir and --with-libevent-dir should specify the install path of the custom built OpenSSL and Libevent.

build plugin using cmake

mkdir main
cd main
CC=`which clang` CXX=`which clang++` cmake ../.. -DTOR_VERSION_A=0 -DTOR_VERSION_B=2 -DTOR_VERSION_C=7 -DTOR_VERSION_D=5
make -jN
make install

Replace N with the number of cores you want to use for a parallel build.

cmake troubleshooting

First try rebuilding to ensure that the cmake cache is up to date

rm -rf build
mkdir build
cd build

using VERBOSE=1 for more verbose output

CC=`which clang` CXX=`which clang++` cmake ..
VERBOSE=1 make

Configuring Shadow to Run the Plug-in

shadow-plugin-tor allows us to configure a private Tor networks on our machine and transfer data through it. The plug-in requires a library to be preloaded with LD_PRELOAD and also requires setting some extra environment variables. Because of these complexities, a python helper script called shadow-tor is also installed in ~/.shadow/bin (or your/prefix/bin). This script is a wrapper for the shadow program.

Example experiment configurations can be found in the shadow-plugin-tor/resource directory. Please untar the shadowtor-minimal-config.tar.xz example to understand how to configure your nodes to run Tor.

cd resource
tar xaf shadowtor-minimal-config.tar.xz
mv shadowtor-minimal-config shadowtor-minimal

Your shadow.config.xml file must specify that the Tor plug-in should be loaded, and that a should run an instance of the Tor plugin as an application, e.g.:

<plugin id="tor" path="~/.shadow/plugins/libshadow-plugin-tor.so" />
<node id="relay">
  <application plugin="tor" arguments="[...]"  [...] />
</>

The arguments attribute of the application XML element specifies application arguments for configuring a node's instance of the plug-in. The arguments will be passed directly to Tor, and so all options as specified in the Tor config manual are allowed.

Running Tor in Shadow

The minimal example can be run as follows:

# the minimal configuration was already extracted above
cd shadowtor-minimal
# run the simulation
shadow-tor -i shadow.config.xml

If this completes successfully (after about a minute or so), we should find that 40 GET requests have completed:

for d in shadow.data/hosts/*client*; do grep "transfer-complete" ${d}/* ; done | wc -l

The raw transfer message lines also contain more useful transfer statistics.

Notes:

• Each Tor node may be configured using the *torrc files and in the arguments attribute in the shadow.config.xml file
• The shadow.data/hosts/ directory contains a private data directory for each host running in the experiment
• All Tor output is redirected to a log file in the data directory for the host that generated it
• All Shadow logging for every node is redirected to a single shadow.log file
• If dstat is installed, its output is redirected to dstat.log

The above minimal example is mostly useful for debugging. A slightly larger toy example is also available in the resource/ directory.

Tor Experimentation and Analysis

Suppose we want to test the performance difference between 2 of Tor's schedulers. We could do the following to setup our experiments:

cd resource
tar xaf shadowtor-toy-config.tar.xz
cp -R shadowtor-toy-config vanilla
cp -R shadowtor-toy-config priority

At this point you should add the string CircuitPriorityHalflife 30 to the end of the *.torrc file located in the priority directory. This will enable on all nodes the scheduler that prioritizes circuits based on exponentially-weighted moving average circuit throughputs.

Now you can run both experiments and plot the results: (NOTE: each experiment may take a few hours to run, so be patient)

cd vanilla
shadow-tor -y -w 3
cd ../priority
shadow-tor -y -w 3
cd ../

Since the shadow-tor script redirects log messages to shadow.log, the following commands can be used to parse and plot those results:

cd vanilla
python ~/shadow/tools/parse-shadow.py --prefix vanilla-results shadow.log
python ~/shadow/tools/parse-tgen.py --prefix vanilla-results shadow.data/hosts
cd ../priority
python ~/shadow/tools/parse-shadow.py --prefix priority-results shadow.log
python ~/shadow/tools/parse-tgen.py --prefix priority-results shadow.data/hosts
cd ..
python ~/shadow/tools/plot-shadow.py --prefix "scheduler" --data vanilla-results/ "vanilla" --data priority-results/ "priority"

Visualize some statistics by opening the scheduler-shadow.pdf file.

Scalability

TODO : This section is out of date and needs to be removed, since these configurations are no longer distributed with shadow.

The sample configurations in resource will take on the order of 30 minutes to several hours, and consume ~4 to ~64 GiB of RAM, depending on the number of nodes running and the Tor software version number. An estimate of the resources required by the sample configurations are given below. Also included is the smallest possible EC2 instance required to run each sample configurations, as a convenience for selecting an EC2 instance type for Running Shadow on EC2.

* For the large configuration, you might run up against open file limits (messages from scallion: Couldn't open ... for locking: Too many open files and Acting on config options left us in a broken state. Dying.). Increasing the limit (e.g., with ulimit -n) to 16384 resolves the issue, but lower limits might work, too.

Generating a new Tor Network

If you don't want to use the pre-generated examples in shadow-plugin-tor/resource or you want to customize the network, you can generate your own. The following process assumes you have an Internet connection and that the Shadow base directory exists in your home directory at ~/.

Prepare Alexa website data

The following produces alexa-top-1000-ips.csv: a csv list of the top 1000 servers.

wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
unzip top-1m.csv.zip
python ~/shadow-plugin-tor/tools/parsealexa.py

Prepare Tor metrics data

This process requires lots of data and metrics from Tor. Be prepared to wait for the following downloads to complete.

NOTE: you may want to modify the URLs below to grab more up-to-date metrics data.

wget https://collector.torproject.org/archive/relay-descriptors/server-descriptors/server-descriptors-2013-04.tar.xz
tar xaf server-descriptors-2013-04.tar.xz
wget https://collector.torproject.org/archive/relay-descriptors/extra-infos/extra-infos-2013-04.tar.xz
tar xaf extra-infos-2013-04.tar.xz
wget https://collector.torproject.org/archive/relay-descriptors/consensuses/consensuses-2013-04.tar.xz
tar xaf consensuses-2013-04.tar.xz
wget https://metrics.torproject.org/stats/clients.csv

Start generating!

NOTES:

• You'll need to use one of the consensus files from the consensuses-2013-04 directory in the following steps.
• Make sure you have already build shadow-plugin-tor (using './setup build') because this process requires the tor and tor-gencert binaries to complete successfully.

export PATH=${PATH}:~/shadow-plugin-tor/build/tor/src/or:~/shadow-plugin-tor/build/tor/src/tools
mkdir mytor
cd mytor
python ~/shadow-plugin-tor/tools/generate.py --help
python ~/shadow-plugin-tor/tools/generate.py --nauths 1 --nrelays 20 --nclients 200 --nservers 20 --fweb 0.90 --fbulk 0.10 --nperf50k 10 --nperf1m 10 --nperf5m 10 ../alexa-top-1000-ips.csv ../2013-04-30-23-00-00-consensus ../server-descriptors-2013-04/ ../extra-infos-2013-04/ ../clients.csv

If everything went smoothly, shadow-tor can be run from inside the 'mytor' directory as usual.