Tensor: An Unified Data Type in PaddlePaddle

[gangliao]

View

I already integrated a lightweight variant into our project, so we can exclude boost now.

It's better to maintain a new repository https://github.com/gangliao/Tensor , because we don't want PaddlePaddle Review Process and Messaging to block our progress. We want more focused on refactoring.

In Tensor repository, I added variant, runnable CMake and unit test framework to accelerate our development progress.

[QiJune]

I think maybe we should take Expression templates into consideration, which could reduce some temp memory allocation. And paddle has a implementation already.

[gangliao]

It looks like pretty useful in linear operations.

[QiJune]

Does Majel introduces other libraries from boost? Let's make a check

[gangliao]

$ for i in $(du -a | grep '\.h$' | cut -f 2); do \
     grep 'boost::' $i; \
  done | \
 grep -o 'boost::[a-zA-Z_]*' | sort | uniq
boost::apply_visitor
boost::bad_get
boost::get
boost::python
boost::shared_ptr
boost::static_visitor
boost::variant

[wangkuiyi]

This is one of the best design doc I've read since I joined PaddlePaddle project. It states the pain point and gradually derives a solution, which is justifiably optimal.

[wangkuiyi]

standard C++ classes ==> C++ class templates

[gangliao]

done

[wangkuiyi]

Thanks for this example of boost:variant.

I noticed an additional interesting thing about variant -- the default value of a varaint typed variable is the first template argument. For example, if Place is defined as

typedef boost::variant<GpuPlace, CpuPlace> Place;

then

Place p;
std::out << p;

prints "GpuPlace(0)".

However, if we reverse the order of GpuPlace and CpuPlace in the definition of Place like

typedef boost:variant<CpuPlace, GpuPlace> Place;

then above printing code would prints "CpuPlace".

[gangliao]

Why Majel did not write as typedef boost:variant<CpuPlace, GpuPlace> Place; If do so, it will default allocate the memory in CPU place.

[wangkuiyi]

A few questions:

1. If we work on https://github.com/gangliao/Tensor, how could we do CI on GPU? Or, would it be more convenient to work on a new directory, say majel, in https://github.com/PaddlePaddle/Paddle/, so could we use the existing CI setup?

2. I am afraid that the name of the repo or directory cannot be Tensor, but should be math, or an abstract name like majel, for our respect to the original authors who gave us a lot of inspirations.

3. boost::varaint is not the only boost package that Majel depends on. I listed all of them:

   $ for i in $(du -a | grep '\.h$' | cut -f 2); do \
        grep 'boost::' $i; \
     done | \
    grep -o 'boost::[a-zA-Z_]*' | sort | uniq
   boost::apply_visitor
   boost::bad_get
   boost::get
   boost::python
   boost::shared_ptr
   boost::static_visitor
   boost::variant

   I am afraid that we cannot reimplement them all. I personally prefer to use boost.

4. We need to be able to build the new work using our standard development procedure, i.e., using the Docker development image. I did some changes with the current PaddlePaddle's Dockerfile here. This might be useful as a reference.

[gangliao]

If we work on https://github.com/gangliao/Tensor, how could we do CI on GPU? Or, would it be more convenient to work on a new directory, say majel, in https://github.com/PaddlePaddle/Paddle/, so could we use the existing CI setup?

Yeah, It's more convenient to work on paddle/majel.

I am afraid that the name of the repo or directory cannot be Tensor, but should be math, or an abstract name like majel, for our respect to the original authors who gave us a lot of inspirations.

I strongly agree with you. we should reserve the namemajel.

boost::varaint is not the only boost package that Majel depends on. I listed all of them:
I am afraid that we cannot reimplement them all. I personally prefer to use boost.

I tested all features of variant in Majel using lightweight header files and it works well.
https://github.com/gangliao/Tensor/tree/develop/test/variant

If add boost, everyone needs to install it and we can even throw c++11 away. It's definitely ok to use the lightweight implementation. If you check majel/contrib/cuda_variant/cuda_variant.h, they also use Bryan Catanzaro's cuda_variant to achieve the same goal for GPU.

We need to be able to build the new work using our standard development procedure, i.e., using the Docker development image. I did some changes with the current PaddlePaddle's Dockerfile here. This might be useful as a reference.

Yeah, I will review, approve and replace makefile to CMake.

[helinwang]

Very good!