Port fully connected operator

[Xreki]

resolve #3926

• fix SegFault during checking gradient
  • In python, when we define an Operator, all the inputs and outputs, including those AsIntermediate outputs, should be explicitly specified. In TestFCGradOp, operator should be defined as:

     op = Operator(
                   "fc",
                   X="X",
                   W="W",
                   b="b",
                   Out="Out",
                   mul_out="mul_out",
                   add_out="add_out",
                   activation="sigmoid")

• support multiple inputs and weights

  • Do not try to call operators with inplace arguments, such as Y = X + Y. In my first implementation, I construct FCOp as a series of operators as follows:

    1. mul_out = X[0] * W[0]
    2. add_out = X[1] * W[1]
    3. mul_out = mul_out + add_out
    4. add_out = add_out + b
    5. Y = Act(add_out)

  • The unittest failed when constructed an add operator during the backward process. I found two add operators were automatically created and the DebugString are:

Op(add), inputs:{X[mul_out@GRAD@RENAME@1@0, mul_out@GRAD@RENAME@1@1], outputs:{Out[mul_out@GRAD]}}
Op(add), inputs:{X[add_out@GRAD@RENAME@1@0, add_out@GRAD@RENAME@1@1], outputs:{Out[add_out@GRAD]}}

Thus I changed the implementation of FCOp to:
1. mul_out[0] = X[0] * W[0]
2. mul_out[1] = X[1] * W[1]
3. sum_out = mul_out[0] + mul_out[1]
4. add_out = sum_out + b
5. Y = Act(add_out)

• support input tensor of multiple dimensions
  • The attributes of type std::vector cannot be set default value, see How to set the default value of a std::vector<int> attribute #4077 - solved.

[reyoung]

Excellent work except some name conventions.

[reyoung]

identity operator is not needed by fc. We could just remove SumOut, like

outputs_["SumOut"].clear();

But it could be done in another PR.

[Xreki]

I have tried such kind of implementation, "renaming SumOut to EMPTY and using mul_out[0] instead", but so sad to meet some problem. I will try again.

[Xreki]

It works. I think there may be some error in my previous implementation.

[reyoung]

Please reference https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/name_convention.md

It seems that our Output prefer to Out.

[Xreki]

I notice that. But I prefer to use Y 😞 Whatever, I will rename it.

[qingqing01]

Very good comments.

[Xreki]

Thanks.

[luotao1]

179-183行不用加了，可参考 #4119

[Xreki]

Done.

[jacquesqiao]

the following code only use x.size() and w.size()
so here can use

size_t x_size = Inputs("X").size();
size_t w_size = Inputs("W").size();

[Xreki]

X and W are duplicatable inputs, and x[i] and w[i] are used in line 73 which creates mul operators. w.size() is only used in the PADDLE_ENFORCE_EQ statement in line 39.