We build a CLIP-based Activity2Vec model to use the power of CLIP. Based on the CLIP pre-trained model, we finetune it on our HAKE data with the human body part state (PaSta) labels. Thus, this model would estimate all the existing PaSta within the whole image (like the visualizations below). We believe it would be useful for action understanding as a more powerful image action semantic extractor. If you have any advice, feel free to drop us an email!

1. install PyTorch 1.5+ and torchvision 0.6+:

conda install -c pytorch pytorch torchvision

2. clone this repository

git clone https://github.com/DirtyHarryLYL/HAKE-Action-Torch.git PaStaNetCLIP
cd PaStaNetCLIP
git checkout CLIP-Activity2Vec

1. Download HAKE dataset from here

tar xzvf hake-large.tgz 

2. Organize your data as follows

PaStaNetCLIP
|_ data
    |_ hake
      |_ hake-large
      |  |_ hico-train
      |  |_ hico-test
      |  |_ ...

3. Download annotations and put them under $PROJECT/data/hake folder.

4. Download the CLIP pretrained model from Baidu Pan and put it in $PROJECT/pretrained/clip.

Or Google Drive (ViT-B-16.new.pt: original CLIP pre-trianed model, ckpt_4.pth: finetuned model with HAKE data).

Backbone: ViT-B/16

data split: train/val

model weights: link

  # by default, we use gpu x batch : 8x4
  # you can use --batch_size and --nproc_per_node to adjust the batch size and number of GPUS. 
  ./run.sh

 # set --vis_test to visualize the predicition
 ./test.sh

If you find our works useful, please consider citing:

@inproceedings{li2020pastanet,
  title={PaStaNet: Toward Human Activity Knowledge Engine},
  author={Li, Yong-Lu and Xu, Liang and Liu, Xinpeng and Huang, Xijie and Xu, Yue and Wang, Shiyi and Fang, Hao-Shu and Ma, Ze and Chen, Mingyang and Lu, Cewu},
  booktitle={CVPR},
  year={2020}
}