CLIP-Driven Referring Image Segmentation

Overview

Vision Language Decoder - Transfer fine grained semantic information from text to pixel level features(patches) - Self-Attention: for long range dependencies - Cross-Attention: for fine structured textual features to pixel level features Text To Pixel contrastive Learning - Explicitly enforce the textual similarity - Align language features with corresponding pixel features

Architecture

![[Pasted image 20240106103205.png]]

Feature-Extraction

• Image Encoder: Res-net
  • Use of 2nd-4th stage features
• Text Encoder: Standard encoder
  • $F_s$: [CLS] token
• Cross Modal Neck: incorporate the textual embedding into image features
• Use of [[CoordConv]]

Vision-Language-Decoder

Takes input $F_v\in R^{N\times C}$ and $F_t\in R^{L\times C}$ generates multi-model features $F_c\in R^{N\times C}$. To capture positional information, sine spacial positional embeddings are added. - Multi head self attention \($F_v^{\prime }=MHSA(LN(F_v))+F_v$\) - Multi head cross attention $$F_c^{\prime }=MHCA(LN(F_v^{\prime }),F_t)+F_v^{\prime }$$ - MLP block $$F_c=MLP(LN(F_c^{\prime }))+F_c^{\prime }$$

Contrastive-Learning

We get a $F_c\in R^{N\times C}$ from vision language decoder and $F_s\in R^{C^{{}^{\prime }}}$ from text feature extractor. We transform this by $$z_v=F_c^{\prime }{W}_v+b_v,F_c^{\prime }=Up(F_c)$$\($z_t=F_s{W}_t+b_t,$\) - $z_t\in R^D$, $z_v\in R^{N\times D}$ , $N=H/4\times W/4$ - Up denotes 4× up-sampling - Then it is trained constrastive style