AI 助理
备案控制台
开发者社区 人工智能 文章 正文

聊一聊计算机视觉中常用的注意力机制以及Pytorch代码实现

2024-10-18 4
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介： 本文介绍了几种常用的计算机视觉注意力机制及其PyTorch实现，包括SENet、CBAM、BAM、ECA-Net、SA-Net、Polarized Self-Attention、Spatial Group-wise Enhance和Coordinate Attention等，每种方法都附有详细的网络结构说明和实验结果分析。通过这些注意力机制的应用，可以有效提升模型在目标检测任务上的性能。此外，作者还提供了实验数据集的基本情况及baseline模型的选择与实验结果，方便读者理解和复现。

聊一聊计算机视觉中常用的注意力机制以及Pytorch代码实现

注意力机制（Attention）是深度学习中常用的tricks，可以在模型原有的基础上直接插入，进一步增强你模型的性能。注意力机制起初是作为自然语言处理中的工作Attention Is All You Need被大家所熟知，从而也引发了一系列的XX is All You Need的论文命题，SENET-Squeeze-and-Excitation Networks是注意力机制在计算机视觉中应用的早期工作之一，并获得了2017年imagenet， 同时也是最后一届Imagenet比赛的冠军，后面就又出现了各种各样的注意力机制，应用在计算机视觉的任务中，今天我们就来一起聊一聊计算机视觉中常用的注意力机制以及他们对应的Pytorch代码实现，另外我还使用这些注意力机制做了一些目标检测的实验，实验效果我也一并放在博客中，大家可以一起对自己感兴趣的部分讨论讨论。

新出的手把手教程，感兴趣的兄弟们快去自己动手试试看！
手把手教你使用YOLOV5训练自己的目标检测模型-口罩检测-视频教程_dejahu的博客-CSDN博客

这里是我数据集的基本情况，这里我使用的是交通标志检测的数据集

CocoDataset Train dataset with number of images 2226, and instance counts: 
+------------+-------+-----------+-------+-----------+-------+-----------------------------+-------+---------------------+-------+
| category   | count | category  | count | category  | count | category                    | count | category            | count |
+------------+-------+-----------+-------+-----------+-------+-----------------------------+-------+---------------------+-------+
| 0 [red_tl] | 1465  | 1 [arr_s] | 1133  | 2 [arr_l] | 638   | 3 [no_driving_mark_allsort] | 622   | 4 [no_parking_mark] | 1142  |
+------------+-------+-----------+-------+-----------+-------+-----------------------------+-------+---------------------+-------+

baseline选择的是fasterrcnn，实验的结果如下：

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.341
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.502
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.400
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.115
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.473
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.655
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.417
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.417
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.417
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.156
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.570
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.726

如果大家遇到论文下载比较慢

推荐使用中科院的 arxiv 镜像: http://xxx.itp.ac.cn, 国内网络能流畅访问
简单直接的方法是, 把要访问 arxiv 链接中的域名从 https://arxiv.org 换成 http://xxx.itp.ac.cn , 比如:

从 https://arxiv.org/abs/1901.07249 改为 http://xxx.itp.ac.cn/abs/1901.07249

1. SeNet: Squeeze-and-Excitation Attention

论文地址：https://arxiv.org/abs/1709.01507

  • 网络结构

    对通道做注意力机制，通过全连接层对每个通道进行加权。

    image-20211210153628963

    image-20211210153655899

  • Pytorch代码

    ```python
    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init

class SEAttention(nn.Module):

  def __init__(self, channel=512, reduction=16):
      super().__init__()
      self.avg_pool = nn.AdaptiveAvgPool2d(1)
      self.fc = nn.Sequential(
          nn.Linear(channel, channel // reduction, bias=False),
          nn.ReLU(inplace=True),
          nn.Linear(channel // reduction, channel, bias=False),
          nn.Sigmoid()
      )

  def init_weights(self):
      for m in self.modules():
          if isinstance(m, nn.Conv2d):
              init.kaiming_normal_(m.weight, mode='fan_out')
              if m.bias is not None:
                  init.constant_(m.bias, 0)
          elif isinstance(m, nn.BatchNorm2d):
              init.constant_(m.weight, 1)
              init.constant_(m.bias, 0)
          elif isinstance(m, nn.Linear):
              init.normal_(m.weight, std=0.001)
              if m.bias is not None:
                  init.constant_(m.bias, 0)

  def forward(self, x):
      b, c, _, _ = x.size()
      y = self.avg_pool(x).view(b, c)
      y = self.fc(y).view(b, c, 1, 1)
      return x * y.expand_as(x)

if name == 'main':
input = torch.randn(50, 512, 7, 7)
se = SEAttention(channel=512, reduction=8)
output = se(input)
print(output.shape)


* 实验结果

  ```bash
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.338
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.511
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.375
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.126
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.458
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.696
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.411
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.411
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.411
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.163
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.551
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.758

2. （有用）CBAM: Convolutional Block Attention Module

论文地址：CBAM: Convolutional Block Attention Module

  • 网络结构

    对通道方向上做注意力机制之后再对空间方向上做注意力机制

    image-20211210154323132

  • Pytorch代码

    ```python
    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init

class ChannelAttention(nn.Module):
def init(self, channel, reduction=16):
super().init()
self.maxpool = nn.AdaptiveMaxPool2d(1)
self.avgpool = nn.AdaptiveAvgPool2d(1)
self.se = nn.Sequential(
nn.Conv2d(channel, channel // reduction, 1, bias=False),
nn.ReLU(),
nn.Conv2d(channel // reduction, channel, 1, bias=False)
)
self.sigmoid = nn.Sigmoid()

  def forward(self, x):
      max_result = self.maxpool(x)
      avg_result = self.avgpool(x)
      max_out = self.se(max_result)
      avg_out = self.se(avg_result)
      output = self.sigmoid(max_out + avg_out)
      return output

class SpatialAttention(nn.Module):
def init(self, kernel_size=7):
super().init()
self.conv = nn.Conv2d(2, 1, kernel_size=kernel_size, padding=kernel_size // 2)
self.sigmoid = nn.Sigmoid()

  def forward(self, x):
      max_result, _ = torch.max(x, dim=1, keepdim=True)
      avg_result = torch.mean(x, dim=1, keepdim=True)
      result = torch.cat([max_result, avg_result], 1)
      output = self.conv(result)
      output = self.sigmoid(output)
      return output

class CBAMBlock(nn.Module):

  def __init__(self, channel=512, reduction=16, kernel_size=49):
      super().__init__()
      self.ca = ChannelAttention(channel=channel, reduction=reduction)
      self.sa = SpatialAttention(kernel_size=kernel_size)

  def init_weights(self):
      for m in self.modules():
          if isinstance(m, nn.Conv2d):
              init.kaiming_normal_(m.weight, mode='fan_out')
              if m.bias is not None:
                  init.constant_(m.bias, 0)
          elif isinstance(m, nn.BatchNorm2d):
              init.constant_(m.weight, 1)
              init.constant_(m.bias, 0)
          elif isinstance(m, nn.Linear):
              init.normal_(m.weight, std=0.001)
              if m.bias is not None:
                  init.constant_(m.bias, 0)

  def forward(self, x):
      b, c, _, _ = x.size()
      residual = x
      out = x * self.ca(x)
      out = out * self.sa(out)
      return out + residual

if name == 'main':
input = torch.randn(50, 512, 7, 7)
kernel_size = input.shape[2]
cbam = CBAMBlock(channel=512, reduction=16, kernel_size=kernel_size)
output = cbam(input)
print(output.shape)


* 实验结果

  ```bash
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.364
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.544
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.425
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.137
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.499
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.674
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.439
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.439
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.439
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.185
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.590
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.755

3. BAM: Bottleneck Attention Module

论文地址：https://arxiv.org/pdf/1807.06514.pdf

  • 网络结构

    image-20211217103020590

  • Pytorch代码

    ```python
    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init

class Flatten(nn.Module):
def forward(self, x):
return x.view(x.shape[0], -1)

class ChannelAttention(nn.Module):
def init(self, channel, reduction=16, num_layers=3):
super().init()
self.avgpool = nn.AdaptiveAvgPool2d(1)
gate_channels = [channel]
gate_channels += [channel // reduction] * num_layers
gate_channels += [channel]

      self.ca = nn.Sequential()
      self.ca.add_module('flatten', Flatten())
      for i in range(len(gate_channels) - 2):
          self.ca.add_module('fc%d' % i, nn.Linear(gate_channels[i], gate_channels[i + 1]))
          self.ca.add_module('bn%d' % i, nn.BatchNorm1d(gate_channels[i + 1]))
          self.ca.add_module('relu%d' % i, nn.ReLU())
      self.ca.add_module('last_fc', nn.Linear(gate_channels[-2], gate_channels[-1]))

  def forward(self, x):
      res = self.avgpool(x)
      res = self.ca(res)
      res = res.unsqueeze(-1).unsqueeze(-1).expand_as(x)
      return res

class SpatialAttention(nn.Module):
def init(self, channel, reduction=16, num_layers=3, dia_val=2):
super().init()
self.sa = nn.Sequential()
self.sa.add_module('conv_reduce1',
nn.Conv2d(kernel_size=1, in_channels=channel, out_channels=channel // reduction))
self.sa.add_module('bn_reduce1', nn.BatchNorm2d(channel // reduction))
self.sa.add_module('relu_reduce1', nn.ReLU())
for i in range(num_layers):
self.sa.addmodule('conv%d' % i, nn.Conv2d(kernel_size=3, in_channels=channel // reduction,
out_channels=channel // reduction, padding=1, dilation=dia_val))
self.sa.addmodule('bn%d' % i, nn.BatchNorm2d(channel // reduction))
self.sa.addmodule('relu%d' % i, nn.ReLU())
self.sa.add_module('last_conv', nn.Conv2d(channel // reduction, 1, kernel_size=1))

  def forward(self, x):
      res = self.sa(x)
      res = res.expand_as(x)
      return res

class BAMBlock(nn.Module):

  def __init__(self, channel=512, reduction=16, dia_val=2):
      super().__init__()
      self.ca = ChannelAttention(channel=channel, reduction=reduction)
      self.sa = SpatialAttention(channel=channel, reduction=reduction, dia_val=dia_val)
      self.sigmoid = nn.Sigmoid()

  def init_weights(self):
      for m in self.modules():
          if isinstance(m, nn.Conv2d):
              init.kaiming_normal_(m.weight, mode='fan_out')
              if m.bias is not None:
                  init.constant_(m.bias, 0)
          elif isinstance(m, nn.BatchNorm2d):
              init.constant_(m.weight, 1)
              init.constant_(m.bias, 0)
          elif isinstance(m, nn.Linear):
              init.normal_(m.weight, std=0.001)
              if m.bias is not None:
                  init.constant_(m.bias, 0)

  def forward(self, x):
      b, c, _, _ = x.size()
      sa_out = self.sa(x)
      ca_out = self.ca(x)
      weight = self.sigmoid(sa_out + ca_out)
      out = (1 + weight) * x
      return out

if name == 'main':
input = torch.randn(50, 512, 7, 7)
bam = BAMBlock(channel=512, reduction=16, dia_val=2)
output = bam(input)
print(output.shape)




* 实验结果

无


## 4. （有用）ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

> 论文地址：https://arxiv.org/pdf/1910.03151.pdf

* 网络结构

  ![image-20211217105517345](https://i-blog.csdnimg.cn/blog_migrate/70c7234711b60959b0aa394f7bdb08a8.png)

* Pytorch代码

  ```python
  import numpy as np
  import torch
  from torch import nn
  from torch.nn import init
  from collections import OrderedDict


  class ECAAttention(nn.Module):

      def __init__(self, kernel_size=3):
          super().__init__()
          self.gap = nn.AdaptiveAvgPool2d(1)
          self.conv = nn.Conv1d(1, 1, kernel_size=kernel_size, padding=(kernel_size - 1) // 2)
          self.sigmoid = nn.Sigmoid()

      def init_weights(self):
          for m in self.modules():
              if isinstance(m, nn.Conv2d):
                  init.kaiming_normal_(m.weight, mode='fan_out')
                  if m.bias is not None:
                      init.constant_(m.bias, 0)
              elif isinstance(m, nn.BatchNorm2d):
                  init.constant_(m.weight, 1)
                  init.constant_(m.bias, 0)
              elif isinstance(m, nn.Linear):
                  init.normal_(m.weight, std=0.001)
                  if m.bias is not None:
                      init.constant_(m.bias, 0)

      def forward(self, x):
          y = self.gap(x)  # bs,c,1,1
          y = y.squeeze(-1).permute(0, 2, 1)  # bs,1,c
          y = self.conv(y)  # bs,1,c
          y = self.sigmoid(y)  # bs,1,c
          y = y.permute(0, 2, 1).unsqueeze(-1)  # bs,c,1,1
          return x * y.expand_as(x)


  if __name__ == '__main__':
      input = torch.randn(50, 512, 7, 7)
      eca = ECAAttention(kernel_size=3)
      output = eca(input)
      print(output.shape)

  • 实验结果

    2021-12-17 12:18:08,911 - mmdet - INFO - 
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.360
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.545
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.414
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.141
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.489
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.676
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.432
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.432
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.432
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.184
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.576
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.748

5. SA-NET: SHUFFLE ATTENTION FOR DEEP CONVOLUTIONAL NEURAL NETWORKS

论文地址：https://arxiv.org/pdf/2102.00240.pdf

  • 网络结构

    image-20211217105138064

  • Pytorch代码

    ```python
    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    from torch.nn.parameter import Parameter

class ShuffleAttention(nn.Module):

  def __init__(self, channel=512, reduction=16, G=8):
      super().__init__()
      self.G = G
      self.channel = channel
      self.avg_pool = nn.AdaptiveAvgPool2d(1)
      self.gn = nn.GroupNorm(channel // (2 * G), channel // (2 * G))
      self.cweight = Parameter(torch.zeros(1, channel // (2 * G), 1, 1))
      self.cbias = Parameter(torch.ones(1, channel // (2 * G), 1, 1))
      self.sweight = Parameter(torch.zeros(1, channel // (2 * G), 1, 1))
      self.sbias = Parameter(torch.ones(1, channel // (2 * G), 1, 1))
      self.sigmoid = nn.Sigmoid()

  def init_weights(self):
      for m in self.modules():
          if isinstance(m, nn.Conv2d):
              init.kaiming_normal_(m.weight, mode='fan_out')
              if m.bias is not None:
                  init.constant_(m.bias, 0)
          elif isinstance(m, nn.BatchNorm2d):
              init.constant_(m.weight, 1)
              init.constant_(m.bias, 0)
          elif isinstance(m, nn.Linear):
              init.normal_(m.weight, std=0.001)
              if m.bias is not None:
                  init.constant_(m.bias, 0)

  @staticmethod
  def channel_shuffle(x, groups):
      b, c, h, w = x.shape
      x = x.reshape(b, groups, -1, h, w)
      x = x.permute(0, 2, 1, 3, 4)

      # flatten
      x = x.reshape(b, -1, h, w)

      return x

  def forward(self, x):
      b, c, h, w = x.size()
      # group into subfeatures
      x = x.view(b * self.G, -1, h, w)  # bs*G,c//G,h,w

      # channel_split
      x_0, x_1 = x.chunk(2, dim=1)  # bs*G,c//(2*G),h,w

      # channel attention
      x_channel = self.avg_pool(x_0)  # bs*G,c//(2*G),1,1
      x_channel = self.cweight * x_channel + self.cbias  # bs*G,c//(2*G),1,1
      x_channel = x_0 * self.sigmoid(x_channel)

      # spatial attention
      x_spatial = self.gn(x_1)  # bs*G,c//(2*G),h,w
      x_spatial = self.sweight * x_spatial + self.sbias  # bs*G,c//(2*G),h,w
      x_spatial = x_1 * self.sigmoid(x_spatial)  # bs*G,c//(2*G),h,w

      # concatenate along channel axis
      out = torch.cat([x_channel, x_spatial], dim=1)  # bs*G,c//G,h,w
      out = out.contiguous().view(b, -1, h, w)

      # channel shuffle
      out = self.channel_shuffle(out, 2)
      return out

if name == 'main':
input = torch.randn(50, 512, 7, 7)
se = ShuffleAttention(channel=512, G=8)
output = se(input)
print(output.shape)


* 实验结果

  ```bash
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.350
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.523
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.401
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.123
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.479
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.662
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.424
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.424
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.424
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.160
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.576
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.733

6. Polarized Self-Attention: Towards High-quality Pixel-wise Regression

论文地址：https://arxiv.org/abs/2107.00782

  • 网络结构

    image-20211217105853958

  • Pytorch代码

    ```python
    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init

class ParallelPolarizedSelfAttention(nn.Module):

  def __init__(self, channel=512):
      super().__init__()
      self.ch_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
      self.ch_wq = nn.Conv2d(channel, 1, kernel_size=(1, 1))
      self.softmax_channel = nn.Softmax(1)
      self.softmax_spatial = nn.Softmax(-1)
      self.ch_wz = nn.Conv2d(channel // 2, channel, kernel_size=(1, 1))
      self.ln = nn.LayerNorm(channel)
      self.sigmoid = nn.Sigmoid()
      self.sp_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
      self.sp_wq = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
      self.agp = nn.AdaptiveAvgPool2d((1, 1))

  def forward(self, x):
      b, c, h, w = x.size()

      # Channel-only Self-Attention
      channel_wv = self.ch_wv(x)  # bs,c//2,h,w
      channel_wq = self.ch_wq(x)  # bs,1,h,w
      channel_wv = channel_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
      channel_wq = channel_wq.reshape(b, -1, 1)  # bs,h*w,1
      channel_wq = self.softmax_channel(channel_wq)
      channel_wz = torch.matmul(channel_wv, channel_wq).unsqueeze(-1)  # bs,c//2,1,1
      channel_weight = self.sigmoid(self.ln(self.ch_wz(channel_wz).reshape(b, c, 1).permute(0, 2, 1))).permute(0, 2,
                                                                                                               1).reshape(
          b, c, 1, 1)  # bs,c,1,1
      channel_out = channel_weight * x

      # Spatial-only Self-Attention
      spatial_wv = self.sp_wv(x)  # bs,c//2,h,w
      spatial_wq = self.sp_wq(x)  # bs,c//2,h,w
      spatial_wq = self.agp(spatial_wq)  # bs,c//2,1,1
      spatial_wv = spatial_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
      spatial_wq = spatial_wq.permute(0, 2, 3, 1).reshape(b, 1, c // 2)  # bs,1,c//2
      spatial_wq = self.softmax_spatial(spatial_wq)
      spatial_wz = torch.matmul(spatial_wq, spatial_wv)  # bs,1,h*w
      spatial_weight = self.sigmoid(spatial_wz.reshape(b, 1, h, w))  # bs,1,h,w
      spatial_out = spatial_weight * x
      out = spatial_out + channel_out
      return out

class SequentialPolarizedSelfAttention(nn.Module):

  def __init__(self, channel=512):
      super().__init__()
      self.ch_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
      self.ch_wq = nn.Conv2d(channel, 1, kernel_size=(1, 1))
      self.softmax_channel = nn.Softmax(1)
      self.softmax_spatial = nn.Softmax(-1)
      self.ch_wz = nn.Conv2d(channel // 2, channel, kernel_size=(1, 1))
      self.ln = nn.LayerNorm(channel)
      self.sigmoid = nn.Sigmoid()
      self.sp_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
      self.sp_wq = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
      self.agp = nn.AdaptiveAvgPool2d((1, 1))

  def forward(self, x):
      b, c, h, w = x.size()

      # Channel-only Self-Attention
      channel_wv = self.ch_wv(x)  # bs,c//2,h,w
      channel_wq = self.ch_wq(x)  # bs,1,h,w
      channel_wv = channel_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
      channel_wq = channel_wq.reshape(b, -1, 1)  # bs,h*w,1
      channel_wq = self.softmax_channel(channel_wq)
      channel_wz = torch.matmul(channel_wv, channel_wq).unsqueeze(-1)  # bs,c//2,1,1
      channel_weight = self.sigmoid(self.ln(self.ch_wz(channel_wz).reshape(b, c, 1).permute(0, 2, 1))).permute(0, 2,
                                                                                                               1).reshape(
          b, c, 1, 1)  # bs,c,1,1
      channel_out = channel_weight * x

      # Spatial-only Self-Attention
      spatial_wv = self.sp_wv(channel_out)  # bs,c//2,h,w
      spatial_wq = self.sp_wq(channel_out)  # bs,c//2,h,w
      spatial_wq = self.agp(spatial_wq)  # bs,c//2,1,1
      spatial_wv = spatial_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
      spatial_wq = spatial_wq.permute(0, 2, 3, 1).reshape(b, 1, c // 2)  # bs,1,c//2
      spatial_wq = self.softmax_spatial(spatial_wq)
      spatial_wz = torch.matmul(spatial_wq, spatial_wv)  # bs,1,h*w
      spatial_weight = self.sigmoid(spatial_wz.reshape(b, 1, h, w))  # bs,1,h,w
      spatial_out = spatial_weight * channel_out
      return spatial_out

if name == 'main':
input = torch.randn(1, 512, 7, 7)
psa = SequentialPolarizedSelfAttention(channel=512)
output = psa(input)
print(output.shape)


* 实验结果

  ```bash
  2021-12-16 20:30:36,981 - mmdet - INFO - 
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.346
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.522
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.385
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.123
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.474
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.676
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.422
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.422
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.422
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.170
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.570
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.743

7. Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks

论文地址：https://arxiv.org/pdf/1905.09646.pdf

  • 网络结构

    主要是用在语义分割上，所以在检测上的效果一般，没有带来多少提升

    image-20211217112822310

  • Pytorch代码

    ```python
    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init

class SpatialGroupEnhance(nn.Module):

  def __init__(self, groups):
      super().__init__()
      self.groups = groups
      self.avg_pool = nn.AdaptiveAvgPool2d(1)
      self.weight = nn.Parameter(torch.zeros(1, groups, 1, 1))
      self.bias = nn.Parameter(torch.zeros(1, groups, 1, 1))
      self.sig = nn.Sigmoid()
      self.init_weights()

  def init_weights(self):
      for m in self.modules():
          if isinstance(m, nn.Conv2d):
              init.kaiming_normal_(m.weight, mode='fan_out')
              if m.bias is not None:
                  init.constant_(m.bias, 0)
          elif isinstance(m, nn.BatchNorm2d):
              init.constant_(m.weight, 1)
              init.constant_(m.bias, 0)
          elif isinstance(m, nn.Linear):
              init.normal_(m.weight, std=0.001)
              if m.bias is not None:
                  init.constant_(m.bias, 0)

  def forward(self, x):
      b, c, h, w = x.shape
      x = x.view(b * self.groups, -1, h, w)  # bs*g,dim//g,h,w
      xn = x * self.avg_pool(x)  # bs*g,dim//g,h,w
      xn = xn.sum(dim=1, keepdim=True)  # bs*g,1,h,w
      t = xn.view(b * self.groups, -1)  # bs*g,h*w

      t = t - t.mean(dim=1, keepdim=True)  # bs*g,h*w
      std = t.std(dim=1, keepdim=True) + 1e-5
      t = t / std  # bs*g,h*w
      t = t.view(b, self.groups, h, w)  # bs,g,h*w

      t = t * self.weight + self.bias  # bs,g,h*w
      t = t.view(b * self.groups, 1, h, w)  # bs*g,1,h*w
      x = x * self.sig(t)
      x = x.view(b, c, h, w)

      return x

if name == 'main':
input = torch.randn(50, 512, 7, 7)
sge = SpatialGroupEnhance(groups=8)
output = sge(input)
print(output.shape)




* 实验结果

  ```bash
  2021-12-16 21:39:42,785 - mmdet - INFO - 
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.342
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.516
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.381
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.117
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.474
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.652
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.415
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.415
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.415
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.155
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.565
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.718

8. Coordinate Attention for Efficient Mobile Network Design

论文地址：https://arxiv.org/abs/2103.02907

  • 网络结构

    主要应用在轻量级网络上，在resnet系列上效果不好。

    image-20211210155718877

  • Pytorch代码

    ```python
    import torch
    import torch.nn as nn
    import torch.nn.functional as F

class h_sigmoid(nn.Module):
def init(self, inplace=True):
super(h_sigmoid, self).init()
self.relu = nn.ReLU6(inplace=inplace)

  def forward(self, x):
      return self.relu(x + 3) / 6

class h_swish(nn.Module):
def init(self, inplace=True):
super(h_swish, self).init()
self.sigmoid = h_sigmoid(inplace=inplace)

  def forward(self, x):
      return x * self.sigmoid(x)

class CoordAtt(nn.Module):
def init(self, inp, oup, reduction=32):
super(CoordAtt, self).init()
self.pool_h = nn.AdaptiveAvgPool2d((None, 1))
self.pool_w = nn.AdaptiveAvgPool2d((1, None))

      mip = max(8, inp // reduction)

      self.conv1 = nn.Conv2d(inp, mip, kernel_size=1, stride=1, padding=0)
      self.bn1 = nn.BatchNorm2d(mip)
      self.act = h_swish()

      self.conv_h = nn.Conv2d(mip, oup, kernel_size=1, stride=1, padding=0)
      self.conv_w = nn.Conv2d(mip, oup, kernel_size=1, stride=1, padding=0)

  def forward(self, x):
      identity = x

      n, c, h, w = x.size()
      x_h = self.pool_h(x)
      x_w = self.pool_w(x).permute(0, 1, 3, 2)

      y = torch.cat([x_h, x_w], dim=2)
      y = self.conv1(y)
      y = self.bn1(y)
      y = self.act(y)

      x_h, x_w = torch.split(y, [h, w], dim=2)
      x_w = x_w.permute(0, 1, 3, 2)

      a_h = self.conv_h(x_h).sigmoid()
      a_w = self.conv_w(x_w).sigmoid()

      out = identity * a_w * a_h

      return out


* 实验结果

  ```bash
  2021-12-16 19:04:16,776 - mmdet - INFO - 
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.340
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.516
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.386
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.127
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.457
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.632
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.408
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.408
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.408
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.162
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.546
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.716

9. Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

论文地址： https://arxiv.org/abs/2112.05561

  • 网络结构

    计算量特别大，效果一般

  • Pytorch代码

    class GAM_Attention(nn.Module):
    def __init__(self, in_channels, out_channels, rate=4):
        super(GAM_Attention, self).__init__()

        self.channel_attention = nn.Sequential(
            nn.Linear(in_channels, int(in_channels / rate)),
            nn.ReLU(inplace=True),
            nn.Linear(int(in_channels / rate), in_channels)
        )

        self.spatial_attention = nn.Sequential(
            nn.Conv2d(in_channels, int(in_channels / rate), kernel_size=7, padding=3),
            nn.BatchNorm2d(int(in_channels / rate)),
            nn.ReLU(inplace=True),
            nn.Conv2d(int(in_channels / rate), out_channels, kernel_size=7, padding=3),
            nn.BatchNorm2d(out_channels)
        )

    def forward(self, x):
        # print(x)
        b, c, h, w = x.shape
        x_permute = x.permute(0, 2, 3, 1).view(b, -1, c)
        x_att_permute = self.channel_attention(x_permute).view(b, h, w, c)
        x_channel_att = x_att_permute.permute(0, 3, 1, 2)

        x = x * x_channel_att

        x_spatial_att = self.spatial_attention(x).sigmoid()
        out = x * x_spatial_att
        # print(out)

        return out

  • 实验结果

    2021-12-16 16:14:20,693 - mmdet - INFO - 
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.350
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.530
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.399
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.131
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.481
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.683
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.424
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.424
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.424
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.171
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.575
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.745

更多注意力

参考：https://github.com/xmu-xiaoma666/External-Attention-pytorch

另外还有一些用在语义分割上面的结构，这里就不测试了，大家可以自行下去测试

双路注意力机制-DANET

论文标题：Fu_Dual_Attention_Network_for_Scene_Segmentation

论文地址：https://openaccess.thecvf.com/content_CVPR_2019/papers/Fu_Dual_Attention_Network_for_Scene_Segmentation_CVPR_2019_paper.pdf

时间：2019

相当于之前是并行的结构，现在改成了串行的结构然后做特征的concat

image-20211210154740171

image-20211210154829462

位置注意力-CCNET

在上面的danet上改的，主要是解决计算量的问题, 通过十字交叉的结构来解决

论文标题：CCNet: Criss-Cross Attention for Semantic Segmentation

论文地址：https://openaccess.thecvf.com/content_ICCV_2019/papers/Huang_CCNet_Criss-Cross_Attention_for_Semantic_Segmentation_ICCV_2019_paper.pdf

时：2019

image-20211210155141717

找到我

你可以通过这些方式来寻找我。

B站：肆十二-

CSDN：肆十二

知乎：肆十二

微博：肆十二-

现在关注以后就是老朋友喽！

image-20211212195912911

文章标签：
机器学习/深度学习
算法框架/工具
PyTorch
计算机视觉
Python
宋晨明
目录
相关文章
石臻臻的杂货铺
|
2天前
|
编解码 Java 程序员
写代码还有专业的编程显示器？
写代码已经十个年头了, 一直都是习惯直接用一台Mac电脑写代码 偶尔接一个显示器, 但是可能因为公司配的显示器不怎么样, 还要接转接头 搞得桌面杂乱无章，分辨率也低，感觉屏幕还是Mac自带的看着舒服
石臻臻的杂货铺
3679 2
绿水长流*z
|
3天前
|
存储 缓存 关系型数据库
MySQL事务日志-Redo Log工作原理分析
事务的隔离性和原子性分别通过锁和事务日志实现，而持久性则依赖于事务日志中的`Redo Log`。在MySQL中，`Redo Log`确保已提交事务的数据能持久保存，即使系统崩溃也能通过重做日志恢复数据。其工作原理是记录数据在内存中的更改，待事务提交时写入磁盘。此外，`Redo Log`采用简单的物理日志格式和高效的顺序IO，确保快速提交。通过不同的落盘策略，可在性能和安全性之间做出权衡。
绿水长流*z
1540 5
弹性计算-百晓生
|
1月前
|
弹性计算 人工智能 架构师
阿里云携手Altair共拓云上工业仿真新机遇
2024年9月12日，「2024 Altair 技术大会杭州站」成功召开，阿里云弹性计算产品运营与生态负责人何川，与Altair中国技术总监赵阳在会上联合发布了最新的“云上CAE一体机”。
弹性计算-百晓生
82310 25
阿里云携手Altair共拓云上工业仿真新机遇
乘风直上
|
7天前
|
人工智能 Rust Java
10月更文挑战赛火热启动，坚持热爱坚持创作！
开发者社区10月更文挑战，寻找热爱技术内容创作的你，欢迎来创作！
乘风直上
578 22
绿水长流*z
|
3天前
|
存储 SQL 关系型数据库
彻底搞懂InnoDB的MVCC多版本并发控制
本文详细介绍了InnoDB存储引擎中的两种并发控制方法：MVCC（多版本并发控制）和LBCC（基于锁的并发控制）。MVCC通过记录版本信息和使用快照读取机制，实现了高并发下的读写操作，而LBCC则通过加锁机制控制并发访问。文章深入探讨了MVCC的工作原理，包括插入、删除、修改流程及查询过程中的快照读取机制。通过多个案例演示了不同隔离级别下MVCC的具体表现，并解释了事务ID的分配和管理方式。最后，对比了四种隔离级别的性能特点，帮助读者理解如何根据具体需求选择合适的隔离级别以优化数据库性能。
绿水长流*z
201 3
大模型服务小助手
|
10天前
|
JSON 自然语言处理 数据管理
阿里云百炼产品月刊【2024年9月】
阿里云百炼产品月刊【2024年9月】，涵盖本月产品和功能发布、活动，应用实践等内容，帮助您快速了解阿里云百炼产品的最新动态。
大模型服务小助手
357 6
阿里云百炼产品月刊【2024年9月】
爱吃龙利鱼
|
10天前
|
Linux 虚拟化 开发者
一键将CentOs的yum源更换为国内阿里yum源
一键将CentOs的yum源更换为国内阿里yum源
爱吃龙利鱼
578 5
Ganos全空间数据库
|
23天前
|
存储 关系型数据库 分布式数据库
GraphRAG：基于PolarDB+通义千问+LangChain的知识图谱+大模型最佳实践
本文介绍了如何使用PolarDB、通义千问和LangChain搭建GraphRAG系统，结合知识图谱和向量检索提升问答质量。通过实例展示了单独使用向量检索和图检索的局限性，并通过图+向量联合搜索增强了问答准确性。PolarDB支持AGE图引擎和pgvector插件，实现图数据和向量数据的统一存储与检索，提升了RAG系统的性能和效果。
Ganos全空间数据库
1596 119
椰椰椰耶
|
6天前
|
XML 安全 Java
【Maven】依赖管理，Maven仓库，Maven核心功能
【Maven】依赖管理，Maven仓库，Maven核心功能
椰椰椰耶
233 3
瓴羊Dataphin
|
9天前
|
存储 人工智能 搜索推荐
数据治理，是时候打破刻板印象了
瓴羊智能数据建设与治理产品Datapin全面升级，可演进扩展的数据架构体系为企业数据治理预留发展空间，推出敏捷版用以解决企业数据量不大但需构建数据的场景问题，基于大模型打造的DataAgent更是为企业用好数据资产提供了便利。
瓴羊Dataphin
327 2