| \[A + B] →食物(UCS) | \[A]中等; \[B]中等 |
| 阻断效应 | \[A] → UCS | \[A + B] → UCS | \[A]强; \[B]无 |
| 抑制效应 | \[A] → UCS | \[A + B] → 0 | \[A]强\[B]无/ \[A]强\[B]负 |
A:光； B: 声音； UCS: 无条件刺激; \[A] 或者 \[B] 代表光或者声音单独出现；\[A + B]代表光和声音一起出现。
### 2. Rescorlar-wagner模型
由之前的章节得到心理学中强化学习的最简递推公式：$$V\_t=V\_{t-1}+a\*(r\_t - V\_{t-1})$$，被称为简单强化学习模型（navie reinforcement learning model）。但是简单的强化学习（PE*i* = r*i* − ；*i* = V*i* -1+ ）无法解释以上三种条件反射现象。
以下通过Python代码模拟基于简单强化学习模型的遮蔽效应的形成过程。
````python
# Naïve model
from scipy.stats import bernoulli
import numpy as np
import matplotlib.pyplot as plt
def overshadow(T):
'''
T: number of trials
'''
r = np.ones(T) # (T,)维全是1的数组
A = np.ones(T) # (T,)维全是1的数组
B = np.ones(T) # (T,)维全是1的数组
return r, A, B
a = 0.05 # learning rate
T = 100 # number of trials
r, A, B = overshadow(T)
Va = np.empty(T)
Vb = np.empty(T)
for i in range(T): # loop trials
if i == 0:
Va[0] = 0 + a*(r[0]-0)
Vb[0] = 0 + a*(r[0]-0)
else:
Va[i] = Va[i-1] + a*(r[i]-Va[i-1])
Vb[i] = Vb[i-1] + a*(r[i]-Vb[i-1])
plt.plot(r, '-k', label='reward')
plt.plot(Va, linewidth=10, label='Va')
plt.plot(Vb, linewidth=2, label='Vb')
plt.xlabel('Trials')
plt.ylabel('Value')
plt.legend()
```
````
图1 简单强化学习模型下刺激A和B的价值更新过程。
基于此，1972年Robert A. Rescorla和Allan R. Wagner提出了瑞思考勒-瓦格纳（Rescorlar-Wagner, RW)模型。根据RW模型，当多个刺激存在的时候，预测误差（Prediction error）应该为奖赏减去所有出现刺激的价值（ $$PE\_i=r\_i-\sum\_K^N{V\_K}$$，其中第i个trial，一种出现N个刺激）。在更新刺激的价值时，只更新出现了刺激的价值。因此对于同时出现两种刺激的刺激价值的更新过程为：
PE\[i] = r\[i]-(Va\[i-1]A\[i]+Vb\[i-1]B\[i])
Va\[i] = Va\[i-1] + aPE\[i] \* A\[i]
Vb\[i] = Vb\[i-1] + aPE\[i] \* B\[i]
RW模型能成功解释上述三种现象。以下通过Python代码模拟基于RW模型的遮蔽效应，阻断效应和抑制效应的形成过程。
```python
# RW model-Overshadowing
a = 0.05 # learning rate
T = 100 # how many trials
r, A, B = overshadow(T)
Va = np.empty(T)
Vb = np.empty(T)
for i in range(T): # loop trials
if i == 0:
Va[0] = 0 + a*(r[0] - 0 * A[i] - 0 * B[i])
Vb[0] = 0 + a*(r[0] - 0 * A[i] - 0 * B[i])
else:
PE = r[i]-(Va[i-1]*A[i]+Vb[i-1]*B[i])
Va[i] = Va[i-1] + a*PE * A[i]
Vb[i] = Vb[i-1] + a*PE * B[i]
plt.plot(r, '-k', label='reward')
plt.plot(Va, linewidth=10, label='Va')
plt.plot(Vb, linewidth=2, label='Vb')
plt.xlabel('Trials')
plt.ylabel('Value')
plt.legend()
```
图2 RW模型模拟的两个刺激同时出现时刺激值的更新过程和相应的遮蔽效应。
```python
def block(T):
T1 = round(T/2) # round是取整数的意思
# phase 1: A -> R
r1 = np.ones(T1)
A1 = np.ones(T1)
B1 = np.zeros(T1)
# phase 2: A, B -> R
r2 = np.ones(T-T1)
A2 = np.ones(T-T1)
B2 = np.ones(T-T1)
# combine two phases
r = np.hstack((r1, r2))
A = np.hstack((A1, A2))
B = np.hstack((B1, B2))
return r, A, B
a = 0.05 # learning rate
T = 1000 # how many trials
r, A, B = block(T)
Va = np.empty(T)
Vb = np.empty(T)
for i in range(T): # loop trials
if i == 0:
Va[0] = 0 + a*(r[0] - 0 * A[i] - 0 * B[i])
Vb[0] = 0 + a*(r[0] - 0 * A[i] - 0 * B[i])
else:
PE = r[i]-Va[i-1]*A[i]-Vb[i-1]*B[i]
Va[i] = Va[i-1] + a*PE * A[i]
Vb[i] = Vb[i-1] + a*PE * B[i]
plt.plot(r, '-k', label='reward')
plt.plot(Va, linewidth=10, label='Va')
plt.plot(Vb, linewidth=2, label='Vb')
plt.xlabel('Trials')
plt.ylabel('Value')
plt.legend()
```
图3 RW模型模拟的两个刺激同时出现时两个刺激值的更新过程和相应的阻断效应。
```python
def inhibition(T):
r = np.empty(T)
A = np.empty(T)
B = np.empty(T)
for i in range(T):
if bernoulli.rvs(0.5, 1)==1:
# A -> R trial
r[i] = 1
A[i] = 1
B[i] = 0
else:
# A,B -> 0 trial
r[i] = 0
A[i] = 1
B[i] = 1
return r, A, B
a = 0.05 # learning rate
T = 1000 # how many trials
r, A, B = inhibition(T)
Va = np.empty(T)
Vb = np.empty(T)
for i in range(T): # loop trials
if i == 0:
Va[0] = 0 + a*(r[0] - 0 * A[i] - 0 * B[i])
Vb[0] = 0 + a*(r[0] - 0 * A[i] - 0 * B[i])
else:
PE = r[i]-Va[i-1]*A[i]-Vb[i-1]*B[i]
Va[i] = Va[i-1] + a*PE * A[i]
Vb[i] = Vb[i-1] + a*PE * B[i]
#plt.plot(r, '-k', label='reward')
plt.plot(Va, linewidth=10, label='Va')
plt.plot(Vb, linewidth=2, label='Vb')
plt.xlabel('Trials')
plt.ylabel('Value')
plt.legend()
```
图4 RW模型模拟的两个刺激同时出现时两个刺激值的更新过程和相应的抑制效应。
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://ruyuanzhang.gitbook.io/compmodcogpsy/di-ba-zhang-qiang-hua-xue-xi/rescorlarwagner-mo-xing.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.