パラメータ管理
==============
アーキテクチャを選び、 ハイパーパラメータを設定したら、
次は学習ループに進む。 ここでの目標は、損失関数を最小化する
パラメータ値を見つけることである。 学習後には、将来の予測を行うために
これらのパラメータが必要になる。
さらに、場合によってはパラメータを取り出して、 別の文脈で再利用したり、
モデルをディスクに保存して 他のソフトウェアで実行できるようにしたり、
あるいは科学的理解を得ることを期待して 調べたりしたいこともある。
ほとんどの場合、パラメータがどのように宣言され、
操作されるかという細かな詳細は気にせず、 深層学習フレームワークに
重い処理を任せることができる。 しかし、標準的な層を積み重ねた
アーキテクチャから離れると、 パラメータの宣言や操作の細部に
踏み込む必要が出てくることがある。 この節では、次の内容を扱う。
- デバッグ、診断、可視化のためのパラメータへのアクセス。
- 異なるモデル構成要素間でのパラメータ共有。
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import torch
from torch import nn
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from mxnet import init, np, npx
from mxnet.gluon import nn
npx.set_np()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from d2l import jax as d2l
from flax import linen as nn
import jax
from jax import numpy as jnp
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import tensorflow as tf
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential(nn.LazyLinear(8),
nn.ReLU(),
nn.LazyLinear(1))
X = torch.rand(size=(2, 4))
net(X).shape
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
torch.Size([2, 1])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential()
net.add(nn.Dense(8, activation='relu'))
net.add(nn.Dense(1))
net.initialize() # Use the default initialization method
X = np.random.uniform(size=(2, 4))
net(X).shape
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[07:06:02] ../src/storage/storage.cc:196: Using Pooled (Naive) StorageManager for CPU
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(2, 1)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential([nn.Dense(8), nn.relu, nn.Dense(1)])
X = jax.random.uniform(d2l.get_key(), (2, 4))
params = net.init(d2l.get_key(), X)
net.apply(params, X).shape
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(2, 1)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = tf.keras.models.Sequential([
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(4, activation=tf.nn.relu),
tf.keras.layers.Dense(1),
])
X = tf.random.uniform((2, 4))
net(X).shape
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
TensorShape([2, 1])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net[2].state_dict()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
OrderedDict([('weight',
tensor([[-0.0915, 0.2323, 0.0672, 0.0951, 0.0561, 0.2846, 0.3074, -0.2919]])),
('bias', tensor([0.2610]))])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net[1].params
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
dense1_ (
Parameter dense1_weight (shape=(1, 8), dtype=float32)
Parameter dense1_bias (shape=(1,), dtype=float32)
)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
params['params']['layers_2']
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
FrozenDict({
kernel: Array([[-0.5666766 ],
[-0.2169885 ],
[ 0.37719882],
[-0.2976346 ],
[ 0.15896064],
[ 0.72404635],
[-0.44518152],
[-0.08496901]], dtype=float32),
bias: Array([0.], dtype=float32),
})
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net.layers[2].weights
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[,
]
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
type(net[2].bias), net[2].bias.data
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(torch.nn.parameter.Parameter, tensor([0.2610]))
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
type(net[1].bias), net[1].bias.data()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(mxnet.gluon.parameter.Parameter, array([0.]))
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
bias = params['params']['layers_2']['bias']
type(bias), bias
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(jaxlib.xla_extension.ArrayImpl, Array([0.], dtype=float32))
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
type(net.layers[2].weights[1]), tf.convert_to_tensor(net.layers[2].weights[1])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(tensorflow.python.ops.resource_variable_ops.ResourceVariable,
)
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net[2].weight.grad == None
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
True
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net[1].weight.grad()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[0., 0., 0., 0., 0., 0., 0., 0.]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
jax.tree_util.tree_map(lambda x: x.shape, params)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
FrozenDict({
params: {
layers_0: {
bias: (8,),
kernel: (4, 8),
},
layers_2: {
bias: (1,),
kernel: (8, 1),
},
},
})
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net.get_weights()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[array([[ 0.8407554 , -0.70623875, -0.5730775 , 0.19883841],
[ 0.22767669, -0.00824153, -0.27324384, -0.05706435],
[ 0.66750854, 0.72794133, -0.57247126, -0.4776808 ],
[ 0.35080236, 0.47631615, -0.53025633, -0.19356781]],
dtype=float32),
array([0., 0., 0., 0.], dtype=float32),
array([[ 0.2972461],
[ 0.2490077],
[-0.9411764],
[ 1.0483162]], dtype=float32),
array([0.], dtype=float32)]
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
[(name, param.shape) for name, param in net.named_parameters()]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[('0.weight', torch.Size([8, 4])),
('0.bias', torch.Size([8])),
('2.weight', torch.Size([1, 8])),
('2.bias', torch.Size([1]))]
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net.collect_params()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
sequential0_ (
Parameter dense0_weight (shape=(8, 4), dtype=float32)
Parameter dense0_bias (shape=(8,), dtype=float32)
Parameter dense1_weight (shape=(1, 8), dtype=float32)
Parameter dense1_bias (shape=(1,), dtype=float32)
)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# We need to give the shared layer a name so that we can refer to its
# parameters
shared = nn.Dense(8)
net = nn.Sequential([nn.Dense(8), nn.relu,
shared, nn.relu,
shared, nn.relu,
nn.Dense(1)])
params = net.init(jax.random.PRNGKey(d2l.get_seed()), X)
# Check whether the parameters are different
print(len(params['params']) == 3)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
True
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# tf.keras behaves a bit differently. It removes the duplicate layer
# automatically
shared = tf.keras.layers.Dense(4, activation=tf.nn.relu)
net = tf.keras.models.Sequential([
tf.keras.layers.Flatten(),
shared,
shared,
tf.keras.layers.Dense(1),
])
net(X)
# Check whether the parameters are different
print(len(net.layers) == 3)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
True
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# We need to give the shared layer a name so that we can refer to its
# parameters
shared = nn.LazyLinear(8)
net = nn.Sequential(nn.LazyLinear(8), nn.ReLU(),
shared, nn.ReLU(),
shared, nn.ReLU(),
nn.LazyLinear(1))
net(X)
# Check whether the parameters are the same
print(net[2].weight.data[0] == net[4].weight.data[0])
net[2].weight.data[0, 0] = 100
# Make sure that they are actually the same object rather than just having the
# same value
print(net[2].weight.data[0] == net[4].weight.data[0])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([True, True, True, True, True, True, True, True])
tensor([True, True, True, True, True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential()
# We need to give the shared layer a name so that we can refer to its
# parameters
shared = nn.Dense(8, activation='relu')
net.add(nn.Dense(8, activation='relu'),
shared,
nn.Dense(8, activation='relu', params=shared.params),
nn.Dense(10))
net.initialize()
X = np.random.uniform(size=(2, 20))
net(X)
# Check whether the parameters are the same
print(net[1].weight.data()[0] == net[2].weight.data()[0])
net[1].weight.data()[0, 0] = 100
# Make sure that they are actually the same object rather than just having the
# same value
print(net[1].weight.data()[0] == net[2].weight.data()[0])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[ True True True True True True True True]
[ True True True True True True True True]
.. raw:: html
.. raw:: html