Create an environment with a discrete action space, and obtain its observation and action specifications. For this example, load the environment used in the exampleCreate Agent Using Deep Network Designer and Train Using Image Observations。该环境有两个观察结果：50 x 50的灰度图像和标量（摆的角速度）。The action is a scalar with five possible elements (a torque of either -2， -1,0,1, or2Nm applied to a swinging pole).

% load predefined environmentenv = rlPredefinedEnv(“简单的pendulumwithimage-discrete”）；

Obtain the observation and action specifications for this environment.

obsInfo = getObservationInfo(env); actInfo = getActionInfo(env);

The agent creation function initializes the actor and critic networks randomly. You can ensure reproducibility by fixing the seed of the random generator. To do so, uncomment the following line.

% rng(0)

Create a TRPO agent from the environment observation and action specifications.

agent = rlTRPOAgent(obsInfo,actInfo);

To check your agent, usegetActionto return the action from a random observation.

getAction(agent,{rand(obsInfo(1).Dimension),rand(obsInfo(2).Dimension)})

ans =1x1 cell array{[-2]}

你现在可以测试和列车内的代理vironment.

创建一个具有连续动作空间的环境，并获得其观察和行动规范。对于此示例，加载示例中使用的环境Train DDPG Agent to Swing Up and Balance Pendulum with Image Observation。该环境有两个观察结果：50 x 50的灰度图像和标量（摆的角速度）。该动作是代表扭矩连续的标量。2to2Nm.

env = rlPredefinedEnv("SimplePendulumWithImage-Continuous"）；

获取有关此环境的观察和行动规范。

obsInfo = getObservationInfo(env); actInfo = getActionInfo(env);

Create an agent initialization options object, specifying that each hidden fully connected layer in the network must have128neurons.

initOpts = rlAgentInitializationOptions('NumHiddenUnit',128);

The agent creation function initializes the actor and critic networks randomly. You can ensure reproducibility by fixing the seed of the random generator. To do so, uncomment the following line.

% rng(0)

使用指定的初始化选项从环境观察和操作规范中创建TRPO代理。

agent = rlTRPOAgent(obsInfo,actInfo,initOpts);

Extract the deep neural networks from both the agent actor and critic.

actorNet = getModel(getActor(agent)); criticNet = getModel(getCritic(agent));

您可以验证网络在其隐藏的完全连接层中有128个单元。例如，显示评论家网络的层。

criticNet.Layers

ans = 11x1层阵列带有层：1'contat'沿尺寸的2个输入的串联串联1 2'relu_body'relu'relu 3'fc_body'完全连接的128完全连接的层4'hody_output'hody_output'relu relu 5'input_1'input_1'input_1'图像输入50x50x15 x10x11图像6'conv_1'卷积64 3x3x1大步[1 1]和填充[0 0 0 0] 7'relu_input_1'relu relu 8'fc_1'完全连接的128完全连接的层9'input_2''input_2'功能输入1特征10'fc_2'fc_2'fc_2''fc_2''fc_2''fc_2''fc_2''完全连接的128完全连接的层11'输出'完全连接1完全连接的层

To check your agent, usegetActionto return the action from a random observation.

getAction(agent,{rand(obsInfo(1).Dimension),rand(obsInfo(2).Dimension)})

ans =1x1 cell array{[0.9228]}

你现在可以测试和列车内的代理vironment.

创造一个环境界面,并获得其observation and action specifications.

env = rlPredefinedEnv("CartPole-Discrete"）；obsInfo = getObservationInfo(env); actInfo = getActionInfo(env);

Create a deep neural network to be used as approximation model within the critic. For PPO agents, the critic estimates a value function, therefore it must take the observation signal as input and return a scalar value.

criticNetwork = [ featureInputLayer(prod(obsInfo.Dimension),。。。'Normalization','none','Name','状态') fullyConnectedLayer(1,'Name','CriticFC'）;

创建评论家criticNetwork。TRPO agents use anrlValueFunctionobject to implement the critic.

critic = rlValueFunction(criticNetwork,obsInfo);

Set some training options for the critic.

criticOpts = rlOptimizerOptions(。。。'LearnRate'，8e-3，'GradientThreshold'，1）;

Create a deep neural network to be used as approximation model within the actor. For TRPO agents, the actor executes a stochastic policy, which for discrete action spaces is implemented by a discrete categorical actor. In this case the network must take the observation signal as input and return a probability for each action. Therefore the output layer must have as many elements as the number of possible actions.

actorNetwork = [ featureInputLayer(prod(obsInfo.Dimension),。。。'Normalization','none','Name','状态') fullyConnectedLayer(numel(actInfo.Dimension),。。。'Name','action') ];

Create the actor usingactorNetwork。PPO代理使用rlDiscreteCategoricalActorobject to implement the actor for discrete action spaces.

actor = rlDiscreteCategoricalActor(actorNetwork,obsInfo,actInfo);

Specify agent options, and create a TRPO agent using the environment, actor, and critic.

agentOpts = rlTRPOAgentOptions(。。。'ExperienceHorizon',1024,。。。“折扣器”,0.95,。。。``批评'',criticOpts); agent = rlTRPOAgent(actor,critic,agentOpts)

代理=具有属性的rltrpoagent：agentOptions：[1x1 rl.option.rltrpoagentoptions] useExplorationPolicy：1 ObservationInfo：[1x1 rl.util.rlnumericspec] ActionInfo：[1x1 rl.util.util.rlfinitesetsetspec] SameSemple。

To check your agent, usegetActionto return the action from a random observation.

getAction(agent,{rand(obsInfo.Dimension)})

ans =1x1 cell array{[-10]}

您现在可以测试和训练代理商针对环境。

Create an environment with a continuous action space, and obtain its observation and action specifications. For this example, load the double integrator continuous action space environment used in the exampleTrain DDPG Agent to Control Double Integrator System。来自环境的观察结果是一个载体，其中包含质量的位置和速度。该动作是代表施加到质量的力的标量，连续从 -2to2Newton.

env = rlPredefinedEnv("DoubleIntegrator-Continuous"）；obsInfo = getObservationInfo(env)

obsInfo = rlNumericSpec with properties: LowerLimit: -Inf UpperLimit: Inf Name: "states" Description: "x, dx" Dimension: [2 1] DataType: "double"

actInfo = getActionInfo(env)

actInfo = rlNumericSpec with properties: LowerLimit: -Inf UpperLimit: Inf Name: "force" Description: [0x0 string] Dimension: [1 1] DataType: "double"

Since the action must be contained in a limited range, set the upper and lower limit of the action signal accordingly, so you can easily retrieve them when building the actor network.

actInfo.LowerLimit=-2; actInfo.UpperLimit=2;

The actor and critic networks are initialized randomly. You can ensure reproducibility by fixing the seed of the random generator.

rng(0)

Create a deep neural network to be used as approximation model within the critic. For TRPO agents, the critic estimates a value function, therefore it must take the observation signal as input and return a scalar value.

criticNet = [ featureInputLayer(prod(obsInfo.Dimension),。。。'Normalization','none','Name','状态'）完整连接的layer（10，'Name','fc_in') reluLayer('Name','relu') fullyConnectedLayer(1,'Name','out'）;

创建评论家criticNet。PPO代理使用rlValueFunctionobject to implement the critic.

评论家= rlvaluefunction（Criticnet，Obsinfo）；

Set some training options for the critic.

criticOpts = rlOptimizerOptions(。。。'LearnRate'，8e-3，'GradientThreshold'，1）;

Create a deep neural network to be used as approximation model within the actor. For PPO agents, the actor executes a stochastic policy, which for continuous action spaces is implemented by a continuous Gaussian actor. In this case the network must take the observation signal as input and return both a mean value and a standard deviation value for each action. Therefore it must have two output layers (one for the mean values the other for the standard deviation values), each having as many elements as the dimension of the action space.

Note that standard deviations must be nonnegative and mean values must fall within the range of the action. Therefore the output layer that returns the standard deviations must be a softplus or ReLU layer, to enforce nonnegativity, while the output layer that returns the mean values must be a scaling layer, to scale the mean values to the output range.

% input path layerinPath = [ featureInputLayer(prod(obsInfo.Dimension),。。。'Normalization','none','Name','状态'）完整连接的layer（10，'Name','ip_fc') reluLayer('Name','ip_relu') fullyConnectedLayer(1,'Name','ip_out') ];% path layers for mean valuemeanPath = [ fullyConnectedLayer(15,'Name','mp_fc1') reluLayer('Name','mp_relu') fullyConnectedLayer(1,'Name','mp_fc2'）；tanhLayer('Name','tanh'）；scalingLayer('Name','mp_out',。。。'规模',actInfo.UpperLimit) ];% range: (-2N,2N)% path layers for standard deviationsdevPath = [ fullyConnectedLayer(15,'Name','VP_FC1') reluLayer('Name','vp_relu') fullyConnectedLayer(1,'Name','VP_FC2'）；SOFTPLESLAYER（'Name','vp_out') ];％范围：（0，+INF）％将图层添加到layergraph网络对象actornet = layergraph（inter）;actornet = addlayers（ActOrnet，Meanpath）;actornet = addlayers（actornet，sdevpath）;% connect layersactornet = connectlayers（actornet，'ip_out','mp_fc1/in'）；actornet = connectlayers（actornet，'ip_out','vp_fc1/in'）；% plot network情节（actornet）

Create the actor usingactorNet。PPO代理使用rlContinuousGaussianActorobject to implement the actor for continuous action spaces.

actor = rlContinuousGaussianActor(actorNet, obsInfo, actInfo,。。。“ actionMeanOutputnames”,'mp_out',。。。'ActionStandardDeviationOutputNames','vp_out',。。。'ObservationInputNames','状态'）；

Specify agent options, and create a TRPO agent using the actor, critic and agent options.

agentOpts = rlTRPOAgentOptions(。。。'ExperienceHorizon',1024,。。。“折扣器”,0.95,。。。``批评'',criticOpts); agent = rlTRPOAgent(actor,critic,agentOpts)

agent = rlTRPOAgent with properties: AgentOptions: [1x1 rl.option.rlTRPOAgentOptions] UseExplorationPolicy: 1 ObservationInfo: [1x1 rl.util.rlNumericSpec] ActionInfo: [1x1 rl.util.rlNumericSpec] SampleTime: 1

To check your agent, usegetActionto return the action from a random observation.

getAction(agent,{rand(2,1)})

ans =1x1 cell array{[0.6668]}

你现在可以测试和列车内的代理vironment.