generateRewardFunction
从控制规范生成奖励函数来训练强化学习代理
语法
描述
例子
从MPC对象生成一个奖励函数
这个例子展示了如何从MPC对象生成一个强化学习奖励函数。
定义工厂和创建MPC控制器
创建一个随机的植物rss函数并将馈通矩阵设置为零。
植物= rss(4,3,2);工厂。D = 0;
指定哪些设备信号是操作变量、测量扰动、测量输出和未测量输出。
植物= setmpcsignals(植物，“MV”, 1“医学博士”，2,“莫”(1 - 2),“UO”3);
创建一个MPC控制器，采样时间为0.1，预测和控制范围分别为10步和3步。
Mpcobj = mpc(plant,0.1,10,3);
- - - >“权重。属性为空。假设默认值为0.00000。- - - >“权重。属性为空。假设默认为0.10000。- - - >“权重。OutputVariables属性为空。假设默认值为1.00000。对于输出y1和零权重的输出y2 y3
为被操纵的变量设置限制和比例因子。
mpcobj.ManipulatedVariables.Min = -2;mpcobj.ManipulatedVariables.Max = 2;mpcobj.ManipulatedVariables.ScaleFactor = 4;
为二次代价函数设置权重。
mpcobj.Weights.OutputVariables = [10 1 0.1];mpcobj.Weights.ManipulatedVariablesRate = 0.2;
生成奖励函数
的规范中生成奖励函数代码货币政策委员会对象使用generateRewardFunction．代码显示在MATLAB编辑器中。
generateRewardFunction (mpcobj)
在本例中，将the保存在MATLAB函数文件中myMpcRewardFcn.m．显示生成的奖励函数。
类型myMpcRewardFcn.m
function reward = myMpcRewardFcn(y,refy,mv,refmv,lastmv) % myMpcRewardFcn根据MPC规格生成奖励。% %输入参数描述:% % y:第k步植物的输出变量+ 1% refy:第k步的参考输出变量+ 1% mv:第k步的操作变量% refmv:第k步的参考操作变量% lastmv:第k步的操作变量% % %限制(MPC和NLMPC): % -基于预测区间第一步计算的奖励。因此，信号预览和控制水平设置被忽略。% -不支持在线成本和约束更新。万博1manbetx% -不考虑自定义成本和约束规格。% -不支持随时间变化的成本权重和约束。万博1manbetx% -不考虑混合约束规范(对于MPC情况)。% MPC对象的规范%在'States'， 'OutputVariables'和% 'ManipulatedVariables'属性中指定的标准线性界限ymin = [-Inf -Inf -Inf];ymax = [Inf Inf Inf];Mvmin = -2; mvmax = 2; mvratemin = -Inf; mvratemax = Inf; % Scale factors as specified in 'States', 'OutputVariables', and % 'ManipulatedVariables' properties Sy = [1 1 1]; Smv = 4; % Standard cost weights as specified in 'Weights' property Qy = [10 1 0.1]; Qmv = 0; Qmvrate = 0.2; %% Compute cost dy = (refy(:)-y(:)) ./ Sy'; dmv = (refmv(:)-mv(:)) ./ Smv'; dmvrate = (mv(:)-lastmv(:)) ./ Smv'; Jy = dy' * diag(Qy.^2) * dy; Jmv = dmv' * diag(Qmv.^2) * dmv; Jmvrate = dmvrate' * diag(Qmvrate.^2) * dmvrate; Cost = Jy + Jmv + Jmvrate; %% Penalty function weight (specify nonnegative) Wy = [1 1 1]; Wmv = 10; Wmvrate = 10; %% Compute penalty % Penalty is computed for violation of linear bound constraints. % % To compute exterior bound penalty, use the exteriorPenalty function and % specify the penalty method as 'step' or 'quadratic'. % % Alternaltively, use the hyperbolicPenalty or barrierPenalty function for % computing hyperbolic and barrier penalties. % % For more information, see help for these functions. % % Set Pmv value to 0 if the RL agent action specification has % appropriate 'LowerLimit' and 'UpperLimit' values. Py = Wy * exteriorPenalty(y,ymin,ymax,'step'); Pmv = Wmv * exteriorPenalty(mv,mvmin,mvmax,'step'); Pmvrate = Wmvrate * exteriorPenalty(mv-lastmv,mvratemin,mvratemax,'step'); Penalty = Py + Pmv + Pmvrate; %% Compute reward reward = -(Cost + Penalty); end
计算出的奖励仅依赖于植物输入输出信号的当前值及其参考值，由两部分组成。
第一种是负成本，它取决于期望和当前工厂投入和产出之间的差的平方。这部分使用MPC对象中指定的代价函数权重。第二部分是惩罚，在当前植物信号违反约束时作为负奖励。
生成的奖励函数是奖励设计的起点。您可以调整权重或使用不同的惩罚函数来为您的强化学习代理定义更合适的奖励。
从验证块生成奖励函数
这个例子展示了如何从Simulink Design Optimization模型验证块生成一个强化学习奖励函数。万博1manbetx
对于本例，打开Simulink模型万博1manbetxLevelCheckBlock.slx，其中包含检查步骤响应特性块命名水平检查．
open_system (“LevelCheckBlock”）
的规范中生成奖励函数代码水平检查块,用generateRewardFunction．代码显示在MATLAB编辑器中。
generateRewardFunction (“LevelCheckBlock /水平检查”）
对于本例，代码保存在MATLAB函数文件中myBlockRewardFcn.m．
显示生成的奖励函数。
类型myBlockRewardFcn.m
function reward = myBlockRewardFcn(x,t) % myBlockRewardFcn根据Simulink块规格生成奖励。万博1manbetx%% x: Input of LevelCheckBlock/Level Check % t: Simulation time (s) % Reinforcement Learning Toolbox % 27-May-2021 16:45:27 %#codegen %% specification from LevelCheckBlock/Level Check Block1_InitialValue = 1;Block1_FinalValue = 2;Block1_StepTime = 0;Block1_StepRange = Block1_FinalValue - Block1_InitialValue;Block1_MinRise = Block1_InitialValue + Block1_StepRange * 80/100;block1_maxsettlement = Block1_InitialValue + Block1_StepRange * (1+2/100);block1_minsettlement = Block1_InitialValue + Block1_StepRange * (1-2/100);Block1_MaxOvershoot = Block1_InitialValue + Block1_StepRange * (1+10/100);Block1_MinUndershoot = Block1_InitialValue - Block1_StepRange * 5/100; if t >= Block1_StepTime if Block1_InitialValue <= Block1_FinalValue Block1_UpperBoundTimes = [0,5; 5,max(5+1,t+1)]; Block1_UpperBoundAmplitudes = [Block1_MaxOvershoot,Block1_MaxOvershoot; Block1_MaxSettling,Block1_MaxSettling]; Block1_LowerBoundTimes = [0,2; 2,5; 5,max(5+1,t+1)]; Block1_LowerBoundAmplitudes = [Block1_MinUndershoot,Block1_MinUndershoot; Block1_MinRise,Block1_MinRise; Block1_MinSettling,Block1_MinSettling]; else Block1_UpperBoundTimes = [0,2; 2,5; 5,max(5+1,t+1)]; Block1_UpperBoundAmplitudes = [Block1_MinUndershoot,Block1_MinUndershoot; Block1_MinRise,Block1_MinRise; Block1_MinSettling,Block1_MinSettling]; Block1_LowerBoundTimes = [0,5; 5,max(5+1,t+1)]; Block1_LowerBoundAmplitudes = [Block1_MaxOvershoot,Block1_MaxOvershoot; Block1_MaxSettling,Block1_MaxSettling]; end Block1_xmax = zeros(1,size(Block1_UpperBoundTimes,1)); for idx = 1:numel(Block1_xmax) tseg = Block1_UpperBoundTimes(idx,:); xseg = Block1_UpperBoundAmplitudes(idx,:); Block1_xmax(idx) = interp1(tseg,xseg,t,'linear',NaN); end if all(isnan(Block1_xmax)) Block1_xmax = Inf; else Block1_xmax = max(Block1_xmax,[],'omitnan'); end Block1_xmin = zeros(1,size(Block1_LowerBoundTimes,1)); for idx = 1:numel(Block1_xmin) tseg = Block1_LowerBoundTimes(idx,:); xseg = Block1_LowerBoundAmplitudes(idx,:); Block1_xmin(idx) = interp1(tseg,xseg,t,'linear',NaN); end if all(isnan(Block1_xmin)) Block1_xmin = -Inf; else Block1_xmin = max(Block1_xmin,[],'omitnan'); end else Block1_xmin = -Inf; Block1_xmax = Inf; end %% Penalty function weight (specify nonnegative) Weight = 1; %% Compute penalty % Penalty is computed for violation of linear bound constraints. % % To compute exterior bound penalty, use the exteriorPenalty function and % specify the penalty method as 'step' or 'quadratic'. % % Alternaltively, use the hyperbolicPenalty or barrierPenalty function for % computing hyperbolic and barrier penalties. % % For more information, see help for these functions. Penalty = sum(exteriorPenalty(x,Block1_xmin,Block1_xmax,'step')); %% Compute reward reward = -Weight * Penalty; end
所生成的奖励函数以验证块输入信号的当前值和仿真时间作为输入参数。当当前块输入信号违反验证块中定义的线性约束时，使用加权惩罚来计算负奖励。
生成的奖励函数是奖励设计的起点。您可以调整权重或使用不同的惩罚函数来为您的强化学习代理定义更合适的奖励。
关闭Simulink万博1manbetx模型。
close_system (“LevelCheckBlock”）
输入参数
提示
默认情况下，外界罚函数exteriorPenalty用于计算惩罚。或者，要计算双曲惩罚和障碍惩罚，可以使用hyperbolicPenalty或barrierPenalty功能。
版本历史
R2021b中引入
另请参阅
功能
对象
您也可以从以下列表中选择一个网站: