SUMOwiki - User contributions [en]

SED:SED toolbox

2011-02-24T13:38:44Z

Kcrombec: /* Introduction */

== Introduction ==
[[Image:SED.png|150 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential ''Design of Experiments (DoE)''.
In traditional experimental design, all the design points are selected up front, before performing any (computer or real-life) experiment, and afterwards, no additional design points are selected. This traditional approach is prone to oversampling and/or undersampling, because it is often very difficult to estimate up front the required number of design points.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate an experimental design in a sequential way, i.e. one design point at a time, without having to provide the total number of design points in advance. This is called ''sequential experimental design (SED)''. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the ''intersite'' (maximin) and ''projected'' (non-collapsing) requirements.
The ''intersite distance'' is the smallest distance between two design points in the design space; this value should be as high as possible, in order to have the points spread out as evenly as possible.
In addition to the intersite distance, the projected distance is also important.
The ''projected distance'' is the smallest distance between all the points after they have been projected on one axes of the design space. This measure is especially important if the relative importance of the design parameters is unknown. E.g., if one of the design parameters does not influence the output behavior, two design points which only differ in this (irrelevant) parameter have the same behavior, and can be seen as the "same" design point. Thus, the projected distance must also be maximized.

All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements. If you prefer to learn by example, you can check out the examples directory in the distribution, which contains several applications and example problems for the toolbox.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints. Additionally, '''mc-intersite-projected-threshold''' and all other methods besides '''mc-intersite-projected''' need the corner points [-1,...,-1] and [1,...,1] to start, and if they violate the constraints they will still be selected. You can later request the design without these corner points using the getAllPointsWithoutInitialDesign() function, so this might not be an issue, but keep it in mind.

=== Quality vs speed ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

File:SED.png

2011-02-24T13:38:13Z

Kcrombec: SED Toolbox logo

SED Toolbox logo

SED:SED toolbox

2011-01-26T15:31:46Z

Kcrombec: /* Introduction */

== Introduction ==
[[Image:SED.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential ''Design of Experiments (DoE)''.
In traditional experimental design, all the design points are selected up front, before performing any (computer or real-life) experiment, and afterwards, no additional design points are selected. This traditional approach is prone to oversampling and/or undersampling, because it is often very difficult to estimate up front the required number of design points.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate an experimental design in a sequential way, i.e. one design point at a time, without having to provide the total number of design points in advance. This is called ''sequential experimental design (SED)''. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the ''intersite'' (maximin) and ''projected'' (non-collapsing) requirements.
The ''intersite distance'' is the smallest distance between two design points in the design space; this value should be as high as possible, in order to have the points spread out as evenly as possible.
In addition to the intersite distance, the projected distance is also important.
The ''projected distance'' is the smallest distance between all the points after they have been projected on one axes of the design space. This measure is especially important if the relative importance of the design parameters is unknown. E.g., if one of the design parameters does not influence the output behavior, two design points which only differ in this (irrelevant) parameter have the same behavior, and can be seen as the "same" design point. Thus, the projected distance must also be maximized.

All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements. If you prefer to learn by example, you can check out the examples directory in the distribution, which contains several applications and example problems for the toolbox.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints. Additionally, '''mc-intersite-projected-threshold''' and all other methods besides '''mc-intersite-projected''' need the corner points [-1,...,-1] and [1,...,1] to start, and if they violate the constraints they will still be selected. You can later request the design without these corner points using the getAllPointsWithoutInitialDesign() function, so this might not be an issue, but keep it in mind.

=== Quality vs speed ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-26T13:16:36Z

Kcrombec: /* Quick start guide */

== Introduction ==
[[Image:SED.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential ''Design of Experiments (DoE)''.
In traditional experimental design, all the design points are selected up front, before performing any (computer or real-life) experiment, and afterwards, no additional design points are selected. This traditional approach is prone to oversampling and/or undersampling, because it is often very difficult to estimate up front the required number of design points.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate an experimental design in a sequential way, i.e. one design point at a time, without having to provide the total number of design points in advance. This is called ''sequential experimental design (SED)''. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the ''intersite'' (maximin) and ''projected'' (non-collapsing) requirements.
The ''intersite distance'' is the smallest distance between two design points in the design space; this value should be as high as possible, in order to have the points spread out as evenly as possible.
In addition to the intersite distance, the projected distance is also important.
The ''projected distance'' is the smallest distance between all the points after they have been projected on one axis of the design space. This measure is especially important if the relative importance of the design parameters is unknown. E.g., if one of the design parameters does not influence the output behavior, two design points which only differ in this (irrelevant) parameter have the same behavior, and can be seen as the "same" design point. Thus, the projected distance must also be maximized.

All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements. If you prefer to learn by example, you can check out the examples directory in the distribution, which contains several applications and example problems for the toolbox.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints. Additionally, '''mc-intersite-projected-threshold''' and all other methods besides '''mc-intersite-projected''' need the corner points [-1,...,-1] and [1,...,1] to start, and if they violate the constraints they will still be selected. You can later request the design without these corner points using the getAllPointsWithoutInitialDesign() function, so this might not be an issue, but keep it in mind.

=== Quality vs speed ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED reference

2011-01-25T12:31:55Z

Kcrombec: /* metrics = seq.getMetrics() */

This page describes the different functions available in the SequentialDesign class, which is the main entry point for using the SED Toolbox. There are three ways to create an instance of this class, as described in the different use cases in the main documentation page. Additionally, several functions are available to make it easy for the user to generate, plot and get data about the design produced by your SequentialDesign instance. In the sections below, we describe each function that is available to the user.

== SequentialDesign ==

=== SequentialDesign(problemStruct) ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign(problemStruct, 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml', 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== [seq, newPoints] = seq.generatePoints(10) ===

Use the sequential design algorithm to generate an additional 10 points on top of the already generated points. Will return the new points as the second return parameter.

=== [seq, newPoints] = seq.generateTotalPoints(10) ===

Use the sequential design algorithm to generate a total of 10 points. If, for example, 6 points were previously generated, 4 additional points will be selected to get the total up to 10. Will return the new points as the second return parameter.

=== seq.getInitialDesign() ===

The initial design is a set of samples which is generated in advance, to get the sequential design algorithm started. In the SED Toolbox, these are kept as small as possible (most methods need at least 2 points to get going, so the initial design will typically be 2 points). Note that the initial design ''might not respect the constraints''. You can manually remove the initial points from the design, or you can request all the points excluding the initial design using the getAllPointsWithoutInitialDesign() function.

=== seq.getAllPoints() ===

Get all points generated thus far, including the initial design.

=== seq.getAllPointsWithoutInitialDesign() ===

Get all points generated thus far, excluding the initial design.

=== seq.plot() ===

Generate a plot of the design generated thus far. Will only work for 1-3D.

=== seq.getMetrics() ===

Will calculate some metrics about the quality of the design. Two metrics are calculated and plotted: the intersite distance (minimum distance between points) and the projected distance (minimum distance between points after being projected onto one of the axes). These can be used as a basis of comparison between designs.

=== metrics = seq.getMetrics() ===

Will return the same metrics as described above in a struct. Will not plot or print any data.

=== metrics = seq.getMetrics(points) ===

Calculate the same metrics as above, but then for the points provided as an argument instead of the design generated by the object. This function can be used to compare a design from another source against the design generated by the SED Toolbox.

=== seq = seq.updatePoints(newPoints, newValues) ===

When using LOLA-Voronoi, you need to provide the outputs produced through simulation after every call of generatePoints. This is required because LOLA-Voronoi uses the outputs to determine the optimal distribution of points. When using the other methods, you do not need to call updatePoints.

=== seq = seq.plotLive(true) ===

This will enable live plotting of sample generation for 2D designs. If this is enabled, after each point that is generated, a plot will be built that shows the current design. This nicely demonstrates how points are selected and distributed over the design space.

=== seq.save('file.txt') ===

Store the entire design in a text file called ''file.txt''. Can later be loaded again by calling data = load('file.txt').

SED:License terms

2011-01-24T14:40:30Z

Kcrombec:

[[Image:osilogo.jpg|90px|right|Open Source Initiative]]
[[Image:Agpl3.png|90px|right|AGPLv3]]

The SED Toolbox is available under a '''dual license''' model.

For '''non-commercial''' use, the toolbox is available under the [http://www.fsf.org/licensing/licenses/agpl-3.0.html GNU Affero General Public License version 3] (AGPLv3), an [http://www.opensource.org/ OSI] approved [http://en.wikipedia.org/wiki/Open_source open source] license.

For use in a '''commercial''' setting, a commercial license must be obtained.

In addition we require that any reference to the SED Toolbox be accompanied by the [[http://sumo.intec.ugent.be/?q=SED#sed_reference|corresponding SED publication]].

== License terms ==

For '''non-commercial''' use, this program is free software; you can redistribute it and/or modify it under the terms of the [http://www.fsf.org/licensing/licenses/agpl-3.0.html GNU Affero General Public License version 3] as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program; if not, see http://www.gnu.org/licenses or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA, or download the license from the following URL: [http://www.fsf.org/licensing/licenses/agpl-3.0.html http://www.fsf.org/licensing/licenses/agpl-3.0.html]

In accordance with Section 7(b) of the GNU Affero General Public License, these Appropriate Legal Notices must retain the display of the "'''SED Toolbox'''" text and homepage. In addition, when mentioning the program in written work, reference must be made to the [[http://sumo.intec.ugent.be/?q=SED#sed_reference|corresponding SED publication]].

You can be released from these requirements by purchasing a '''commercial''' license.
Buying such a license is in most cases mandatory as soon as you develop commercial activities involving the SED Toolbox software. Commercial activities include: consultancy services or using the SED Toolbox in commercial projects (standalone, on a server, through a webservice or other remote access technology).

For details about a commercial license please [[contact]] us.

SED:License terms

2011-01-24T14:39:57Z

Kcrombec: New page: Open Source Initiative AGPLv3 The SED Toolbox is available under a '''dual license''' model. For '''non-c...

[[Image:osilogo.jpg|90px|right|Open Source Initiative]]
[[Image:Agpl3.png|90px|right|AGPLv3]]

The SED Toolbox is available under a '''dual license''' model.

For '''non-commercial''' use, the toolbox is available under the [http://www.fsf.org/licensing/licenses/agpl-3.0.html GNU Affero General Public License version 3] (AGPLv3), an [http://www.opensource.org/ OSI] approved [http://en.wikipedia.org/wiki/Open_source open source] license.

For use in a '''commercial''' setting, a commercial license must be obtained.

In addition we require that any reference to the blindDACE Toolbox be accompanied by the [[http://sumo.intec.ugent.be/?q=SED#sed_reference|corresponding blindDACE publication]].

== License terms ==

For '''non-commercial''' use, this program is free software; you can redistribute it and/or modify it under the terms of the [http://www.fsf.org/licensing/licenses/agpl-3.0.html GNU Affero General Public License version 3] as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program; if not, see http://www.gnu.org/licenses or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA, or download the license from the following URL: [http://www.fsf.org/licensing/licenses/agpl-3.0.html http://www.fsf.org/licensing/licenses/agpl-3.0.html]

In accordance with Section 7(b) of the GNU Affero General Public License, these Appropriate Legal Notices must retain the display of the "'''SED Toolbox'''" text and homepage. In addition, when mentioning the program in written work, reference must be made to the [[http://sumo.intec.ugent.be/?q=SED#sed_reference|corresponding blindDACE publication]].

You can be released from these requirements by purchasing a '''commercial''' license.
Buying such a license is in most cases mandatory as soon as you develop commercial activities involving the blindDACE Toolbox software. Commercial activities include: consultancy services or using the SED Toolbox in commercial projects (standalone, on a server, through a webservice or other remote access technology).

For details about a commercial license please [[contact]] us.

SED:SED toolbox

2011-01-24T14:22:57Z

Kcrombec: /* TODO */

== Introduction ==
[[Image:SED.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential ''Design of Experiments (DoE)''.
In traditional experimental design, all the design points are selected up front, before performing any (computer or real-life) experiment, and afterwards, no additional design points are selected. This traditional approach is prone to oversampling and/or undersampling, because it is often very difficult to estimate up front the required number of design points.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate an experimental design in a sequential way, i.e. one design point at a time, without having to provide the total number of design points in advance. This is called ''sequential experimental design (SED)''. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the ''intersite'' (maximin) and ''projected'' (non-collapsing) requirements.
The ''intersite distance'' is the smallest distance between two design points in the design space; this value should be as high as possible, in order to have the points spread out as evenly as possible.
In addition to the intersite distance, the projected distance is also important.
The ''projected distance'' is the smallest distance between all the points after they have been projected on one axis of the design space. This measure is especially important if the relative importance of the design parameters is unknown. E.g., if one of the design parameters does not influence the output behavior, two design points which only differ in this (irrelevant) parameter have the same behavior, and can be seen as the "same" design point. Thus, the projected distance must also be maximized.

All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints. Additionally, '''mc-intersite-projected-threshold''' and all other methods besides '''mc-intersite-projected''' need the corner points [-1,...,-1] and [1,...,1] to start, and if they violate the constraints they will still be selected. You can later request the design without these corner points using the getAllPointsWithoutInitialDesign() function, so this might not be an issue, but keep it in mind.

=== Quality vs speed ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED reference

2011-01-24T14:22:42Z

Kcrombec: /* SequentialDesign */

This page describes the different functions available in the SequentialDesign class, which is the main entry point for using the SED Toolbox. There are three ways to create an instance of this class, as described in the different use cases in the main documentation page. Additionally, several functions are available to make it easy for the user to generate, plot and get data about the design produced by your SequentialDesign instance. In the sections below, we describe each function that is available to the user.

== SequentialDesign ==

=== SequentialDesign(problemStruct) ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign(problemStruct, 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml', 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== [seq, newPoints] = seq.generatePoints(10) ===

Use the sequential design algorithm to generate an additional 10 points on top of the already generated points. Will return the new points as the second return parameter.

=== [seq, newPoints] = seq.generateTotalPoints(10) ===

Use the sequential design algorithm to generate a total of 10 points. If, for example, 6 points were previously generated, 4 additional points will be selected to get the total up to 10. Will return the new points as the second return parameter.

=== seq.getInitialDesign() ===

The initial design is a set of samples which is generated in advance, to get the sequential design algorithm started. In the SED Toolbox, these are kept as small as possible (most methods need at least 2 points to get going, so the initial design will typically be 2 points). Note that the initial design ''might not respect the constraints''. You can manually remove the initial points from the design, or you can request all the points excluding the initial design using the getAllPointsWithoutInitialDesign() function.

=== seq.getAllPoints() ===

Get all points generated thus far, including the initial design.

=== seq.getAllPointsWithoutInitialDesign() ===

Get all points generated thus far, excluding the initial design.

=== seq.plot() ===

Generate a plot of the design generated thus far. Will only work for 1-3D.

=== metrics = seq.getMetrics() ===

Will calculate some metrics about the quality of the design. Two metrics are calculated and plotted: the intersite distance (minimum distance between points) and the projected distance (minimum distance between points after being projected onto one of the axes). These can be used as a basis of comparison between designs. Will optionally return the values in a struct.

=== metrics = seq.getMetrics(points) ===

Calculate the same metrics as above, but then for the points provided as an argument instead of the design generated by the object. This function can be used to compare a design from another source against the design generated by the SED Toolbox.

=== seq = seq.updatePoints(newPoints, newValues) ===

When using LOLA-Voronoi, you need to provide the outputs produced through simulation after every call of generatePoints. This is required because LOLA-Voronoi uses the outputs to determine the optimal distribution of points. When using the other methods, you do not need to call updatePoints.

=== seq = seq.plotLive(true) ===

This will enable live plotting of sample generation for 2D designs. If this is enabled, after each point that is generated, a plot will be built that shows the current design. This nicely demonstrates how points are selected and distributed over the design space.

=== seq.save('file.txt') ===

Store the entire design in a text file called ''file.txt''. Can later be loaded again by calling data = load('file.txt').

SED:SED reference

2011-01-24T13:51:32Z

Kcrombec: /* SequentialDesign */

This page describes the different functions available in the SequentialDesign class, which is the main entry point for using the SED Toolbox. There are three ways to create an instance of this class, as described in the different use cases in the main documentation page. Additionally, several functions are available to make it easy for the user to generate, plot and get data about the design produced by your SequentialDesign instance. In the sections below, we describe each function that is available to the user.

== SequentialDesign ==

=== SequentialDesign(problemStruct) ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign(problemStruct, 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml', 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== [seq, newPoints] = seq.generatePoints(10) ===

Use the sequential design algorithm to generate an additional 10 points on top of the already generated points. Will return the new points as the second return parameter.

=== [seq, newPoints] = seq.generateTotalPoints(10) ===

Use the sequential design algorithm to generate a total of 10 points. If, for example, 6 points were previously generated, 4 additional points will be selected to get the total up to 10. Will return the new points as the second return parameter.

=== seq.getInitialDesign() ===

The initial design is a set of samples which is generated in advance, to get the sequential design algorithm started. In the SED Toolbox, these are kept as small as possible (most methods need at least 2 points to get going, so the initial design will typically be 2 points). Note that the initial design ''might not respect the constraints''. You can manually remove the initial points from the design, or you can request all the points excluding the initial design using the getAllPointsWithoutInitialDesign() function.

=== seq.getAllPoints() ===

Get all points generated thus far, including the initial design.

=== seq.getAllPointsWithoutInitialDesign() ===

Get all points generated thus far, excluding the initial design.

=== seq.plot() ===

Generate a plot of the design generated thus far. Will only work for 1-3D.

=== metrics = seq.getMetrics() ===

Will calculate some metrics about the quality of the design. Two metrics are calculated and plotted: the intersite distance (minimum distance between points) and the projected distance (minimum distance between points after being projected onto one of the axes). These can be used as a basis of comparison between designs. Will optionally return the values in a struct.

=== metrics = seq.getMetrics(points) ===

Calculate the same metrics as above, but then for the points provided as an argument instead of the design generated by the object. This function can be used to compare a design from another source against the design generated by the SED Toolbox.

=== seq = seq.updatePoints(newPoints, newValues) ===

When using LOLA-Voronoi, you need to provide the outputs produced through simulation after every call of generatePoints. This is required because LOLA-Voronoi uses the outputs to determine the optimal distribution of points. When using the other methods, you do not need to call updatePoints.

=== seq.save('file.txt') ===

Store the entire design in a text file called ''file.txt''. Can later be loaded again by calling data = load('file.txt').

SED:SED reference

2011-01-24T13:51:02Z

Kcrombec: /* SequentialDesign */

This page describes the different functions available in the SequentialDesign class, which is the main entry point for using the SED Toolbox. There are three ways to create an instance of this class, as described in the different use cases in the main documentation page. Additionally, several functions are available to make it easy for the user to generate, plot and get data about the design produced by your SequentialDesign instance. In the sections below, we describe each function that is available to the user.

== SequentialDesign ==

=== SequentialDesign(problemStruct) ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the default algorithm ('''mc-intersite-projected-threshold''') to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign(problemStruct, 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in problemStruct. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== seq = SequentialDesign('problem.xml', 'methods/method.xml') ===

Create a sequentual design object for the specified problem, as described in the problem.xml XML file. Uses the algorithm described in methods/method.xml to generate the design. For more information, see the [[SED:SED_toolbox#You_want_to_use_the_more_advanced_features_of_the_SED_Toolbox|main page]].

=== [seq, newPoints] = seq.generatePoints(10) ===

Use the sequential design algorithm to generate an additional 10 points on top of the already generated points. Will return the new points as the second return parameter.

=== [seq, newPoints] = seq.generateTotalPoints(10) ===

Use the sequential design algorithm to generate a total of 10 points. If, for example, 6 points were previously generated, 4 additional points will be selected to get the total up to 10. Will return the new points as the second return parameter.

=== seq.getInitialDesign() ===

The initial design is a set of samples which is generated in advance, to get the sequential design algorithm started. In the SED Toolbox, these are kept as small as possible (most methods need at least 2 points to get going, so the initial design will typically be 2 points). Note that the initial design ''might not respect the constraints''. You can manually remove the initial points from the design, or you can request all the points excluding the initial design using the getAllPointsWithoutInitialDesign() function.

=== seq.getAllPoints() ===

Get all points generated thus far, including the initial design.

=== seq.getAllPointsWithoutInitialDesign() ===

Get all points generated thus far, excluding the initial design.

=== seq.plot() ===

Generate a plot of the design generated thus far. Will only work for 1-3D.

=== metrics = seq.getMetrics() ===

Will calculate some metrics about the quality of the design. Two metrics are calculated and plotted: the intersite distance (minimum distance between points) and the projected distance (minimum distance between points after being projected onto one of the axes). These can be used as a basis of comparison between designs. Will optionally return the values in a struct.

=== metrics = seq.getMetrics(points) ===

Calculate the same metrics as above, but then for the points provided as an argument instead of the design generated by the object. This function can be used to compare a design from another source against the design generated by the SED Toolbox.

=== seq = seq.updatePoints(newPoints, newValues) ===

When using LOLA-Voronoi, you need to provide the outputs produced through simulation after every call of generatePoints. This is required because LOLA-Voronoi uses the outputs to determine the optimal distribution of points. When using the other methods, you do not need to call updatePoints.

=== seq.save('file.txt') ===

Store the entire design in a text file called ''file.txt''. Can later be loaded again by calling data = load('file.txt').

SED:SED toolbox

2011-01-24T13:34:06Z

Kcrombec: /* Constraints */

== Introduction ==
[[Image:SED.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential ''Design of Experiments (DoE)''.
In traditional experimental design, all the design points are selected up front, before performing any (computer or real-life) experiment, and afterwards, no additional design points are selected. This traditional approach is prone to oversampling and/or undersampling, because it is often very difficult to estimate up front the required number of design points.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate an experimental design in a sequential way, i.e. one design point at a time, without having to provide the total number of design points in advance. This is called ''sequential experimental design (SED)''. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the ''intersite'' (maximin) and ''projected'' (non-collapsing) requirements.
The ''intersite distance'' is the smallest distance between two design points in the design space; this value should be as high as possible, in order to have the points spread out as evenly as possible.
In addition to the intersite distance, the projected distance is also important.
The ''projected distance'' is the smallest distance between all the points after they have been projected on one axis of the design space. This measure is especially important if the relative importance of the design parameters is unknown. E.g., if one of the design parameters does not influence the output behavior, two design points which only differ in this (irrelevant) parameter have the same behavior, and can be seen as the "same" design point. Thus, the projected distance must also be maximized.

All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints. Additionally, '''mc-intersite-projected-threshold''' and all other methods besides '''mc-intersite-projected''' need the corner points [-1,...,-1] and [1,...,1] to start, and if they violate the constraints they will still be selected. You can later request the design without these corner points using the getAllPointsWithoutInitialDesign() function, so this might not be an issue, but keep it in mind.

=== Quality vs speed ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== TODO ==
TODO:
* mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
* optimizer-projected unsupported
* zie selectie van punten gebeuren met optimizer-intersite

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED reference

2011-01-24T13:18:21Z

Kcrombec: /* getInitialDesign() */

SED:SED reference

2011-01-24T13:17:32Z

Kcrombec: /* SequentialDesign */

SED:SED reference

2011-01-24T13:00:27Z

Kcrombec: /* SequentialDesign(problemStruct) */

SED:SED reference

2011-01-24T13:00:04Z

Kcrombec: /* SequentialDesign */

SED:SED reference

2011-01-24T12:58:05Z

Kcrombec:

SED:SED reference

2011-01-24T12:56:23Z

Kcrombec: New page: This page describes the different functions available in the SequentialDesign class, which is the main entry point for using the SED Toolbox. == SequentialDesign ==

This page describes the different functions available in the SequentialDesign class, which is the main entry point for using the SED Toolbox.

== SequentialDesign ==

SED:SED toolbox

2011-01-20T16:36:46Z

Kcrombec: /* TODO */

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints.

=== Quality vs speed ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== TODO ==
TODO:
* mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
* optimizer-projected unsupported
* zie selectie van punten gebeuren met optimizer-intersite

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:36:24Z

Kcrombec: /* Quality vs time cost */

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints.

=== Quality vs speed ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:36:10Z

Kcrombec: /* Rules of thumb for selecting the right sequential design method */

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

=== Constraints ===

the default method '''mc-intersite-projected-threshold''' can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, '''mc-intersite-projected''' can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints.

=== Quality vs time cost ===

The slowest method available in SED is '''optimizer-intersite''', but this method also generates the best designs (slightly better than '''mc-intersite-projected-threshold'''). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For '''mc-intersite-projected-threshold''', the ''candidatesPerSample'' option can be increased to improve the quality at the cost of speed. For '''optimizer-intersite''', both the ''nPop'' and ''maxIterations'' options can be increased.

=== Dimensionality ===

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. '''optimizer-intersite''' should work up to 10D, but will run into memory problems for higher dimensions.

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:18:17Z

Kcrombec: /* Rules of thumb for selecting the right sequential design method */

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

{| class="wikitable" border="1" cellspacing="5"
|-
! Method
! Use when...
|-
| '''mc-intersite-projected-threshold'''
| todo
|-
| '''mc-intersite-projected'''
| todo
|-
| '''optimizer-intersite'''
| tiodo
|}

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:17:21Z

Kcrombec:

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

{| class="wikitable"
|-
! Method
! Use when...
|-
| '''mc-intersite-projected-threshold'''
| todo
|-
| '''mc-intersite-projected'''
| todo
|-
| '''optimizer-intersite'''
| tiodo
|}

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:16:55Z

Kcrombec: /* You want full control over all the method parameters */

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at '''Karel dot Crombecq at ua.ac.be'''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

{| class="wikitable"
|-
! Method
! Use when...
|-
| ''mc-intersite-projected-threshold''
| todo
|-
| ''mc-intersite-projected''
| todo
|-
| ''optimizer-intersite''
| tiodo
|}

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:16:39Z

Kcrombec: /* Rules of thumb for selecting the right sequential design method */

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at ''Karel dot Crombecq at ua.ac.be''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

{| class="wikitable"
|-
! Method
! Use when...
|-
| ''mc-intersite-projected-threshold''
| todo
|-
| ''mc-intersite-projected''
| todo
|-
| ''optimizer-intersite''
| tiodo
|}

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:16:23Z

Kcrombec:

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at ''Karel dot Crombecq at ua.ac.be''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

{| class="wikitable"
|-
! Method
! Use when...
|-
| mc-intersite-projected-threshold
| todo
|-
| mc-intersite-projected
| todo
|-
| optimizer-intersite
| tiodo
|}

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:08:46Z

Kcrombec:

== Introduction ==
[[Image:blindDACE.gif|250 px|right|SED Toolbox]]

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential design of experiments. In traditional design of experiments, the all the design points are selected at once, and all the experiments are performed at once without selecting any additional design points. This method is prone to over- or undersampling, because it is often very difficult to predict the required number of design points in advance.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate a design of experiments one point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design. The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two points in the design; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they are projected on one of the axes. This is an important property if it is unknown up front how important each design parameters included in the experiment is. If one of the design parameters is irrelevant, two points which differ only in this value can be considered the same point. Thus, the projected distance must also be maximized. All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

== Download ==

See: [http://sumo.intec.ugent.be/?q=SED download page]

== Quick start guide ==

'''IMPORTANT''': Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

<source lang="matlab">
startup
</source>

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements.

=== You want an ND design of X points ===

In order to quickly generate a good ND design in X points, you can use the following code:

<source lang="matlab">
startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design
</source>

=== You want to use the more advanced features of the SED Toolbox ===

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

<source lang="matlab">
% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');
</source>

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

<source lang="matlab">
config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design
</source>

=== You want full control over all the method parameters ===

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at ''Karel dot Crombecq at ua.ac.be''.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

== SED toolbox interface ==

A reference of all the functions available in the SED Toolbox can be found on [[SED:SED_reference|this page]].

== Rules of thumb for selecting the right sequential design method ==

The default sequential design method for the SED Toolbox is ''mc-intersite-projected-threshold.xml''. This is an intelligent Monte Carlo method which performs a trade-off.

== TODO ==
TODO:
- problemen:
- mc-intersite-projected-th en optimizer-intersite kunnen zonder kandidaten vallen
- optimizer-projected unsupported
- zie selectie van punten gebeuren met optimizer-intersite

--TO BE UPDATED--

two scripts dacefit.m and predictor.m that emulate the behavior of the DACE toolbox ([http://www2.imm.dtu.dk/~hbn/dace/]). Note, that full compatibility between blindDACE and the DACE toolbox is not provided. The scripts merely aim to ease the transition from the DACE toolbox to the blindDACE toolbox.

Example code:
<source lang="matlab">
krige = dacefit(samples, values, 'regpoly0', 'corrgauss', theta0, lb, ub )
y = predictor([1 2], krige)
</source>

Obviously, a lot less code is used to copy the setup described above. However, less code means less flexibility (e.g., blind kriging and regression kriging are not available using the wrapper scripts). Hence, it is suggested to learn the object oriented interface of SED and use it instead.

== Contribute ==

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the [[Feedback | feedback]] page.

SED:SED toolbox

2011-01-20T16:00:03Z

Kcrombec: /* SED toolbox interface */

SED:SED toolbox

2011-01-20T15:57:22Z

Kcrombec:

SED:SED toolbox

2011-01-20T14:32:44Z

Kcrombec:

SED:SED toolbox

2011-01-20T14:23:52Z

Kcrombec:

Configuration

2010-08-30T10:14:24Z

Kcrombec:

The full behavior of the SUMO Toolbox is configurable through 2 [[FAQ#What_is_XML.3F| XML]] files:

* [[Toolbox configuration]] xml file: all the options related to the toolbox itself, which model types to use, how to score models, what sample selection strategy to use, etc. All options related to the 'modeling'.
** The [[Config:ToolboxConfiguration|default configuration]] can be used as a starting point to the several components of the toolbox. A tool is available called the [[Config:ToolboxConfigurationGUI|SUMO configuration editor]] that provides a GUI for editing the default configuration. It is advised for new users to use this tool to set up their experiment, as it avoids having to learn to work with the XML structure.
* [[Simulator configuration]] xml file: defines the interface with the [[simulator]], how many inputs, how many outputs, ranges, name of the executable or dataset, etc. This file is found inside each project directory in the <code>examples/</code> subdirectory. If you want to add your own, please see [[Adding an example]].

The Toolbox configuration file will contain the name of the simulator project to use (specified in the <code><Simulator></code> tag). For information how to run different examples see [[Running#Running_different_examples]].

Config:SequentialDesign

2009-09-28T13:09:41Z

Kcrombec:

'''This wiki page is based on version 6.2 of the SUMO Toolbox.'''

''We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower. We are are a university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files. If something is unclear please dont hesitate to [[Reporting problems|ask]].''
== SampleSelector ==

=== empty ===
Dont select any new samples, useful when modeling multiple outputs in paralel
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#EmptySampleSelector|EmptySampleSelector]]" combineOutputs="false"/>
</source>
=== random ===
Each sampling iterations new samples are selected randomly
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#RandomSampleSelector|RandomSampleSelector]]" combineOutputs="false"/>
</source>
=== delaunay ===
An adaptive sample selection algorithm that does a trade-off between error and density
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#DelaunaySampleSelector|DelaunaySampleSelector]]" combineOutputs="false">

<Option key="sampleSelect" value="all"/>

<Option key="nLastModels" value="2"/>

<Option key="scoreFunction" value="weightedLinear"/>
<Option key="lambda" value="0.5"/>
<Option key="mu" value="0.5"/>

<Option key="volumeScaling" value="max"/>
<Option key="differenceScaling" value="capmax"/>

<Option key="snapToEdge" value="enable"/>
<Option key="snapThreshold" value=".2"/>
</[[Config:SampleSelector|SampleSelector]]>
</source>
=== density ===
A simple density based sample selection algorithm
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#DensitySampleSelector|DensitySampleSelector]]" combineOutputs="false"/>
</source>
=== error ===
An adaptive sample selection algorithm (error based), driven by the evaluation of your model on a dense grid
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#ErrorSampleSelector|ErrorSampleSelector]]" combineOutputs="false">

<Option key="nLastModels" value="4"/>

<Option key="differenceScaling" value="none"/>

<Option key="gridSize" value="50"/>

<Option key="maxGridSize" value="100000"/>

<Option key="closenessThreshold" value="0.2"/>

<Option key="randomPercentage" value="20"/>
</[[Config:SampleSelector|SampleSelector]]>
</source>
=== lola ===
A highly adaptive sampling algorithm, error and density based
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#LOLASampleSelector|LOLASampleSelector]]" combineOutputs="false">

<Option key="neighbourhoodSize" value="2"/>
</[[Config:SampleSelector|SampleSelector]]>
</source>
=== autoSampling ===
A wrapper around another sample selector that filters out the auto-sampled dimensions. This is useful if one or more inputs are sampled automatically by the simulator.
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#AutoSamplingSampleSelector|AutoSamplingSampleSelector]]" combineOutputs="false">


<Option key="function" value="max"/>


<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#LOLASampleSelector|LOLASampleSelector]]" combineOutputs="false">
<Option key="neighbourhoodSize" value="2"/>
</[[Config:SampleSelector|SampleSelector]]>

</[[Config:SampleSelector|SampleSelector]]>
</source>
=== rationalPoleSupression ===
A sampling algorithm aimed at supressing poles by sampling them (only for Rational models)
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#OptimizeCriterion|OptimizeCriterion]]" combineOutputs="false">


<Option key="criterion" value="rationalPoleSupression"/> 


<[[Config:Optimizer|Optimizer]] type="[[Optimizer#DirectOptimizer|DirectOptimizer]]">
<Option key="maxevals" value="1000"/>
<Option key="maxits" value="300"/>
</[[Config:Optimizer|Optimizer]]>


<Option key="debug" value="off"/>
</[[Config:SampleSelector|SampleSelector]]>
</source>
=== expectedImprovement ===
A sampling algorithm aimed at optimization problems (only for Kriging and RBF)
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#OptimizeCriterion|OptimizeCriterion]]" combineOutputs="false">


<Option key="criterion" value="gExpectedImprovement"/> 
<Option key="criterion_opts" value="1"/> 


<[[Config:Optimizer|Optimizer]] type="[[Optimizer#DirectOptimizer|DirectOptimizer]]">
<Option key="maxevals" value="1000"/>
<Option key="maxits" value="300"/>
</[[Config:Optimizer|Optimizer]]>


<Option key="debug" value="off"/>
</[[Config:SampleSelector|SampleSelector]]>
</source>
=== extremaLOLA ===
LOLA sample selector supplemented with 1 sample at the minimum and maximum
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#CombinedSampleSelector|CombinedSampleSelector]]" combineOutputs="false">

<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#LOLASampleSelector|LOLASampleSelector]]" combineOutputs="false">

<Option key="neighbourhoodSize" value="2"/>
</[[Config:SampleSelector|SampleSelector]]>

<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#OptimizeCriterion|OptimizeCriterion]]" combineOutputs="false">
<Option key="criterion" value="minmodel"/> 


<[[Config:Optimizer|Optimizer]] type="[[Optimizer#DirectOptimizer|DirectOptimizer]]">
<Option key="maxevals" value="1000"/>
<Option key="maxits" value="300"/>
</[[Config:Optimizer|Optimizer]]>

<Option key="debug" value="off"/>
</[[Config:SampleSelector|SampleSelector]]>

<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#OptimizeCriterion|OptimizeCriterion]]" combineOutputs="false">
<Option key="criterion" value="maxmodel"/> 


<[[Config:Optimizer|Optimizer]] type="[[Optimizer#DirectOptimizer|DirectOptimizer]]">
<Option key="maxevals" value="1000"/>
<Option key="maxits" value="300"/>
</[[Config:Optimizer|Optimizer]]>

<Option key="debug" value="off"/>
</[[Config:SampleSelector|SampleSelector]]>

</[[Config:SampleSelector|SampleSelector]]>
</source>
=== default ===
LOLA sample selector combined with error based sample selector (default)
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#CombinedSampleSelector|CombinedSampleSelector]]" combineOutputs="false">

<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#LOLASampleSelector|LOLASampleSelector]]" combineOutputs="false">

<Option key="neighbourhoodSize" value="2"/>
</[[Config:SampleSelector|SampleSelector]]>


<[[Config:SampleSelector|SampleSelector]] type="[[SampleSelector#ErrorSampleSelector|ErrorSampleSelector]]" combineOutputs="false">

<Option key="nLastModels" value="4"/>

<Option key="differenceScaling" value="none"/>

<Option key="gridSize" value="50"/>

<Option key="maxGridSize" value="100000"/>

<Option key="closenessThreshold" value="0.2"/>

<Option key="randomPercentage" value="0"/>
</[[Config:SampleSelector|SampleSelector]]>
</[[Config:SampleSelector|SampleSelector]]>
</source>

Add Sampling Algorithm

2009-09-28T13:09:28Z

Kcrombec:

'''This wiki page is based on version 6.2 of the SUMO Toolbox.'''

The toolbox comes with a number of sample selection algorithms, both for experimental design (initial samples) and sequential design. Of course you are free to add your own.

You can govern how new samples are selected in a number of ways. The easiest method is by implementing your own sample selector class that derives from the SampleSelector base class in src/matlab/sampleSelectors. Again, only two methods are needed:

* a constructor for reading in the configuration extracted from the XML file. See other sample selectors for the structure of this configuration.
* a selectSamples.m file that, given the toolbox state, returns the next batch of samples.

The toolbox state is a Matlab struct with the following fields:
* samples: the samples that were previously evaluated.
* values: the output that must be used to select new samples for.
* lastModels: the best models so far.
* numNewSamples: the amount of new samples that must be selected. This is based on environmental information such as the modeling time, the number of available computational nodes (cpu cores, grid nodes) and so on.

The SUMO Toolbox contains an extensive framework for sampling, which makes it possible to combine your sampling algorithm in a number of ways with the existing algorithms. Below, a number of options are discussed. Note that these are optional, and it is also possible to write your own sample selector as mentioned above, without paying attention to the options below.

= CombinedSampleSelector =

The combined sample selector can be used to have multiple different sample selectors sample the same output during one run of the toolbox. This can be useful if there are multiple criteria that you want to use together to select new samples. Different weights can be given to the sample selectors, so that each sample selector gets to sample a number of samples proportional to its weight. Sample code for combining 3 selectors with different weights can be found below.

Note the <MergeCriterion> tag. This tag defines the criterion which is used to combine the selected samples from the different sample selectors in one set of new samples. The ClosenessThreshold merge criterion simply filters out samples that are too close to each other. This avoids the problem that two sample selectors might select new samples really close to each other, thus evaluating two (almost) identical samples. The code from this example can be copied to your own CombinedSampleSelector as is.

<source xmlns:saxon="http://icl.com/saxon" lang="xml">


<SampleSelector id="combo" type="CombinedSampleSelector" combineOutputs="false">


<SampleSelector id="lola" type="LOLASampleSelector" combineOutputs="false" weight="0.5" />


<SampleSelector id="error" type="ErrorSampleSelector" combineOutputs="false" weight="0.3" />


<SampleSelector id="random" type="RandomSampleSelector" combineOutputs="false" weight="0.2" />


<MergeCriterion type="ClosenessThreshold">


<Option key="closenessThreshold" value="0.2"/>

<Option key="randomPercentage" value="0"/>

<Option key="debug" value="off" />
</MergeCriterion>

</SampleSelector>
</source>

= PipelineSampleSelector =

The pipeline sample selector is an extensive sampling framework used by several of the predefined sample selectors. It splits the sampling process up into three separate tasks, which are executed one after each other, to come to a final set of sample locations. In this section, we will briefly discuss the three different steps in the pipeline process. Please look into the existing sample selectors (such as delaunay and error) for example implementations.

== CandidateGenerator ==

The candidate generator is responsible for generating an initial set of candidate new samples. Out if this candidates, eventually, a number of new samples will be picked. Examples of candidate generators are a grid, a set of random points, etc. To implement your own candidate generator, you need only make a function with the following declaration:

<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
function [state, candidates] = MyCandidateGenerator(state)
</source>

== CandidateRanker ==

All the candidates, generated by the candidate generator, are ranked by one or more candidate rankers. These rankers give a score (or ranking) to all the candidates. To make your own candidate ranker, you have to derive your class from the CandidateRanker base class. A minimal example candidate ranker is shown below.

<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
classdef MyCandidateRanker < CandidateRanker
methods (Access = public)

function this = MyCandidateRanker(config)
this = this@CandidateRanker(config);
end

function ranking = scoreCandidates(this, candidates, state)
ranking = rand(size(candidates,1), 1);
end
end
end
</source>

== MergeCriterion ==

Finally, all the rankings provided by the different candidate rankers, are used to select the final set of new samples out of the candidate samples. The merge criterion has the task of somehow combining the different rankings, and using these rankings to select the most appropriate candidates. In addition to a set of new samples, the merge criterion also has to assign priority values to each sample. These priorities (high value means high priority) are used to determine in which order the samples are evaluated. A minimal example merge criterion is shown below.
<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
classdef MyMergeCriterion < MergeCriterion

methods (Access = public)

function [this] = MyMergeCriterion()
end

function [this, newSamples, priorities] = selectSamples(this, candidates, rankings, state)

newSamples = candidates(1:state.numNewSamples, :);
priorities = rand(state.numNewSamples,1);
end
end
end
</source>

Add Sampling Algorithm

2009-09-28T13:09:21Z

Kcrombec:

'''This wiki page is written for version 6.2 of the SUMO Toolbox.'''

The toolbox comes with a number of sample selection algorithms, both for experimental design (initial samples) and sequential design. Of course you are free to add your own.

You can govern how new samples are selected in a number of ways. The easiest method is by implementing your own sample selector class that derives from the SampleSelector base class in src/matlab/sampleSelectors. Again, only two methods are needed:

* a constructor for reading in the configuration extracted from the XML file. See other sample selectors for the structure of this configuration.
* a selectSamples.m file that, given the toolbox state, returns the next batch of samples.

The toolbox state is a Matlab struct with the following fields:
* samples: the samples that were previously evaluated.
* values: the output that must be used to select new samples for.
* lastModels: the best models so far.
* numNewSamples: the amount of new samples that must be selected. This is based on environmental information such as the modeling time, the number of available computational nodes (cpu cores, grid nodes) and so on.

The SUMO Toolbox contains an extensive framework for sampling, which makes it possible to combine your sampling algorithm in a number of ways with the existing algorithms. Below, a number of options are discussed. Note that these are optional, and it is also possible to write your own sample selector as mentioned above, without paying attention to the options below.

= CombinedSampleSelector =

The combined sample selector can be used to have multiple different sample selectors sample the same output during one run of the toolbox. This can be useful if there are multiple criteria that you want to use together to select new samples. Different weights can be given to the sample selectors, so that each sample selector gets to sample a number of samples proportional to its weight. Sample code for combining 3 selectors with different weights can be found below.

Note the <MergeCriterion> tag. This tag defines the criterion which is used to combine the selected samples from the different sample selectors in one set of new samples. The ClosenessThreshold merge criterion simply filters out samples that are too close to each other. This avoids the problem that two sample selectors might select new samples really close to each other, thus evaluating two (almost) identical samples. The code from this example can be copied to your own CombinedSampleSelector as is.

<source xmlns:saxon="http://icl.com/saxon" lang="xml">


<SampleSelector id="combo" type="CombinedSampleSelector" combineOutputs="false">


<SampleSelector id="lola" type="LOLASampleSelector" combineOutputs="false" weight="0.5" />


<SampleSelector id="error" type="ErrorSampleSelector" combineOutputs="false" weight="0.3" />


<SampleSelector id="random" type="RandomSampleSelector" combineOutputs="false" weight="0.2" />


<MergeCriterion type="ClosenessThreshold">


<Option key="closenessThreshold" value="0.2"/>

<Option key="randomPercentage" value="0"/>

<Option key="debug" value="off" />
</MergeCriterion>

</SampleSelector>
</source>

= PipelineSampleSelector =

The pipeline sample selector is an extensive sampling framework used by several of the predefined sample selectors. It splits the sampling process up into three separate tasks, which are executed one after each other, to come to a final set of sample locations. In this section, we will briefly discuss the three different steps in the pipeline process. Please look into the existing sample selectors (such as delaunay and error) for example implementations.

== CandidateGenerator ==

The candidate generator is responsible for generating an initial set of candidate new samples. Out if this candidates, eventually, a number of new samples will be picked. Examples of candidate generators are a grid, a set of random points, etc. To implement your own candidate generator, you need only make a function with the following declaration:

<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
function [state, candidates] = MyCandidateGenerator(state)
</source>

== CandidateRanker ==

All the candidates, generated by the candidate generator, are ranked by one or more candidate rankers. These rankers give a score (or ranking) to all the candidates. To make your own candidate ranker, you have to derive your class from the CandidateRanker base class. A minimal example candidate ranker is shown below.

<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
classdef MyCandidateRanker < CandidateRanker
methods (Access = public)

function this = MyCandidateRanker(config)
this = this@CandidateRanker(config);
end

function ranking = scoreCandidates(this, candidates, state)
ranking = rand(size(candidates,1), 1);
end
end
end
</source>

== MergeCriterion ==

Finally, all the rankings provided by the different candidate rankers, are used to select the final set of new samples out of the candidate samples. The merge criterion has the task of somehow combining the different rankings, and using these rankings to select the most appropriate candidates. In addition to a set of new samples, the merge criterion also has to assign priority values to each sample. These priorities (high value means high priority) are used to determine in which order the samples are evaluated. A minimal example merge criterion is shown below.
<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
classdef MyMergeCriterion < MergeCriterion

methods (Access = public)

function [this] = MyMergeCriterion()
end

function [this, newSamples, priorities] = selectSamples(this, candidates, rankings, state)

newSamples = candidates(1:state.numNewSamples, :);
priorities = rand(state.numNewSamples,1);
end
end
end
</source>

Add Sampling Algorithm

2009-09-28T12:57:46Z

Kcrombec:

The toolbox comes with a number of sample selection algorithms, both for experimental design (initial samples) and sequential design. Of course you are free to add your own.

You can govern how new samples are selected in a number of ways. The easiest method is by implementing your own sample selector class that derives from the SampleSelector base class in src/matlab/sampleSelectors. Again, only two methods are needed:

* a constructor for reading in the configuration extracted from the XML file. See other sample selectors for the structure of this configuration.
* a selectSamples.m file that, given the toolbox state, returns the next batch of samples.

The toolbox state is a Matlab struct with the following fields:
* samples: the samples that were previously evaluated.
* values: the output that must be used to select new samples for.
* lastModels: the best models so far.
* numNewSamples: the amount of new samples that must be selected. This is based on environmental information such as the modeling time, the number of available computational nodes (cpu cores, grid nodes) and so on.

The SUMO Toolbox contains an extensive framework for sampling, which makes it possible to combine your sampling algorithm in a number of ways with the existing algorithms. Below, a number of options are discussed. Note that these are optional, and it is also possible to write your own sample selector as mentioned above, without paying attention to the options below.

= CombinedSampleSelector =

The combined sample selector can be used to have multiple different sample selectors sample the same output during one run of the toolbox. This can be useful if there are multiple criteria that you want to use together to select new samples. Different weights can be given to the sample selectors, so that each sample selector gets to sample a number of samples proportional to its weight. Sample code for combining 3 selectors with different weights can be found below.

Note the <MergeCriterion> tag. This tag defines the criterion which is used to combine the selected samples from the different sample selectors in one set of new samples. The ClosenessThreshold merge criterion simply filters out samples that are too close to each other. This avoids the problem that two sample selectors might select new samples really close to each other, thus evaluating two (almost) identical samples. The code from this example can be copied to your own CombinedSampleSelector as is.

<source xmlns:saxon="http://icl.com/saxon" lang="xml">


<SampleSelector id="combo" type="CombinedSampleSelector" combineOutputs="false">


<SampleSelector id="lola" type="LOLASampleSelector" combineOutputs="false" weight="0.5" />


<SampleSelector id="error" type="ErrorSampleSelector" combineOutputs="false" weight="0.3" />


<SampleSelector id="random" type="RandomSampleSelector" combineOutputs="false" weight="0.2" />


<MergeCriterion type="ClosenessThreshold">


<Option key="closenessThreshold" value="0.2"/>

<Option key="randomPercentage" value="0"/>

<Option key="debug" value="off" />
</MergeCriterion>

</SampleSelector>
</source>

= PipelineSampleSelector =

The pipeline sample selector is an extensive sampling framework used by several of the predefined sample selectors. It splits the sampling process up into three separate tasks, which are executed one after each other, to come to a final set of sample locations. In this section, we will briefly discuss the three different steps in the pipeline process. Please look into the existing sample selectors (such as delaunay and error) for example implementations.

== CandidateGenerator ==

The candidate generator is responsible for generating an initial set of candidate new samples. Out if this candidates, eventually, a number of new samples will be picked. Examples of candidate generators are a grid, a set of random points, etc. To implement your own candidate generator, you need only make a function with the following declaration:

<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
function [state, candidates] = MyCandidateGenerator(state)
</source>

== CandidateRanker ==

All the candidates, generated by the candidate generator, are ranked by one or more candidate rankers. These rankers give a score (or ranking) to all the candidates. To make your own candidate ranker, you have to derive your class from the CandidateRanker base class. A minimal example candidate ranker is shown below.

<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
classdef MyCandidateRanker < CandidateRanker
methods (Access = public)

function this = MyCandidateRanker(config)
this = this@CandidateRanker(config);
end

function ranking = scoreCandidates(this, candidates, state)
ranking = rand(size(candidates,1), 1);
end
end
end
</source>

== MergeCriterion ==

Finally, all the rankings provided by the different candidate rankers, are used to select the final set of new samples out of the candidate samples. The merge criterion has the task of somehow combining the different rankings, and using these rankings to select the most appropriate candidates. In addition to a set of new samples, the merge criterion also has to assign priority values to each sample. These priorities (high value means high priority) are used to determine in which order the samples are evaluated. A minimal example merge criterion is shown below.
<source xmlns:saxon="http://icl.com/saxon" lang="matlab">
classdef MyMergeCriterion < MergeCriterion

methods (Access = public)

function [this] = MyMergeCriterion()
end

function [this, newSamples, priorities] = selectSamples(this, candidates, rankings, state)

newSamples = candidates(1:state.numNewSamples, :);
priorities = rand(state.numNewSamples,1);
end
end
end
</source>

Add Sampling Algorithm

2009-09-28T12:52:46Z

Kcrombec: /* PipelineSampleSelector */

Add Sampling Algorithm

2009-09-28T12:30:48Z

Kcrombec:

Toolbox configuration

2009-09-28T12:03:49Z

Kcrombec: /* Components */

The toolbox can be configured by means of an [[FAQ#What is XML?|XML]] file.
Examples can be found in the <code>config/</code> and <code>demo/</code> subdirectories of the SUMO installation directory.
The default configuration file is '''<code>config/default.xml</code>'''.

== Structure ==

If you do not know what a tag or XML is please see [[FAQ#What is XML?]] first.

=== Plans and Runs ===

The general structure of the toolbox is as follows:

* The top-level <[[Config:Plan|Plan]]> type defines a surrogate modeling experiment, and an experiment may consist of multiple <[[Config:Plan#Run|Run]]> tags.
* Each <[[Config:Plan#Run|Run]]> tag can be configured separately.

For example, say you want to model some problem from electronics and you have at your disposal 3 algorithms for selecting data points. Now lets assume you want to compare the different algorithms on your problem and see which one gives you the best model with the least number of data samples. In this case your <[[Config:Plan|Plan]]> tag would contain 3 <[[Config:Plan#Run|Run]]> tags and each <[[Config:Plan#Run|Run]]> tag would contain a different <[[SampleSelector|SampleSelector]]> tag. For example:

<source lang="xml">
<Plan>
...
<Run name="gradient-run" repeat="1">
<SampleSelector>gradient</SampleSelector>
</Run>
<Run name="grid-run" repeat="1">
<SampleSelector>grid</SampleSelector>
</Run>
<Run name="random-run" repeat="1">
<SampleSelector>random</SampleSelector>
</Run>
...
</Plan>
...
</source>

Thus, this concept of a plan and multiple runs allows you to setup different configurations beforehand and try them all in one go.

As you can see it is also possible to specify a '<code>repeat</code>' attribute. Setting it to 5, for example, will ensure that that particular run is repeated 5 times. This is usually a good idea if there is a lot of randomness in the algorithms (as is usually the case).

Remember though to set the '<code>[[Random_state|seedRandomState]]</code>' option in the [[Config:SUMO| <SUMO>]] tag to '<code>random</code>', or otherwise you will still get deterministic results:

<source lang="xml">
<Option key="seedRandomState" value="random"/>
</source>

=== Declarations and Definitions ===

Each component in the toolbox has its own configuration section. Inside the <[[Config:Plan#Run|Run]]> tag you ''declare'' what components you would like to use. This declaration refers to the ''definition'' of each component, further down the file. So when you see line like:

<source lang="xml">
<SampleSelector>lola</SampleSelector>
</source>

This means we want to use the '<code>lola</code>' sample selection algorithm, the word '<code>lola</code>' is a unique identifier that refers to the <[[SampleSelector|SampleSelector]]> tag that has '<code>[[SampleSelector#GradientSampleSelector|lola]]</code>' as its "id" attribute. In this case your configuration file would have the following structure:

<source lang="xml">
<Plan>
<Run>

<SampleSelector>lola</SampleSelector>
...
</Run>
...
<Plan>
...


<SampleSelector id="lola">
...

</SampleSelector>
...
</source>

If you would like to use a different algorithm (e.g., '<code>[[SampleSelector#ErrorSampleSelector|error]]</code>' for the Error sample selector), you simply fill in a different id in the the <[[SampleSelector|SampleSelector]]> tag in the <[[Config:Plan#Run|Run]]> tag:

<source lang="xml">
<SampleSelector>error</SampleSelector>
</source>

You just have to make sure there is a matching definition lower down the file for the id you have filled in.

All the other components ([[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]], [[Config:SampleEvaluator|SampleEvaluator]], ...) work in exactly the same way.

== Running a configuration ==

See the [[Running]] page for how to run the toolbox with a different example or with a your own configuration file.

== Components ==

''We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower. We are are a university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files. If something is unclear please dont hesitate to [[Reporting problems|ask]].''

The following components can be configured separately:

* [[Config:Plan|Plan]]
* [[Outputs]]
** [[Measures]]
* [[Config:Plan|Inputs]]
* [[Config:Plan|Simulator tag]]
* [[Config:ContextConfig|ContextConfig]]
* [[Config:SUMO|SUMO]]
** [[Config:InitialDesign|Initial Designs]]
* [[Config:SampleEvaluator|Sample Evaluators]]
* [[Config:SampleSelector|Sample Selectors]]
* [[Config:AdaptiveModelBuilder|Adaptive Model Builders]]

== General guidelines ==

Some general guidelines on how to configure the toolbox for different situations can be found on [[General guidelines|this page]].

General guidelines

2009-02-09T11:52:15Z

Kcrombec:

The <code>[[Config:ToolboxConfiguration|default.xml]]</code> file can be used as a starting point for default behavior for the SUMO Toolbox. If you are a new user, you should initially leave most options at their default values. The default settings were chosen since they produce good results on average.

However, usually the optimal choice of components depends on the problem itself, so that the default settings aren't necessarily the best. This page will give the user general guidelines to decide which component to use for each situation they may encounter. The user is of course free to ignore these rules and experiment with other settings.

Note this list is very brief and incomplete, feel free to [[Contact]] us if you have any further questions.

== Measures ==

The default [[Measures| Measure]] is [[Measures#CrossValidation| CrossValidation]]. Even though this is a very good, accurate, overall measure, there are some considerations to make in the following cases:

* '''Expensive modelers (ann):''' If it is relatively expensive to train a model (for example, with neural networks), CrossValidation is also very slow, because it has to train a model for each fold (which is 5 by default). If modeling takes too long, you might want to use a faster alternative, such as [[Measures#ValidationSet|ValidationSet]].
* '''ErrorSampleSelector:''' CrossValidation might give a biased result when combined with the [[SampleSelector#ErrorSampleSelector|ErrorSampleSelector]]. This is because the ErrorSampleSelector tends to cluster samples around one point, which will result in very accurate surrogate models for all the points in this cluster (and thus good results with CrossValidation ). So when using CrossValidation and ErrorSampleSelector together, keep in mind that the real accuracy might be slightly lower than the estimated one.
* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.

== Sample Selectors ==

The default [[SampleSelector|SampleSelector]] is the [[SampleSelector#LOLASampleSelector|LOLASampleSelector]] combined with the [[SampleSelector#ErrorSampleSelector|ErrorSampleSelector]], with a weight of 0.8 for LOLA and 0.2 for error. This is a very robust sample selector, capable of dealing with most situations. There are, however, some cases in which it is advisable to choose a different one:

* '''Large-scale problems (1000+ samples):''' The GradientSampleSelector's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, the GradientSampleSelector becomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with the GradientSampleSelector might still be negligible.
* '''Rational modeler:''' Benchmarks have shown that the gain of using the LOLASampleSelector over the [[SampleSelector#ErrorSampleSelector|ErrorSampleSelector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) ErrorSampleSelector when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.

When using the ErrorSampleSelector instead of the LOLASampleSelector, it is always a good idea to combine it with the [[SampleSelector#DensitySampleSelector|DensitySampleSelector]], to combat stability/robustness issues the ErrorSampleSelector often causes. It is a good idea to select about 60% of the samples with the ErrorSampleSelector, and 40% with the DensitySampleSelector. This will ensure that at least the entire design space is covered to a certain degree. This additional sample selector is NOT necessary when using the LOLASampleSelector.

== Adaptive Model Builders ==

The question that always gets asked is ''Which model type should I use for my data?'' Unfortunately there is no straightforward since it all depends on your problem: how many dimensions, how many points, is your function rugged, smooth, or both, is there noise, etc, etc. Based on this knowledge it is possible to say which model types are more likely to do well but it remains a heuristic. Best is to try a few and see what happens, or use the ''heterogenetic'' model builder to try multiple model types in parallel and automatically try to determine the best type.

InitialDesign

2009-01-16T09:55:03Z

Kcrombec:

== Latin Hypercube Design ==
Choose an initial sampleset in such a way that they form a latin hypercube.
{{OptionsHeader}}
{{Option
|name = points
|values = positive integer
|default = 30
|description = The amount of samples that will be taken.
}}

== Dataset Design ==
Reads an initial design from a dataset. Each row should contain one sample. The inputvalues come first. The outputvalues, if there are any, come second.

If only input values are supplied (hasOutputs=false), each value has to be in the fixed [-1,1] range (model space). The samples are later automatically transformed to their original range before they are passed to the sample evaluator. This is done to easily support different ranges for variables without having to change the initial dataset design.

If, however, output values are also provided (hasOutputs=true), the samples are instantly fed to the toolbox for modelling, since they don't have to be evaluated anymore. Because of this property, the inputs have to be in the original range defined in the simulator xml file, instead of the fixed [-1,1] range.

{{OptionsHeader}}
{{Option
|name = filename
|values = path
|default = none
|description = Path of the file that contains the dataset. '''This option is required!'''
}}
{{Option
|name = hasOutputs
|values = boolean
|default = true
|description = Does the datasetfile contain evaluated samples or just sample locations?
}}

== Combined Design ==

Combines two other initial designs.

{{NodesHeader}}
{{Node
|name = InitialDesign
|required = one or more
|description = Configuration for each InitialDesign Method.
}}

== Factorial Design ==
Choose an initial sampleset in such a way that they form an uniform grid
{{OptionsHeader}}
{{Option
|name = levels
|values = vector of positive integers > 1
|default = [5,...,5]
|description = vector specifying for each dimension the number of samples. Sampling level1*level2*...*leveln points. If a scalar s is specified, The toolbox assumes [s s ... s].
}}

Add Initial Design

2009-01-16T09:52:01Z

Kcrombec:

The toolbox comes with a number of sample selection algorithms, both for experimental design (initial samples) and sequential design. Of course you are free to add your own.

The available initial designs can be found in the folder src/matlab/initialDesigns. Adding a new type means creating a new class in this folder analogous to the existing ones (subclass InitialDesign). Two methods are required:

1. A constructor for reading in the configuration extracted from the XML file. Don't forget to call the constructor of the base class using the following syntax:

<code>
this = this@InitialDesign(config);
</code>

2. A generate method that returns the following values:

* A set of initial samples (input values only) to be evaluated. To easily support different ranges for variables, each sample has to be in a fixed [-1,1] range in each dimension. The samples are later automatically transformed to their original range before they are passed to the sample evaluator.

* An optional set of evaluated samples (inputs and outputs), which may come from any source (for example: a dataset). These samples are instantly fed to the toolbox for modelling, because they don't have to be evaluated anymore. Because of this property, they have to be in the original range defined in the simulator xml file, instead of the fixed [-1,1] range.

Add Initial Design

2009-01-16T09:51:52Z

Kcrombec:

The toolbox comes with a number of sample selection algorithms, both for experimental design (initial samples) and sequential design. Of course you are free to add your own.

The available initial designs can be found in the folder src/matlab/initialDesigns. Adding a new type means creating a new class in this folder analogous to the existing ones (subclass InitialDesign). Two methods are required:

1. a constructor for reading in the configuration extracted from the XML file. Don't forget to call the constructor of the base class using the following syntax:

<code>
this = this@InitialDesign(config);
</code>

2. a generate method that returns the following values:

* A set of initial samples (input values only) to be evaluated. To easily support different ranges for variables, each sample has to be in a fixed [-1,1] range in each dimension. The samples are later automatically transformed to their original range before they are passed to the sample evaluator.

* An optional set of evaluated samples (inputs and outputs), which may come from any source (for example: a dataset). These samples are instantly fed to the toolbox for modelling, because they don't have to be evaluated anymore. Because of this property, they have to be in the original range defined in the simulator xml file, instead of the fixed [-1,1] range.

Add Initial Design

2009-01-16T09:49:50Z

Kcrombec:

The toolbox comes with a number of sample selection algorithms, both for experimental design (initial samples) and sequential design. Of course you are free to add your own.

The available initial designs can be found in the folder src/matlab/initialDesigns. Adding a new type means creating a new class in this folder analogous to the existing ones (subclass InitialDesign). Two methods are required:

* a constructor for reading in the configuration extracted from the XML file. Don't forget to call the constructor of the base class using the following syntax:

[code]
this = this@InitialDesign(config);
[/code]

* a generate.m file that returns two matrices:

** A set of initial samples (input values only) to be evaluated. To easily support different ranges for variables, each sample has to be in a fixed [-1,1] range in each dimension. The samples are later automatically transformed to their original range before they are passed to the sample evaluator.

** An optional set of evaluated samples (inputs and outputs), which may come from any source (for example: a dataset). These samples are instantly fed to the toolbox for modelling, because they don't have to be evaluated anymore. Because of this property, they have to be in the original range defined in the simulator xml file, instead of the fixed [-1,1] range.

Known bugs

2008-11-04T14:27:17Z

Kcrombec: /* Version 6.0.1 */

While we always try to test each release as much as we can, inevitably some bugs will always slip through unnoticed. This page only shows the most important bugs that have surfaced after the release was made.

To stay up to date with the latest news and releases, we also recommend subscribing to our newsletter [http://www.sumo.intec.ugent.be here]. Traffic will be kept to a minimum and you can unsubscribe at any time.

If you happen to encounter something not listed here, please [[reporting problems|report it]].

== Version 6.0.1 ==

{| border="0" style="text-align:left"
|-
!'''Bug'''
| getExpression does not work for rational models with complex outputs. It produces output different from evaluate, and evaluate is the one producing the correct output.
|-
!'''Status'''
| Fixed in next version
|-
!'''Workaround'''
| Please wait for release
|-
|}

== Version 6.0 ==

{| border="0" style="text-align:left"
|-
!'''Bug'''
| You get an error like "No appropriate method or public field isProjectMode for class ibbt.sumo.config.ContextConfig"
|-
!'''Status'''
| Fixed in 6.0.1
|-
!'''Workaround'''
| Please upgrade to 6.0.1
|-
|}

{| border="0" style="text-align:left"
|-
!'''Bug'''
| The Kriging models do not work with combineOutputs=true in some cases
|-
!'''Status'''
| Fixed in SVN
|-
!'''Workaround'''
| Please upgrade to 6.0.1
|-
|}

{| border="0" style="text-align:left"
|-
!'''Bug'''
| The automatic model type selection algorithm (heterogeneous evolution) does not work with Matlab 2008a and later
|-
!'''Status'''
| Fixed in SVN
|-
!'''Workaround'''
| Use an older Matlab version (e.g., 2007a)
|-
|}

== Version 5.0 ==

We have found some important bugs in 5.0 that affect various parts of the model generation process. This could mean (in some cases) that the final models you get are not really as good as they could be. This has been fixed in 6.0 and we are working hard to release it as soon as possible.

{| border="0" style="text-align:left"
|-
!'''Bug'''
| Consecutive runs on the same dataset are not independent (can cause dataset depleted exceptions)
|-
!'''Status'''
| Fixed in SVN.
|-
!'''Workaround'''
| Wait for next release.
|-
|}

{| border="0" style="text-align:left"
|-
!'''Bug'''
| Duplicate samples and samples with NaN/Inf values are not filtered properly (for example you get Matrix dimension mismatch errors).
|-
!'''Status'''
| Fixed in SVN.
|-
!'''Workaround'''
| Wait for next release or simply disable the relevant code. Simulators that may return invalid values or duplicate samples should be avoided until then.
|-
|}

{| border="0" style="text-align:left"
|-
!'''Bug'''
| You get the following error: "Failed to create object of type ANNGeneticInterface, error is "Undefined variable "logger" or class "logger.severe"
|-
!'''Status'''
| Fixed in SVN.
|-
!'''Workaround'''
| Simply remove the offending line or set complex handling to split or modulus (you are trying to model complex data with ANNs).
|-
|}

Interfacing with the toolbox

2008-07-25T13:13:08Z

Kcrombec: /* Gridded datasets */

For how to add an example to the toolbox see the [[Adding an example]] page.

== IMPORTANT ==

* The SUMO Toolbox works on any '''input domain''' (= design space = input parameter ranges) specified in the [[simulator configuration]] file by a '<code>minimum</code>' and '<code>maximum</code>' attribute, for each input parameter.
** If a '<code>minimum</code>' is not specified, the default value of '<code>-1</code>' is assumed.
** If a '<code>maximum</code>' is not specified, the default value of '<code>+1</code>' is assumed.
** Example:

<code><pre>
<InputParameters>
<Parameter name="a" type="real" minimum="47.0" maximum="50.0"/>
<Parameter name="b" type="real" minimum="-20.0"/>
</InputParameters>
</pre></code>

* Be aware that all input values that are not in the specified input domain are trimmed, and thus not used in the modeling process.

Also remember that:
* '''Complex output''' should always be returned as '''2 real values''' (i.e., real part and imaginary part separately).

Make sure your data source complies with these requirements. This is your responsibility.

== Passing data directly ==

As mentioned on the [[Running]] page you pass your (input and corresponding output) data directly to the toolbox.
Remember though that the dimensions of your data must still match the information in the [[Toolbox configuration|toolbox configuration file]] used.

== Native simulator ==

If your simulator is a native code or script it is expected to produce one '''output''' value per line. So every output should be on a new line, with complex outputs using two lines.

There are 2 '''input''' methods supported for native simulators: '''batch mode''' and '''command line mode'''.

In '''command line mode''' (= the default option), the inputs are given to the simulator as command line arguments. A call to a simulator in command line mode looks like (for a problem with 3 input parameters):

<code><pre>
>> ./someSimulationCode 0.5 0.6 0.5
</pre></code>

The code should then produce one value per output per line.

In '''batch mode''', multiple samples can be evaluated in batches. The simulation code is called with no command line arguments (except for optional options, see below). The inputs for a batch are instead given to the simulator on standard input (<code>stdin</code>). First, the size of the batch (the number of samples) is placed on <code>stdin</code>. Then, one line is written for each sample. this means that in total, <code>1 + (batchSize * inputDimension)</code> numbers are written to <code>stdin</code>. An example of the format looks like:

<code><pre>
3
0.5 0.6 0.5
0.2 0.7 0.3
0.2 0.6 0.8
</pre></code>

The executable '<code>someSimulationCode</code>' must be in your path (easiest is just to place it in the <code><SUMO-Toolbox-installation-dir>/bin/c</code> or the absolute path to the executable must be specified in the [[simulator configuration]] xml file.

If your xml file contains options, these will be passed to the simulator as command line arguments (both in single and batch mode). For example:
<code><pre>
>> ./someSimulationCode 0.5 0.6 0.5 option1=value1 option2=value2 etc..
</pre></code>

== Matlab simulator ==

=== Matlab function ===

If your simulator is a Matlab file you just have to provide the following function to your code (for the same 3D example):

<code><pre>
function [output1 output2 output3] = mySimulationCode(input1, input2 ,input3)
...
% do the calculation
</pre></code>

Then you just need to make sure the Matlab file is in the toolbox path (e.g., you can place it in <code>src/matlab/examples</code>).

Options (if present) are passed to the simulator as an extra cell array parameter:

<code><pre>
function [output1 output2 output3] = mySimulationCode(input1, input2 ,input3, options)
</pre></code>

where '<code>options</code>' is a cell array of strings of the form:

<code><pre>
options : {'option1','value1','option2','value2',...}
</pre></code>

'''Note''': since Matlab is single threaded it is impossible to execute the simulator script while modeling (and vica versa). Thus the two are done sequentially which can lead to significant slowdowns if your simulator takes a long time to execute. In that case it is best to use a native executable or script. This will change in future versions when we start using the parallel/distributed computing toolbox.

== Java simulator ==

You can also implement your simulator as a [http://en.wikipedia.org/wiki/Java_%28programming_language%29 Java] class. All you need to do is write a class that implements the ''Simulator'' interface. And make sure the class file is in the Matlab Java path.

Options are passed as a java Properties object.

== Scattered datasets ==

Each row should contain exactly one data point, with one column per dimension.

== Gridded datasets ==
Gridded datasets assume that the data is spread uniformly over a grid. By making this assumption, it becomes pointless to store the sample locations as in a scattered dataset: only the outputs are stored. In order to know the grid size, you do need to specify how many samples are taken in each dimension in the simulator file. For example, a gridSize of 20,40,50 will result in a grid of 20x40x50 or a total of 40000 samples for which the outputs have to be saved on disk.

Because the input values are not stored, the dataset must adhere to a strict order in which the points are specified: the points must be specified in lexicographic order. For example, if you want to define a 3-dimensional dataset with grid size 2x3x2 on the [-1,1] domain, you must provide the outputs for the samples in the following order:

<code><pre>
[-1, -1, -1]
[-1, -1, 1]
[-1, 0, -1]
[-1, 0, 1]
[-1, 1, -1]
[-1, 1, 1]
[ 1, -1, -1]
[ 1, -1, 1]
[ 1, 0, -1]
[ 1, 0, 1]
[ 1, 1, -1]
[ 1, 1, 1]
</pre></code>

Interfacing with the toolbox

2008-07-25T13:12:27Z

Kcrombec: /* Gridded datasets */

For how to add an example to the toolbox see the [[Adding an example]] page.

== IMPORTANT ==

* The SUMO Toolbox works on any '''input domain''' (= design space = input parameter ranges) specified in the [[simulator configuration]] file by a '<code>minimum</code>' and '<code>maximum</code>' attribute, for each input parameter.
** If a '<code>minimum</code>' is not specified, the default value of '<code>-1</code>' is assumed.
** If a '<code>maximum</code>' is not specified, the default value of '<code>+1</code>' is assumed.
** Example:

<code><pre>
<InputParameters>
<Parameter name="a" type="real" minimum="47.0" maximum="50.0"/>
<Parameter name="b" type="real" minimum="-20.0"/>
</InputParameters>
</pre></code>

* Be aware that all input values that are not in the specified input domain are trimmed, and thus not used in the modeling process.

Also remember that:
* '''Complex output''' should always be returned as '''2 real values''' (i.e., real part and imaginary part separately).

Make sure your data source complies with these requirements. This is your responsibility.

== Passing data directly ==

As mentioned on the [[Running]] page you pass your (input and corresponding output) data directly to the toolbox.
Remember though that the dimensions of your data must still match the information in the [[Toolbox configuration|toolbox configuration file]] used.

== Native simulator ==

If your simulator is a native code or script it is expected to produce one '''output''' value per line. So every output should be on a new line, with complex outputs using two lines.

There are 2 '''input''' methods supported for native simulators: '''batch mode''' and '''command line mode'''.

In '''command line mode''' (= the default option), the inputs are given to the simulator as command line arguments. A call to a simulator in command line mode looks like (for a problem with 3 input parameters):

<code><pre>
>> ./someSimulationCode 0.5 0.6 0.5
</pre></code>

The code should then produce one value per output per line.

In '''batch mode''', multiple samples can be evaluated in batches. The simulation code is called with no command line arguments (except for optional options, see below). The inputs for a batch are instead given to the simulator on standard input (<code>stdin</code>). First, the size of the batch (the number of samples) is placed on <code>stdin</code>. Then, one line is written for each sample. this means that in total, <code>1 + (batchSize * inputDimension)</code> numbers are written to <code>stdin</code>. An example of the format looks like:

<code><pre>
3
0.5 0.6 0.5
0.2 0.7 0.3
0.2 0.6 0.8
</pre></code>

The executable '<code>someSimulationCode</code>' must be in your path (easiest is just to place it in the <code><SUMO-Toolbox-installation-dir>/bin/c</code> or the absolute path to the executable must be specified in the [[simulator configuration]] xml file.

If your xml file contains options, these will be passed to the simulator as command line arguments (both in single and batch mode). For example:
<code><pre>
>> ./someSimulationCode 0.5 0.6 0.5 option1=value1 option2=value2 etc..
</pre></code>

== Matlab simulator ==

=== Matlab function ===

If your simulator is a Matlab file you just have to provide the following function to your code (for the same 3D example):

<code><pre>
function [output1 output2 output3] = mySimulationCode(input1, input2 ,input3)
...
% do the calculation
</pre></code>

Then you just need to make sure the Matlab file is in the toolbox path (e.g., you can place it in <code>src/matlab/examples</code>).

Options (if present) are passed to the simulator as an extra cell array parameter:

<code><pre>
function [output1 output2 output3] = mySimulationCode(input1, input2 ,input3, options)
</pre></code>

where '<code>options</code>' is a cell array of strings of the form:

<code><pre>
options : {'option1','value1','option2','value2',...}
</pre></code>

'''Note''': since Matlab is single threaded it is impossible to execute the simulator script while modeling (and vica versa). Thus the two are done sequentially which can lead to significant slowdowns if your simulator takes a long time to execute. In that case it is best to use a native executable or script. This will change in future versions when we start using the parallel/distributed computing toolbox.

== Java simulator ==

You can also implement your simulator as a [http://en.wikipedia.org/wiki/Java_%28programming_language%29 Java] class. All you need to do is write a class that implements the ''Simulator'' interface. And make sure the class file is in the Matlab Java path.

Options are passed as a java Properties object.

== Scattered datasets ==

Each row should contain exactly one data point, with one column per dimension.

== Gridded datasets ==
Gridded datasets assume that the data is spread uniformly over a grid. By making this assumption, it becomes pointless to store the sample locations as in a scattered dataset: only the outputs are stored. In order to know the grid size, you do need to specify how many samples are taken in each dimension in the simulator file. For example, a gridSize of 20,40,50 will result in a grid of 20x40x50 or a total of 40000 samples for which the outputs have to be saved on disk.

Because the input values are not stored, the dataset must adhere to a strict order in which the points are specified: the points must be specified in lexicographic order. For example, if you want to define a 3-dimensional dataset with grid size 2x3x2 on the [-1,1] domain, you must provide the outputs for the samples in the following order:
[-1, -1, -1]
[-1, -1, 1]
[-1, 0, -1]
[-1, 0, 1]
[-1, 1, -1]
[-1, 1, 1]
[ 1, -1, -1]
[ 1, -1, 1]
[ 1, 0, -1]
[ 1, 0, 1]
[ 1, 1, -1]
[ 1, 1, 1]

Interfacing with the toolbox

2008-07-17T08:42:50Z

Kcrombec: /* Gridded datasets */

== IMPORTANT ==

* The SUMO Toolbox works on any '''input domain''' (= design space = input parameter ranges) specified in the [[simulator configuration]] file by a '<code>minimum</code>' and '<code>maximum</code>' attribute, for each input parameter.
** If a '<code>minimum</code>' is not specified, the default value of '<code>-1</code>' is assumed.
** If a '<code>maximum</code>' is not specified, the default value of '<code>+1</code>' is assumed.
** Example:

<code><pre>
<InputParameters>
<Parameter name="a" type="real" minimum="47.0" maximum="50.0"/>
<Parameter name="b" type="real" minimum="-20.0"/>
</InputParameters>
</pre></code>

* Be aware that all input values that are not in the specified input domain are trimmed, and thus not used in the modeling process.

Also remember that:
* '''Complex output''' should always be returned as '''2 real values''' (i.e., real part and imaginary part separately).

Make sure your data source complies with these requirements. This is your responsibility.

== Passing data directly ==

As mentioned on the [[Running]] page you can call the toolbox as follows:

<code><pre>
"go('MyConfigFile.xml',xValues, yValues, [options])"
</pre></code>

This allows you to pass your (input and corresponding output) data directly to the toolbox.

Remember though that the dimensions of your data must still match the information in '<code>[[Toolbox configuration|MyConfigFile.xml]]</code>' file.

== Native simulator ==

If your simulator is a native code or script it is expected to produce one '''output''' value per line. So every output should be on a new line, with complex outputs using two lines.

There are 2 '''input''' methods supported for native simulators: '''batch mode''' and '''command line mode'''.

In '''command line mode''' (= the default option), the inputs are given to the simulator as command line arguments. A call to a simulator in command line mode looks like (for a problem with 3 input parameters):

<code><pre>
>> ./someSimulationCode 0.5 0.6 0.5
</pre></code>

The code should then produce one value per output per line.

In '''batch mode''', multiple samples can be evaluated in batches. The simulation code is called with no command line arguments (except for optional options, see below). The inputs for a batch are instead given to the simulator on standard input (<code>stdin</code>). First, the size of the batch (the number of samples) is placed on <code>stdin</code>. Then, one line is written for each sample. this means that in total, <code>1 + (batchSize * inputDimension)</code> numbers are written to <code>stdin</code>. An example of the format looks like:

<code><pre>
3
0.5 0.6 0.5
0.2 0.7 0.3
0.2 0.6 0.8
</pre></code>

The executable '<code>someSimulationCode</code>' must be in your path (easiest is just to place it in the <code><SUMO-Toolbox-installation-dir>/bin/c</code> or the absolute path to the executable must be specified in the [[simulator configuration]] xml file.

If your xml file contains options, these will be passed to the simulator as command line arguments (both in single and batch mode). For example:
<code><pre>
>> ./someSimulationCode 0.5 0.6 0.5 option1=value1 option2=value2 etc..
</pre></code>

== Matlab simulator ==

=== Matlab function ===

If your simulator is a Matlab file you just have to provide the following function to your code (for the same 3D example):

<code><pre>
function [output1 output2 output3] = mySimulationCode(input1, input2 ,input3)
...
% do the calculation
</pre></code>

Then you just need to make sure the Matlab file is in the toolbox path (e.g., you can place it in <code>src/matlab/examples</code>).

Options (if present) are passed to the simulator as an extra cell array parameter:

<code><pre>
function [output1 output2 output3] = mySimulationCode(input1, input2 ,input3, options)
</pre></code>

where '<code>options</code>' is a cell array of strings of the form:

<code><pre>
options : {'option1','value1','option2','value2',...}
</pre></code>

=== Matlab class ===

Your simulator can also be a Matlab object of a particular class. Provide your class with the following function:

<code><pre>
function [output1 output2 output3] = evaluate(input1, input2 ,input3)
...
% do the calculation
</pre></code>

Options work in the same way as the pure Matlab simulator.

== Java simulator ==

You can also implement your simulator as a [http://en.wikipedia.org/wiki/Java_%28programming_language%29 Java] class. All you need to do is write a class that implements the ''Simulator'' interface.

Options are passed as a java Properties object.

== Scattered datasets ==

Each row should contain exactly one data point, with one column per dimension.

== Gridded datasets ==
Gridded datasets assume that the data is spread uniformly over a grid. By making this assumption, it becomes pointless to store the sample locations as in a scattered dataset: only the outputs are stored. In order to know the grid size, you do need to specify how many samples are taken in each dimension in the simulator file. For example, a gridSize of 20,40,50 will result in a grid of 20x40x50 or a total of 40000 samples for which the outputs have to be saved on disk.

Measures

2008-07-17T08:39:44Z

Kcrombec: /* CrossValidation */

A measure is used to measure the quality of a new model. The measure decides whether the modeling process can be halted (the target accuracy has been reached) or whether new samples should be selected / new models should be built. The choice of measure is therefore very important for the success of the toolbox.

The rule of thumb is that the default measure, CrossValidation, is the best choice, but is also very expensive, as it requires that a number (5 by default) of models are built for each new model that has to be evaluated. Especially when using neural networks this can become an unacceptable overhead. If the modeling takes unacceptably long, the best alternative is ValidationSet, which by default behaves as cross validation in which only one fold is considered, reducing the cost of the measure by a factor 5. A second measure, called MinMax, is also activated by default, enabling the user to force the model to remain within certain bounds, to speed up the convergence. See below for more details in how to set these bounds.

However, in certain situations it might be very effective to use different measures, or use multiple measures together. When multiple measures are used, an intelligent pareto-based method is used to decide which model is the best choice. Models that score high on a particular measure but low on another are not discarded immediately, but are given a chance to set things right in further iterations of the toolbox. This encourages variety in the models, while still ensuring convergence to the optimal accuracy for each measure. An often used combination is CrossValidation with the MinMax measure, to ensure that no poles are present in the model domain.

Note that also an [[FAQ#How_do_I_change_the_error_function_.28relative_error.2C_RMS.2C_....29.3F| error function can be defined]]. The default error function is '<code>rootRelativeSquareError</code>'.

Below is a list of available measures and the configuration options available for each of them. Each measure also has a target accuracy attribute, which can be omitted and which defaults to 0.001. In certain cases, such as the binary MinMax measure, the target accuracy is irrelevant.

== CrossValidation ==

The CrossValidation measure is the default choice and performs an n-fold cross validation on the model to create an efficient estimation of the accuracy of the model. Several options are available to customize this measure.

{{OptionsHeader}}
{{Option
|name = folds
|values = positive integer
|default = 5
|description = The number of folds used for the measure. A higher number means that more models will be built, but that a better accuracy estimate is achieved.
}}
{{Option
|name = storeModels
|values = boolean
|default = no
|description = When this option is turned on, the models built for each fold are stored on disk for future reference.
}}
{{Option
|name = partitionMethod
|values = [uniform,random]
|default = uniform
|description = This option defines whether the test sets for the folds are chosen randomly, or are chosen in such a way as to maximize the domain coverage. Random is generally much faster, but might result in pessimistic scoring, as unlucky test set choice can result in an inaccurate error. This can partly be fixed by enabling the resetFolds option.
}}
{{Option
|name = resetFolds
|values = boolean
|default = no
|description = Folds are generated from scratch for each model that is evaluated using this measure. If the same model is evaluated twice (for example, after a rebuild), new folds are used. Enabling this feature can be very costly for large sample sizes. As a rule of thumb, enable this in combination with the random partition method, or disable this when using the uniform method.
}}

== ValidationSet ==

The ValidationSet measure has two different methods of operation.

# In the first method, the list of samples that have been evaluated is split into a validation set and a training set. A model is then built using the training set, and evaluated using the validation set (which is by default 20% of the total sample pool).
# However, an external data file containing a validation set can also specified. In this case, all the evaluated samples are used for training, and the external set is used for validation only. Which of these two operation methods is used, depends on the configuration options below. By default, no external validation set is loaded.

If you want to use an external validation set, you will have to provide a SampleEvaluator configuration so that the validation set can be loaded from an external source. Here is a ValidationSet configuration example which loads the validation set from the scattered data file provided in the simulator file:

<pre><nowiki>
<Measure type="ValidationSet" target=".001">
<Option key="type" value="file"/>
<SampleEvaluator type="ibbt.sumo.SampleEvaluators.ScatteredDatasetSampleEvaluator"/>
</Measure>
</nowiki></pre>

{{OptionsHeader}}
{{Option
|name = type
|values = [distance, random, file]
|default = distance
|description = Method used to acquire samples for the validation set. The default method, 'distance', tries to select a validation set which covers the entire domain as good as possible, ensuring that not all validation samples are chosen in the same part of the domain. This is achieved using a distance heuristic, which gives no guarantees on optimal coverage but performs very well in almost all situations. The 'random' method just picks a random set of samples from the entire pool to be used for validation set.
Finally, the 'file' method does not take samples at all from the pool, but loads a validation set from an external dataset.
}}
{{Option
|name = percentUsed
|values = [0,100]
|default = 20
|description = Percent of samples used for the validation set. By default 20% of all samples are used for validation, while the remaining 80% are used for training. This option is irrelevant if the 'type' option is set to 'file'.
}}
{{Option
|name = randomThreshold
|values = positive integer
|default = 1000
|description = When the sample pool is very large, the distance heuristic used by default becomes too slow, and the toolbox switches to random sample selection automatically. This is done when the amount of samples is larger than this value, which defaults to 1000. This option should not be changed unless the performance is unacceptable even for sample sets smaller than this amount.
}}

== LeaveNOut ==

== MinMax ==

The MinMax measure is used to eliminate models whose response falls below a given minimum or above a given maximum. This measure can be used to detect models that have poles in the model domain and to guide the modeling process in the right direction. If the output is known to lie within certain value bounds, these can be added to the simulator file as follows:

<pre><nowiki>
<OutputParameters>
<Parameter name="out" type="real" minimum="-1" maximum="1"/>
</OutputParameters>
</nowiki></pre>

When the MinMax measure is defined, these values will be used to ensure that all models stay within these bounds. If only the minimum or only the maximum is defined, naturally only these are enforced. There are no further configuration options for this measure. In case of complex outputs the modulus is used.

Remember though, that no guarantee can be given that the poles will really disappear. Using this measure only combats the symptoms and not the cause of the problem. Also, this measure can be reasonably slow, because it evaluates a dense grid to decide wether the model is crossing boundaries. If the model is slow to evaluate, this can take a considerable amount of time.

'''Tip''': even if you don't know the exact bounds on your output, you can still use this measure by specifying very broad bounds (e.g., [-10000 10000]). This can still allow you to catch poles since, by definition, they reach until infinity.

== ModelDifference ==

== SampleError ==

FAQ

2008-07-17T08:34:16Z

Kcrombec: /* Using */

== General ==

=== What is a global surrogate model? ===

A global [http://en.wikipedia.org/wiki/Surrogate_model surrogate model] is a mathematical model that mimics the behavior of a computationally expensive simulation code over '''the complete parameter space''' as accurately as possible, using as little data points as possible. So note that optimization is not the primary goal, although it can be done as a post-processing step. Global surrogate models are useful for:

* design space exploration, to get a ''feel'' of how the different parameters behave
* sensitivity analysis
* ''what-if'' analysis
* prototyping
* ...

In addition they are a cheap way to model large scale systems, multiple global surrogate models can be chained together in a model cascade.

=== What about surrogate driven optimization? ===

When coining the term '''surrogate driven optimization''' most people associate it with trust-region strategies and simple polynomial models. These frameworks first construct a local surrogate which is optimized to find an optimum. Afterwards, a move limit strategy decides how the local surrogate is scaled and/or moved through the input space. Subsequently the surrogate is rebuild and optimized. I.e. the surrogate zooms in to the global optimum. For instance the [http://www.cs.sandia.gov/DAKOTA/ DAKOTA] Toolbox implements such strategies where the surrogate construction is separated from optimization.

Such a framework was earlier implemented in the SUMO Toolbox but was deprecated as it didn't fit the philosophy and design of the toolbox.

Instead another, equally powerful, approach was taken. The current optimization framework is in fact a sampling selection strategy that balances local and global search. In other words, it balances between exploring the input space and exploiting the information the surrogate gives us.

A configuration example can be found [[Config:SampleSelector#isc|here]]. For more information see the [[Sample_Selectors#InfillSamplingCriterion| InfillSamplingCriterion ]].

=== What is the difference between the M3-Toolbox and the SUMO-Toolbox? ===

The SUMO toolbox is a complete, feature-full framework for automatically generating approximation models and performing adaptive sampling. In contrast, the M3-Toolbox was more of a proof-of-principle.

=== What happened to the M3-Toolbox? ===

The M3 Toolbox project has been discontinued (Fall 2007) and superseded by the SUMO Toolbox. Please contact tom.dhaene@ua.ac.be for any inquiries and requests about the M3 Toolbox.

=== How can I stay up to date with the latest news? ===

To stay up to date with the latest news and releases, we also recommend subscribing to our newsletter [http://www.sumo.intec.ugent.be here]. Traffic will be kept to a minimum (1 message every 2-3 months) and you can unsubscribe at any time.

=== What is the roadmap for the future? ===

There is no explicit roadmap since much depends on where our research leads us, what feedback we get, which problems we are working on, etc. However, to get an idea of features to come you can always check the [[Whats new]] page.

== Installation and Configuration ==

=== Will the SUMO Toolbox work with Matlab R2008 and later? ===
Starting from version 6.0 Matlab R2007b-R2008b should be fully supported. Initial tests have shown that v5.0 will work with R2008, but we do not guarantee that every component will work flawlessly. R2008 also introduced a new way of handling classes and objects to which the SUMO Toolbox will be ported when R2008a and later become more widespread. At that point compatibility with previous Matlab versions will be dropped.

=== What is the relationship between Matlab and Java? ===

Many people do not know this, but your Matlab installation automatically includes a Java virtual machine. By default, Matlab seamlessly integrates with Java, allowing you to create Java objects from the command line (e.g., 's = java.lang.String'). It is possible to disable java support but in order to use the SUMO Toolbox it should not be. To check if Java is enabled you can use the 'usejava' command.

=== What is Java, why do I need it, do I have to install it, etc. ? ===

The short answer is: no, dont worry about it. The long answer is: Some of the code of the SUMO Toolbox is written in [http://en.wikipedia.org/wiki/Java_(programming_language) Java], since it makes a lot more sense in many situations and is a proper programming language instead of a scripting language like Matlab. Since Matlab automatically includes a JVM to run Java code there is nothing you need to do or worry about (see the previous FAQ entry). Unless its not working of course, in that case see [[FAQ#When_running_the_toolbox_you_get_something_like_.27.3F.3F.3F_Undefined_variable_.22ibbt.22_or_class_.22ibbt.sumo.config.ContextConfig.setRootDirectory.22.27]].

== Upgrading ==

=== How do I upgrade to a newer version? ===

Delete your old <code><SUMO-Toolbox-directory></code> and replace it by the new one. Restart Matlab and make sure the default run works. To port your old configuration files to the new version: make a copy of default.xml and copy over your custom changes one by one. This should prevent any weirdness if the XML structure has changed between releases.

== Using ==

=== I have no idea how to use the toolbox, what should I do? ===

See: [[Running#Getting_started]]

=== I want to model my own problem ===

See : [[Adding an example]].

=== I want to contribute some data/patch/documentation/... ===

See : [[Contributing]].

=== How do I interface with the SUMO Toolbox? ===

See : [[Interfacing with the toolbox]].

=== Ok, I generated a model, what can I do with it? ===

See: [[Using a model]].

=== How can I share a model created by the SUMO Toolbox? ===

See : [[Running#Model_portability| Model portability]].

=== Why are the Neural Networks so slow? ===

You are probably using the [[Measures#CrossValidation| CrossValidation]] measure. CrossValidation is used by default if you have not defined a [[Measures| measure]] yourself. Since you need to train them, neural nets will always be slower than the other models. Using CrossValidation will slow things down much much more (5-times slower by default). Therefore, when using one of the neural network model types, please use a different measure, such as [[Measures#ValidationSet| ValidationSet]] or [[Measures#SampleError| SampleError]]. See the comments in <code>default.xml</code> for examples.

Note: Starting from version 5.0, two new neural network backends are available (based on [http://leenissen.dk/fann/ FANN] and [http://www.iau.dtu.dk/research/control/nnsysid.html NNSYSID]). These are a lot faster than the default backend based on the [http://www.mathworks.com/products/neuralnet/ Matlab Neural Network Toolbox]. However, the accuracy it not as good.

=== How can I speed things up? ===

There are a number of things you can do to speed things up:

* Disable some, or even all of the [[Config:ContextConfig#Profiling| profilers]] or disable the output handlers that draw charts
* Turn off the plotting of models in [[Config:ContextConfig#PlotOptions| ContextConfig]], you can always generate plots from the saved mat files
* Decrease the logging granularity, a log level of FINE (the default is FINEST or ALL) is more then granular enough. Setting it to FINE, INFO, or even WARNING should speed things up.
* If you have a multi-core/multi-cpu machine, set the threadCount variable in [[Config:SampleEvaluator#LocalSampleEvaluator| LocalSampleEvaluator]] equal to the number of cores/CPUs (only do this if it is ok to start multiple instances of your simulation script in parallel!)
* Avoid cross validation and use a validation set
* Dont use the Min-Max measure, it can slow things down.
* Instead of anngenetic use fanngenetic or nanngenetic, these are faster but the quality of the models is lower. However it may be good enough for your problem.

If you are having problems with very slow or seemingly hanging runs

* Do a run inside the [http://www.mathworks.com/access/helpdesk/help/techdoc/index.html?/access/helpdesk/help/techdoc/matlab_env/f9-17018.html&http://www.google.be/search?client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&channel=s&hl=nl&q=matlab+profiler&meta=&btnG=Google+zoeken Matlab profiler] and see where most time is spent.
* Monitor CPU and physical/virtual memory usage while the SUMO toolbox is running and see if you notice anything strange.

Also note that by default Matlab only allocates about 117 MB memory space for the Java Virtual Machine. If you would like to increase this limit (which you should) please follow the instructions [http://www.mathworks.com/support/solutions/data/1-18I2C.html?solution=1-18I2C here]. See also the general memory instructions [http://www.mathworks.com/support/tech-notes/1100/1106.html here].

To check if your SUMO run has hanged, monitor your log file (with the level set at least to FINE). If you see no changes for about 30 minutes the toolbox will probably have stalled. [[Reporting problems| report the problems here]].

Such problems are hard to identify and fix so it is best to work towards a reproducible test case if you think you found a performance or scalability issue.

=== How do I turn off adaptive sampling (run the toolbox for a fixed set of samples)? ===

See : [[Adaptive Modeling Mode]].

=== How do I change the error function (relative error, RMS, ...)? ===

The [[Measures| <Measure>]] tag specifies the algorithm to use to assign models a score, e.g., [[Measures#CrossValidation| CrossValidation]]. It is also possible to specify which '''error function''' to use, in the measure. The default error function is '<code>rootRelativeSquareError</code>'.

Say you want to use [[Measures#CrossValidation| CrossValidation]] with the maximum absolute error, then you would put:

<code><pre>
<Measure type="CrossValidation" target="0.001" errorFcn="maxAbsoluteError"/>
</pre></code>

On the other hand, if you wanted to use the [[Measures#ValidationSet| ValidationSet]] measure with a relative root-mean-square error you would put:

<code><pre>
<Measure type="ValidationSet" target="0.001" errorFcn="relativeRms"/>
</pre></code>

The default error function is '<code>rootRelativeSquareError</code>'. These error functions can be found in the <code>src/matlab/tools/errorFunctions</code> directory. You are free to modify them and add your own.

=== How do I enable more profilers? ===

Go to the [[Config:ContextConfig#Profiling| <Profiling>]] tag and put <code>"<nowiki>*</nowiki>"</code> as the regular expression. See also the next question.

=== What regular expressions can I use to filter profilers? ===

See the syntax [http://java.sun.com/j2se/1.5.0/docs/api/java/util/regex/Pattern.html here].

=== How can I ensure deterministic results? ===

See : [[Random state]].

=== How do I get a simple closed-form model (symbolic expression)? ===

See : [[Using a model]].
Use the <code>getExpression(..)</code> function.

=== How do I enable the Heterogenous evolution to automatically select the best model type? ===

Due to a limitation of the [http://www.mathworks.com/products/gads/ Matlab Genetic Algorithm and Direct Search (GADS) Toolbox], you first have to manually edit the file src/matlab/contrib/modifiedMigrate.m. Open it and follow the instructions. Once that is done you can use the [[Config:AdaptiveModelBuilder#heterogenetic| heterogenetic modelbuilder]] as you would any other.

=== What is the combineOutputs option? ===

Some model types support building a single model with multiple outputs (to enable this set combineOutputs=true for the particular component). This means you dont have to build different models for each output in parallel. Thus this saves time and results in a nice fully self-contained model. However, note that in that case the same model parameters are used for each output, thus this only makes sense if there is some correlation between the outputs. The more correlation, the more this makes sense.

In the future this setting will also be supported for the sample selection components (so you can take into account correlation between outputs when choosing new samples).

=== What error function should I use? ===

The default error function is the root relative square error (RRSE). For example, if you get a RRSE of 0.16 it means you are doing 16% better than the most simple model (= the mean). The RRSE is a global relative error that measures the deviation from the mean, it is closely related to the well known [http://en.wikipedia.org/wiki/Coefficient_of_determination R2 (R-squared)] error function. In general the RRSE is a good choice, however, if your function is very smooth (=> meaning the mean is already a very good fit) then it is hard to get a low value for the RRSE (even though your model may be very good). On the other hand meanRelativeError may be more intuitive but in that case you have to be careful if you have function values close to zero since in that case the relative error explodes or even gives infinity. You could also use one of the combined relative error functions (contain a +1 in the denominator to account for small values) but then you get something between a relative and absolute error (=> hard to interpret).

So to be sure an absolute error seems the safest bet (like the RMSE), however in that case you have to come up with sensible accuracy targets and realize that you will build models that try to fit the regions of high absolute value better than the low ones.

Picking an error function is a very tricky business and many people do not realize this. Which one is best for you and what targets you use ultimately depends on your application and on what kind of model you want. There is no general answer.

=== I just want to generate an initial design (no sampling, no modeling) ===

Do a regular SUMO run, except set the 'maxModelingIterations' in the SUMO tag to 0. The resulting run will only generate (and evaluate) the initial design and save it to samples.txt in the output directory.

=== How do I start a run with the samples of of a previous run, or with a custom initial design? ===

Use a Dataset design component, for example:

<code>
<InitialDesign type="DatasetDesign">
<Option key="file" value="/path/to/the/file/containing/the/points.txt"/>
</InitialDesign>
</code>

=== What is a level plot? ===

A level plot is a plot that shows how the error histogram changes as the best model improves. An example is:
<gallery>
Image:levelplot.png
</gallery>
Level plots only work if you have a separate dataset (test set) that the model can be checked against. See the comments in default.xml for how to enable level plots.

===I am getting a java out of memory error, what happened?===
Datasets are loaded through java my the toolbox. This means that the java heap space is used for storing the data. If you try to load a huge dataset (> 50MB), you might experience problems with the maximum heap size. You can solve this by raising the heap size as described on the following webpage:
[http://www.mathworks.com/support/solutions/data/1-18I2C.html]

== Troubleshooting ==

=== I have a problem and I want to report it ===

See : [[Reporting problems]].

=== I sometimes get flat models when using rational functions ===

First make sure the model is indeed flat, and does not just appear so on the plot. You can verify this by looking at the output axis range and making sure it is within reasonable bounds. When there are poles in the model, the axis range is sometimes stretched to make it possible to plot the high values around the pole, causing the rest of the model to appear flat. If the model contains poles, refer to the next question for the solution.

The [[Config:AdaptiveModelBuilder#rational| RationalModel]] tries to do a least squares fit, based on which monomials are allowed in numerator and denominator. We have experienced that some models just find a flat model as the best least squares fit. There are two causes for this:

* The number of sample points is few, and the model parameters (as explained [[Model types explained#PolynomialModel|here]] and [[Adaptive Model Builders|here]]) force the model to use only a very small set of degrees of freedom. The solution in this case is to increase the minimum percentage bound in the RationalXYZInterface ([[Config:AdaptiveModelBuilder#rational| RationalSequentialInterface]] or [[Config:AdaptiveModelBuilder#rationalgenetic| RationalGeneticInterface]]) section of your configuration file: change the <code>"percentBounds"</code> option to <code>"60,100"</code>, <code>"80,100"</code>, or even <code>"100,100"</code>. A setting of <code>"100,100"</code> will force the polynomial models to always exactly interpolate. However, note that this does not scale very well with the number of samples (to counter this you can set <code>"maxDegrees"</code>). If, after increasing the <code>"percentBounds"</code> you still get weird, spiky, models you simply need more samples or you should switch to a different model type.
* Another possibility is that given a set of monomial degrees, the flat function is just the best possible least squares fit. In that case you simply need to wait for more samples.

=== When using rational functions I sometimes get 'spikes' (poles) in my model ===

When the denominator polynomial of a rational model has zeros inside the domain, the model will tend to infinity near these points. In most cases these models will only be recognized as being `the best' for a short period of time. As more samples get selected these models get replaced by better ones and the spikes should disappear.

So, it is possible that a rational model with 'spikes' (caused by poles inside the domain) will be selected as best model. This may or may not be an issue, depending on what you want to use the model for. If it doesn't matter that the model is very inaccurate at one particular, small spot (near the pole), you can use the model with the pole and it should perform properly.

However, if the model should have a reasonable error on the entire domain, several methods are available to reduce the chance of getting poles or remove the possibility altogether. The possible solutions are:

* Simply wait for more data, usually spikes disappear (but not always).
* Lower the maximum of the <code>"percentBounds"</code> option in the RationalXYZInterface ([[Config:AdaptiveModelBuilder#rational| RationalSequentialInterface]] or [[Config:AdaptiveModelBuilder#rationalgenetic| RationalGeneticInterface]]) section of your configuration file. For example, say you have 500 data points and if the maximum of the <code>"percentBounds"</code> option is set to 100 percent it means the degrees of the polynomials in the rational function can go up to 500. If you set the maximum of the <code>"percentBounds"</code> option to 10, on the other hand, the maximum degree is set at 50 (= 10 percent of 500). You can also use the <code>"maxDegrees"</code> option to set an absolute bound.
* If you roughly know the output range your data should have, an easy way to eliminate poles is to use the [[Measures#MinMax| MinMax]] [[Measures| Measure]] together with your current measure ([[Measures#CrossValidation| CrossValidation]] by default). This will cause models whose response falls outside the min-max bounds to be penalized extra, thus spikes should disappear. See : [[Outputs|Combining measures]].
* Use a different model type (RBF, ANN, SVM,...), as spikes are a typical problem of rational functions.
* Try using the [[SampleSelector#RationalPoleSuppressionSampleSelector| RationalPoleSuppressionSampleSelector]], it was designed to get rid of this problem more quickly, but it only selects one sample at the time.

=== There is no noise in my data yet the rational functions don't interpolate ===

See : [[FAQ#I sometimes get flat models when using rational functions | this question]].

=== When loading a model from disk I get "Warning: Class ':all:' is an unknown object class. Object 'model' of this class has been converted to a structure." ===

You are trying to load a model file without the SUMO Toolbox in your Matlab path. Make sure the toolbox is in your Matlab path.

In short: Start Matlab, run <code><SUMO-Toolbox-directory>/startup.m</code> (to ensure the toolbox is in your path) and then try to load your model.

=== When running the SUMO Toolbox you get an error like "No component with id 'ann' of type 'adaptive model builder' found in config file." ===

This means you have specified to use a component with a certain id (in this case an AdaptiveModelBuilder component with id 'ann') but a component with that id does not exist further down in the configuration file (in this particular case 'ann' does not exist but 'anngenetic' does, as a quick search through the configuration file will show). So make sure you only declare components which have a definition lower down. So see which components are available, simply scroll down the configuration file and see which id's are specified. Please also refer to the [[Toolbox configuration#Declarations and Definitions | Declarations and Definitions]] page.

=== When using RBF neural network models I sometimes get get a crash in "[http://www.mathworks.com/access/helpdesk/help/toolbox/nnet/index.html?/access/helpdesk/help/toolbox/nnet/newrb.html&http://www.google.com/search?q=newrb&rls=com.microsoft:en-US:IE-SearchBox&ie=UTF-8&oe=UTF-8&sourceid=ie7&rlz=1I7GGIC newrb]" ===

This is an error in the [http://www.mathworks.com/products/neuralnet/ Matlab Neural Network Toolbox] implementation and not anything we can do about (a workaround is available on [[Contact|request]]). This should be fixed by Matlab 7.5.

=== When using NANN models I sometimes get "Runtime error in matrix library, Choldc failed. Matrix not positive definite" ===

This is a problem in the mex implementation of the [http://www.iau.dtu.dk/research/control/nnsysid.html NNSYSID] toolbox. Simply delete the mex files, the Matlab implementation will be used and this will not cause any problems.

=== When using FANN models I sometimes get "Invalid MEX-file createFann.mexa64, libfann.so.2: cannot open shared object file: No such file or directory." ===

This means Matlab cannot find the [http://leenissen.dk/fann/ FANN] library itself to link to dynamically. Make sure it is in your library path, ie, on unix systems, make sure it is included in LD_LIBRARY_PATH.

=== When trying to use SVM models I get 'Error during fitness evaluation: Error using ==> svmtrain at 170, Group must be a vector' ===

You forgot to build the SVM mex files for your platform. For windows they are pre-compiled for you, on other systems you have to compile them yourself with the makefile.

=== When running the toolbox you get something like '??? Undefined variable "ibbt" or class "ibbt.sumo.config.ContextConfig.setRootDirectory"' ===

First see [[FAQ#What_is_the_relationship_between_Matlab_and_Java.3F | this FAQ entry]].

This means Matlab cannot find the needed Java classes. This typically means that you forgot to run 'startup' (to set the path correctly) before running the toolbox (using 'go'). So make sure you always run 'startup' before running 'go' and that both commands are always executed in the toolbox root directory.

If you did run 'startup' correctly and you are still getting an error, check that Java is properly enabled:

# typing 'usejava jvm' should return 1
# typing 's = java.lang.String', this should ''not'' give an error
# typing 'version('-java')' should return at least version 1.5.0

If (1) returns 0, then the jvm of your Matlab installation is not enabled. Check your Matlab installation or startup parameters (did you start Matlab with -nojvm?)
If (2) fails but (1) is ok, there is a very weird problem, check the Matlab documentation.
If (3) returns a version before 1.5.0 you will have to upgrade Matlab to a newer version or force Matlab to use a custom, newer, jvm (See the Matlab docs for how to do this).

=== You get errors related to ''gaoptimset'',''psoptimset'',''saoptimset'',''newff'' not being found or unknown ===

You are trying to use a component of the SUMO toolbox that requires a Matlab toolbox that you do not have. See the [[System requirements]] for more information.

=== After upgrading I get all kinds of weird errors or warnings when I run my XML files ===

See [[FAQ#How_do_I_upgrade_to_a_newer_version.3F]]

=== I sometimes see the error of the best model go up, shouldn't it decrease monotonically? ===

There is no short answer here, it depends on the situation. Below 'single objective' refers to the case where during the hyperparameter optimization (= the modeling iteration) combineOutputs=false, and there is only a single measure set to 'on'. The other cases are classified as 'multi objective'.

# '''Sampling off'''
## ''Single objective'': the error should always decrease monotonically, you should never see it rise. If it does [[reporting problems|report it as a bug]]
## ''Multi objective'': There is a very small chance the error can temporarily decrease but it should be safe to ignore. In this case it is best to use a multi objective enabled modeling algorithm (available in version 6.1.)
# '''Sampling on'''
## ''Single objective'': inside each modeling iteration the error should always monotonically decrease. At each sampling iteration the best models are updated (to reflect the new data), thus there the best model score may increase, this is normal behavior(*). It is possible that the error increases for a short while, but as more samples come in it should decrease again. If this does not happen you are using a poor measure or poor hyperparameter optimization algorithm, or there is a problem with the modeling technique itself (e.g., clustering in the datapoints is causing numerical problems).
## ''Multi objective'': Combination of 1.2 and 2.1.

(*) This is normal if you are using a measure like cross validation that is less reliable on little data than on more data. However, in some cases you may wish to override this behavior if you are using a measure that is independent of the number of samples the model is trained with (e.g., a dense, external validation set). In this case you can force a monotonic decrease by setting the 'keepOldModels' option in the SUMO tag to true. Use with caution!