General guidelines - Revision history

Javdrher: /* Adaptive Model Builders */

2014-02-27T15:30:16Z

Adaptive Model Builders

← Older revision		Revision as of 15:30, 27 February 2014
Line 25:		Line 25:
	When using the [[Config:SequentialDesign#error\|error-based sample selector]] separately, it is always a good idea to combine it with the [[Config:SequentialDesign#voronoi\|voronoi]], to combat stability/robustness issues the error-based sample selector often causes. It is a good idea to select about 60% of the samples with error, and 40% with the voronoi. This will ensure that at least the entire design space is covered to a certain degree. This additional sample selector is NOT necessary when using LOLA-Voronoi. To combine sample selectors, create a CombinedSampleSelector. See the [[Config:SequentialDesign#default\|default sample selector]] for an example.		When using the [[Config:SequentialDesign#error\|error-based sample selector]] separately, it is always a good idea to combine it with the [[Config:SequentialDesign#voronoi\|voronoi]], to combat stability/robustness issues the error-based sample selector often causes. It is a good idea to select about 60% of the samples with error, and 40% with the voronoi. This will ensure that at least the entire design space is covered to a certain degree. This additional sample selector is NOT necessary when using LOLA-Voronoi. To combine sample selectors, create a CombinedSampleSelector. See the [[Config:SequentialDesign#default\|default sample selector]] for an example.

	== ~~Adaptive~~ Model Builders ==		== Model Builders ==

	The question that always gets asked is ''Which model type should I use for my data?'' Unfortunately there is no straightforward since it all depends on your problem: how many dimensions, how many points, is your function rugged, smooth, or both, is there noise, etc, etc. Based on this knowledge it is possible to say which model types are more likely to do well but it remains a heuristic. Best is to try a few and see what happens, or use the ''heterogenetic'' model builder to try multiple model types in parallel and automatically try to determine the best type.		The question that always gets asked is ''Which model type should I use for my data?'' Unfortunately there is no straightforward since it all depends on your problem: how many dimensions, how many points, is your function rugged, smooth, or both, is there noise, etc, etc. Based on this knowledge it is possible to say which model types are more likely to do well but it remains a heuristic. Best is to try a few and see what happens, or use the ''heterogenetic'' model builder to try multiple model types in parallel and automatically try to determine the best type.

Javdrher: /* Sample Selectors */

2014-02-27T15:30:04Z

Sample Selectors

← Older revision		Revision as of 15:30, 27 February 2014
Line 15:		Line 15:
	Selecting a good Measure '''is a very important''' part of the modeling process! It is CRUCIAL that you think well about this. Make sure you also read [[Multi-Objective Modeling]].		Selecting a good Measure '''is a very important''' part of the modeling process! It is CRUCIAL that you think well about this. Make sure you also read [[Multi-Objective Modeling]].

	== ~~Sample Selectors~~ ==		== Sequential Design ==

	The default [[Config:SequentialDesign\|Sequential Design]] is the [[Config:SequentialDesign#lola-voronoi\|LOLA-Voronoi sample selector]] combined with the [[Config:SequentialDesign#error\|error-based sample selector]], with a weight of 0.7 for LOLA and 0.3 for error. This is a very robust sample selector, capable of dealing with most situations. There are, however, some cases in which it is advisable to choose a different one:		The default [[Config:SequentialDesign\|Sequential Design]] is the [[Config:SequentialDesign#lola-voronoi\|LOLA-Voronoi sample selector]] combined with the [[Config:SequentialDesign#error\|error-based sample selector]], with a weight of 0.7 for LOLA and 0.3 for error. This is a very robust sample selector, capable of dealing with most situations. There are, however, some cases in which it is advisable to choose a different one:

	* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:~~SampleSelector~~#voronoi\|voronoi]] or [[Config:~~SampleSelector~~#error\|error]] instead.		* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SequentialDesign#voronoi\|voronoi]] or [[Config:SequentialDesign#error\|error]] instead.
	* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[Config:~~SampleSelector~~#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.		* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[Config:SequentialDesign#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.
	* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:~~SampleSelector~~#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.		* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SequentialDesign#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.

	When using the [[Config:~~SampleSelector~~#error\|error-based sample selector]] separately, it is always a good idea to combine it with the [[Config:~~SampleSelector~~#voronoi\|voronoi]], to combat stability/robustness issues the error-based sample selector often causes. It is a good idea to select about 60% of the samples with error, and 40% with the voronoi. This will ensure that at least the entire design space is covered to a certain degree. This additional sample selector is NOT necessary when using LOLA-Voronoi. To combine sample selectors, create a CombinedSampleSelector. See the [[Config:~~SampleSelector~~#default\|default sample selector]] for an example.		When using the [[Config:SequentialDesign#error\|error-based sample selector]] separately, it is always a good idea to combine it with the [[Config:SequentialDesign#voronoi\|voronoi]], to combat stability/robustness issues the error-based sample selector often causes. It is a good idea to select about 60% of the samples with error, and 40% with the voronoi. This will ensure that at least the entire design space is covered to a certain degree. This additional sample selector is NOT necessary when using LOLA-Voronoi. To combine sample selectors, create a CombinedSampleSelector. See the [[Config:SequentialDesign#default\|default sample selector]] for an example.

	== Adaptive Model Builders ==		== Adaptive Model Builders ==

Javdrher: /* Sample Selectors */

2014-02-27T15:28:40Z

Sample Selectors

← Older revision		Revision as of 15:28, 27 February 2014
Line 17:		Line 17:
	== Sample Selectors ==		== Sample Selectors ==

	The default [[Config:~~SampleSelector~~\|~~SampleSelector~~]] is the [[Config:~~SampleSelector~~#lola-voronoi\|LOLA-Voronoi sample selector]] combined with the [[Config:~~SampleSelector~~#error\|error-based sample selector]], with a weight of 0.7 for LOLA and 0.3 for error. This is a very robust sample selector, capable of dealing with most situations. There are, however, some cases in which it is advisable to choose a different one:		The default [[Config:SequentialDesign\|Sequential Design]] is the [[Config:SequentialDesign#lola-voronoi\|LOLA-Voronoi sample selector]] combined with the [[Config:SequentialDesign#error\|error-based sample selector]], with a weight of 0.7 for LOLA and 0.3 for error. This is a very robust sample selector, capable of dealing with most situations. There are, however, some cases in which it is advisable to choose a different one:

	* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.		* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.

Admin: /* Sample Selectors */

2009-10-02T14:50:23Z

Sample Selectors

← Older revision		Revision as of 14:50, 2 October 2009
Line 20:		Line 20:

	* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.		* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.
	* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[SampleSelector#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.		* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[Config:SampleSelector#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.
	* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.		* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.

Admin: /* Sample Selectors */

2009-10-02T14:49:59Z

Sample Selectors

← Older revision		Revision as of 14:49, 2 October 2009
Line 20:		Line 20:

	* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.		* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.
	* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[SampleSelector#~~ErrorSampleSelector~~\|~~ErrorSampleSelector~~]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.		* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[SampleSelector#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.
	* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.		* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.

Admin: /* Sample Selectors */

2009-10-02T14:49:30Z

Sample Selectors

← Older revision		Revision as of 14:49, 2 October 2009
Line 17:		Line 17:
	== Sample Selectors ==		== Sample Selectors ==

	The default [[SampleSelector\|SampleSelector]] is the [[SampleSelector#~~LOLASampleSelector~~\|~~LOLASampleSelector~~]] combined with the [[SampleSelector#~~ErrorSampleSelector~~\|~~ErrorSampleSelector~~]], with a weight of 0.8 for LOLA and 0.2 for error. This is a very robust sample selector, capable of dealing with most situations. There are, however, some cases in which it is advisable to choose a different one:		The default [[Config:SampleSelector\|SampleSelector]] is the [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi sample selector]] combined with the [[Config:SampleSelector#error\|error-based sample selector]], with a weight of 0.7 for LOLA and 0.3 for error. This is a very robust sample selector, capable of dealing with most situations. There are, however, some cases in which it is advisable to choose a different one:

	* '''Large-scale problems (1000+ samples):''' ~~The GradientSampleSelector~~'s time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, ~~the GradientSampleSelector becomes~~ quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with ~~the GradientSampleSelector~~ might still be negligible.		* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.
	* '''Rational modeler:''' Benchmarks have shown that the gain of ~~using the LOLASampleSelector~~ over the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) ~~ErrorSampleSelector~~ when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.		* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.
			* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.

	When using the ~~ErrorSampleSelector instead of the LOLASampleSelector~~, it is always a good idea to combine it with the [[SampleSelector#~~DensitySampleSelector~~\|~~DensitySampleSelector~~]], to combat stability/robustness issues the ~~ErrorSampleSelector~~ often causes. It is a good idea to select about 60% of the samples with ~~the ErrorSampleSelector~~, and 40% with the ~~DensitySampleSelector~~. This will ensure that at least the entire design space is covered to a certain degree. This additional sample selector is NOT necessary when using the ~~LOLASampleSelector~~.		When using the [[Config:SampleSelector#error\|error-based sample selector]] separately, it is always a good idea to combine it with the [[Config:SampleSelector#voronoi\|voronoi]], to combat stability/robustness issues the error-based sample selector often causes. It is a good idea to select about 60% of the samples with error, and 40% with the voronoi. This will ensure that at least the entire design space is covered to a certain degree. This additional sample selector is NOT necessary when using LOLA-Voronoi. To combine sample selectors, create a CombinedSampleSelector. See the [[Config:SampleSelector#default\|default sample selector]] for an example.

	== Adaptive Model Builders ==		== Adaptive Model Builders ==

Dgorissen: /* Adaptive Model Builders */

2009-03-26T15:23:32Z

Adaptive Model Builders

← Older revision		Revision as of 15:23, 26 March 2009
Line 40:		Line 40:
	# The ANN models generally perfom very well across all problems but are very slow to use. Also if the function is uniformly rugged the Kriging/RBF/... models will give a better fit with much less points (eg. ackley function).		# The ANN models generally perfom very well across all problems but are very slow to use. Also if the function is uniformly rugged the Kriging/RBF/... models will give a better fit with much less points (eg. ackley function).
	# The FANN and NANN models are much faster than the ANN models, but usually the accuracy of the ANN models is much better		# The FANN and NANN models are much faster than the ANN models, but usually the accuracy of the ANN models is much better

			Finally, a related question is, which model builder variant should I use (e.g., svmsim, svmga, svmps, svnoptim, etc). The best optimization algorithm to use will usually depend on how many model parameters you have. For example, since SVM models only have 2 or 3 parameters most algorithms do well and you wont see that much difference. On the other hand, if you are fitting a 5D Kriging model (thus you have at least 5 model parameters to optimize) you will most likely see better performance using the GA or PSO versions over for example the pattern search or gradient descent versions.

			However, our general experience is that it does not make that much of a difference (outside the obvious extremes like gradient descent vs GA). Only if data is really expensive and you want to be sure of the best model with least samples should you really start worrying about this.

	'''Note this is just some very rough intuition gained from our experience with different datasets, your mileage may vary! If you have any suggestions [[Contact\|let us know]]'''		'''Note this is just some very rough intuition gained from our experience with different datasets, your mileage may vary! If you have any suggestions [[Contact\|let us know]]'''

Dgorissen: /* Measures */

2009-03-10T14:59:54Z

Measures

← Older revision		Revision as of 14:59, 10 March 2009
Line 13:		Line 13:
	* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.		* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.

	Selecting a good Measure '''is a very important''' part of the modeling process! Make sure you also read [[Multi-Objective Modeling]].		Selecting a good Measure '''is a very important''' part of the modeling process! It is CRUCIAL that you think well about this. Make sure you also read [[Multi-Objective Modeling]].

	== Sample Selectors ==		== Sample Selectors ==

Dgorissen: /* Measures */

2009-03-10T14:59:26Z

Measures

← Older revision		Revision as of 14:59, 10 March 2009
Line 9:		Line 9:
	The default [[Measures\| Measure]] is [[Measures#CrossValidation\| CrossValidation]]. Even though this is a very good, accurate, overall measure, there are some considerations to make in the following cases:		The default [[Measures\| Measure]] is [[Measures#CrossValidation\| CrossValidation]]. Even though this is a very good, accurate, overall measure, there are some considerations to make in the following cases:

	* '''Expensive modelers (ann):''' If it is relatively expensive to train a model (for example, with neural networks), CrossValidation is also very slow, because it has to train a model for each fold (which is 5 by default). If modeling takes too long, you might want to use a faster alternative, such as [[Measures#ValidationSet\|ValidationSet]].		* '''Expensive modelers (ann):''' If it is relatively expensive to train a model (for example, with neural networks), CrossValidation is also very slow, because it has to train a model for each fold (which is 5 by default). If modeling takes too long, you might want to use a faster alternative, such as [[Measures#ValidationSet\|ValidationSet]] or a combination of [[Measures#SampleError\|SampleError]] and [[Measures#LRMMeasure\|LRMMeasure]].
	* '''ErrorSampleSelector:''' CrossValidation might give a biased result when combined with the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]]. This is because the ErrorSampleSelector tends to cluster samples around one point, which will result in very accurate surrogate models for all the points in this cluster (and thus good results with CrossValidation ). So when using CrossValidation and ErrorSampleSelector together, keep in mind that the real accuracy might be slightly lower than the estimated one.		* '''ErrorSampleSelector:''' CrossValidation might give a biased result when combined with the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]]. This is because the ErrorSampleSelector tends to cluster samples around one point, which will result in very accurate surrogate models for all the points in this cluster (and thus good results with CrossValidation ). So when using CrossValidation and ErrorSampleSelector together, keep in mind that the real accuracy might be slightly lower than the estimated one.
	* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.		* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.

Dgorissen: /* Measures */

2009-03-10T14:58:19Z

Measures

← Older revision		Revision as of 14:58, 10 March 2009
Line 12:		Line 12:
	* '''ErrorSampleSelector:''' CrossValidation might give a biased result when combined with the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]]. This is because the ErrorSampleSelector tends to cluster samples around one point, which will result in very accurate surrogate models for all the points in this cluster (and thus good results with CrossValidation ). So when using CrossValidation and ErrorSampleSelector together, keep in mind that the real accuracy might be slightly lower than the estimated one.		* '''ErrorSampleSelector:''' CrossValidation might give a biased result when combined with the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]]. This is because the ErrorSampleSelector tends to cluster samples around one point, which will result in very accurate surrogate models for all the points in this cluster (and thus good results with CrossValidation ). So when using CrossValidation and ErrorSampleSelector together, keep in mind that the real accuracy might be slightly lower than the estimated one.
	* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.		* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.

			Selecting a good Measure '''is a very important''' part of the modeling process! Make sure you also read [[Multi-Objective Modeling]].

	== Sample Selectors ==		== Sample Selectors ==

← Older revision		Revision as of 14:50, 2 October 2009
Line 20:		Line 20:

	* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.		* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.
	* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[SampleSelector#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.		* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[Config:SampleSelector#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.
	* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.		* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.

← Older revision		Revision as of 14:49, 2 October 2009
Line 20:		Line 20:

	* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.		* '''Large-scale problems (1000+ samples):''' LOLA-Voronoi's time complexity is O(n²) to the number of samples n, so for large-scale experiments in which many samples are taken, LOLA-Voronoibecomes quite slow. Depending on the time it takes to perform one simulation, this may or may not be a problem. If it takes a long time to perform one simulation, the cost for selecting new samples with LOLA-Voronoi might still be negligible. If, however, you need a quicker sample selector, it is advized to use [[Config:SampleSelector#voronoi\|voronoi]] or [[Config:SampleSelector#error\|error]] instead.
	* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[SampleSelector#~~ErrorSampleSelector~~\|~~ErrorSampleSelector~~]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.		* '''Rational modeler:''' Benchmarks have shown that the gain of LOLA-Voronoi over the [[SampleSelector#error\|error-based sample selector]] when using global approximation methods (mainly rational/polynomial) is pretty much zero. It is therefore advisable to use the (much faster) [[Config:SampleSelector#error\|error-based sample selector]] when using the Rational modeler. This can be done by changing the weights in default.xml to 1.0 for error and 0.0 for LOLA.
	* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.		* If you need to sample multiple outputs at once, with one sample selector, or you need an auto-sampled input (for example: a frequency input), you should use [[Config:SampleSelector#lola-voronoi\|LOLA-Voronoi]]. It is the only sample selector with fully integrated and optimized support for these features.

← Older revision		Revision as of 14:59, 10 March 2009
Line 9:		Line 9:
	The default [[Measures\| Measure]] is [[Measures#CrossValidation\| CrossValidation]]. Even though this is a very good, accurate, overall measure, there are some considerations to make in the following cases:		The default [[Measures\| Measure]] is [[Measures#CrossValidation\| CrossValidation]]. Even though this is a very good, accurate, overall measure, there are some considerations to make in the following cases:

	* '''Expensive modelers (ann):''' If it is relatively expensive to train a model (for example, with neural networks), CrossValidation is also very slow, because it has to train a model for each fold (which is 5 by default). If modeling takes too long, you might want to use a faster alternative, such as [[Measures#ValidationSet\|ValidationSet]].		* '''Expensive modelers (ann):''' If it is relatively expensive to train a model (for example, with neural networks), CrossValidation is also very slow, because it has to train a model for each fold (which is 5 by default). If modeling takes too long, you might want to use a faster alternative, such as [[Measures#ValidationSet\|ValidationSet]] or a combination of [[Measures#SampleError\|SampleError]] and [[Measures#LRMMeasure\|LRMMeasure]].
	* '''ErrorSampleSelector:''' CrossValidation might give a biased result when combined with the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]]. This is because the ErrorSampleSelector tends to cluster samples around one point, which will result in very accurate surrogate models for all the points in this cluster (and thus good results with CrossValidation ). So when using CrossValidation and ErrorSampleSelector together, keep in mind that the real accuracy might be slightly lower than the estimated one.		* '''ErrorSampleSelector:''' CrossValidation might give a biased result when combined with the [[SampleSelector#ErrorSampleSelector\|ErrorSampleSelector]]. This is because the ErrorSampleSelector tends to cluster samples around one point, which will result in very accurate surrogate models for all the points in this cluster (and thus good results with CrossValidation ). So when using CrossValidation and ErrorSampleSelector together, keep in mind that the real accuracy might be slightly lower than the estimated one.
	* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.		* '''Rational modeler:''' When using Rational modeler, you might want to manually add a [[Measures#MinMax\| MinMax]] measure (if you got a rough estimate of the minimum and maximum values for your outputs) and use it together with CrossValidation. By adding the MinMax measure, you eliminate models which have poles in the design space, because these poles always break the minimum and maximum bounds. This usually results in better models and quicker convergence.