Measures - Revision history

Admin at 10:49, 26 July 2014

2014-07-26T10:49:33Z

← Older revision		Revision as of 10:49, 26 July 2014
Line 86:		Line 86:
	== Tips on using measures ==		== Tips on using measures ==

	A recommended read is [http://~~www.~~sumo.intec.ugent.be/files/2009_08_EWC.pdf the paper available here].		A recommended read is [http://sumo.intec.ugent.be/sites/sumo/files/sumo/2009_08_EWC.pdf the paper available here].

	In general, the default measure, 5 fold CrossValidation, is an acceptable choice. However, it is also very expensive, as it requires that a model be re-trained for each fold. This may slow things down if a model is expensive to train (e.g., neural nets). Also CrossValidation can give biased results if data is clustered or scarce. Increasing the number of folds may help here. A cheaper alternative is ValidationSet (see below) or AIC. For a full list of available measures see the <code>src/matlab/measures</code> subdirectory.		In general, the default measure, 5 fold CrossValidation, is an acceptable choice. However, it is also very expensive, as it requires that a model be re-trained for each fold. This may slow things down if a model is expensive to train (e.g., neural nets). Also CrossValidation can give biased results if data is clustered or scarce. Increasing the number of folds may help here. A cheaper alternative is ValidationSet (see below) or AIC. For a full list of available measures see the <code>src/matlab/measures</code> subdirectory.

Admin at 08:22, 17 March 2014

2014-03-17T08:22:23Z

← Older revision		Revision as of 08:22, 17 March 2014
Line 156:		Line 156:
	# However, an external data file containing a validation set can also specified. In this case, all the evaluated samples are used for training, and the external set is used for validation only. Which of these two operation methods is used, depends on the configuration options below. By default, no external validation set is loaded.		# However, an external data file containing a validation set can also specified. In this case, all the evaluated samples are used for training, and the external set is used for validation only. Which of these two operation methods is used, depends on the configuration options below. By default, no external validation set is loaded.

	If you want to use an external validation set, you will have to provide a ~~SampleEvaluator~~ configuration so that the validation set can be loaded from an external source. Here is a ValidationSet configuration example which loads the validation set from the scattered data file provided in the simulator file:		If you want to use an external validation set, you will have to provide a DataSource configuration so that the validation set can be loaded from an external source. Here is a ValidationSet configuration example which loads the validation set from the scattered data file provided in the simulator file:

	<source lang="xml">		<source lang="xml">
	<Measure type="ValidationSet" target=".001">		<Measure type="ValidationSet" target=".001">
	<Option key="type" value="file"/>		<Option key="type" value="file"/>
	<~~SampleEvaluator~~ type="~~ibbt~~.sumo.~~sampleevaluators~~.datasets.~~ScatteredDatasetSampleEvaluator~~"/>		<DataSource type="iminds.sumo.DataSource.datasets.ScatteredDatasetDataSource"/>
	</Measure>		</Measure>
	</source>		</source>

Admin at 08:16, 17 March 2014

2014-03-17T08:16:25Z

← Older revision		Revision as of 08:16, 17 March 2014
Line 17:		Line 17:
	That choice will depend on your problem characteristics, the data distribution and the model type you will use to fit the data. It is '''extremely important''' to think about this well ("what do you want?") before starting any modeling.		That choice will depend on your problem characteristics, the data distribution and the model type you will use to fit the data. It is '''extremely important''' to think about this well ("what do you want?") before starting any modeling.

	As a side remark, note that Measures and ~~[[SampleSelector\|sample selectors]]~~ are closely related. The same criteria that is used to decide if a model is good or bad, can be used to identify interesting locations to select new data points.		As a side remark, note that Measures and sequential designs are closely related. The same criteria that is used to decide if a model is good or bad, can be used to identify interesting locations to select new data points.

	Also note that some model builders (notably GeneticModelBuilder) support constraints directly. This means you can also implement what you would otherwise implement as a Measure as constraint in the model parameter optimization algorithm itself. This is a superior solution to using a Measure.		Also note that some model builders (notably GeneticModelBuilder) support constraints directly. This means you can also implement what you would otherwise implement as a Measure as constraint in the model parameter optimization algorithm itself. This is a superior solution to using a Measure.

Mnguyen: /* Using multiple measures */

2012-02-02T15:24:03Z

Using multiple measures

← Older revision		Revision as of 15:24, 2 February 2012
Line 73:		Line 73:

	Go to the [[Multi-Objective_Modeling\|Multi-Objective Modeling page]] to find out how to build models using multiple measures.		Go to the [[Multi-Objective_Modeling\|Multi-Objective Modeling page]] to find out how to build models using multiple measures.
	If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding ~~them~~ to the Plan but with the <code>use="off"</code> keyword.		If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding the desired measure(s) you want to use for monitoring to the Plan but with the <code>use="off"</code> keyword.

	<source xmlns:saxon="http://icl.com/saxon" lang="xml">		<source xmlns:saxon="http://icl.com/saxon" lang="xml">

Mnguyen: /* Using multiple measures */

2012-02-02T15:23:15Z

Using multiple measures

← Older revision		Revision as of 15:23, 2 February 2012
Line 72:		Line 72:
	The SUMO-Toolbox also allows you to use several measures to guide modeling process. You may want to use more than one measure because for example you need a model which has a mean square error of 0.01 but its largest error cannot exceed a certain threshold. Or you want to use several measures to monitor the modeling process.		The SUMO-Toolbox also allows you to use several measures to guide modeling process. You may want to use more than one measure because for example you need a model which has a mean square error of 0.01 but its largest error cannot exceed a certain threshold. Or you want to use several measures to monitor the modeling process.

	Go to the [[Multi-Objective_Modeling\|Multi-Objective Modeling page]] to find out how to ~~use~~ build models using multiple measures.		Go to the [[Multi-Objective_Modeling\|Multi-Objective Modeling page]] to find out how to build models using multiple measures.
	If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding them to the Plan but with the <code>use="off"</code> keyword.		If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding them to the Plan but with the <code>use="off"</code> keyword.

Mnguyen: /* Using multiple measures */

2012-01-30T12:45:54Z

Using multiple measures

← Older revision		Revision as of 12:45, 30 January 2012
Line 73:		Line 73:

	Go to the [[Multi-Objective_Modeling\|Multi-Objective Modeling page]] to find out how to use build models using multiple measures.		Go to the [[Multi-Objective_Modeling\|Multi-Objective Modeling page]] to find out how to use build models using multiple measures.
	If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding them to the ~~plan~~ with the <code>use="off"</code> keyword.		If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding them to the Plan but with the <code>use="off"</code> keyword.

	<source xmlns:saxon="http://icl.com/saxon" lang="xml">		<source xmlns:saxon="http://icl.com/saxon" lang="xml">

Mnguyen: /* Using multiple measures */

2012-01-30T12:37:32Z

Using multiple measures

@@ Line 72: / Line 72: @@
 The SUMO-Toolbox also allows you to use several measures to guide modeling process. You may want to use more than one measure because for example you need a model which has a mean square error of 0.01 but its largest error cannot exceed a certain threshold. Or you want to use several measures to monitor the modeling process.
-To use more than one measure, simply add another <Measure> to the Plan section, the final model scores will be the average of the two scores.
+Go to the [[Multi-Objective_Modeling|Multi-Objective Modeling page]] to find out how to use build models using multiple measures.
+If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding them to the plan with the <code>use="off"</code> keyword.
 <source xmlns:saxon="http://icl.com/saxon" lang="xml">
@@ Line 105: / Line 82: @@
      ...
 </Plan>
 </source>
 == Tips on using measures ==

Mnguyen: /* Using measures */

2012-01-27T14:17:47Z

Using measures

← Older revision		Revision as of 14:17, 27 January 2012
Line 23:		Line 23:
	== Using measures ==		== Using measures ==

	To use a particular measure in the SUMO Toolbox, you have to edit the <Measure> tag in Plan section of the configuration XML-file. If you are using the configuration XML-file default.xml, please note that the <Measure> tag does NOT refer to any configuration sections below the Plan section. You can of course always add Measure configuration sections and refer to them in the Plan section using their id. Here are three equivalent examples of defining measure in the configuration xml.		To use a particular measure in the SUMO Toolbox, you have to edit the <Measure> tag in Plan section of the configuration XML-file. If you are using the configuration XML-file default.xml, please note that the <Measure> tag does NOT refer to any configuration sections below the Plan section. Instead they are defined directly in the Plan section. You can of course always add Measure configuration sections and refer to them in the Plan section using their id. Here are three equivalent examples of defining measure in the configuration xml. The default.xml also contains examples of measures in the commented out sections.

	The first is the ~~form~~ used in the default.xml and can be found within the Plan section; the measure type used is the SampleError which scores models on how well the capture the training data. The error function used to aggregate the scores for each training sample is meanSquareError. The target determines the target accuracy that has to be satisfied by the models, in this case it means that mean square error of all training examples of a model should be 0.01 or less. The option <code>use="on"</code> indicates that this measure should be used to guide the model selection criterion.		The first way is the one used in the default.xml and can be found within the Plan section; the measure type used is the SampleError which scores models on how well the capture the training data. The error function used to aggregate the scores for each training sample is meanSquareError. The target determines the target accuracy that has to be satisfied by the models, in this case it means that mean square error of all training examples of a model should be 0.01 or less. The option <code>use="on"</code> indicates that this measure should be used to guide the model selection criterion.

	<source xmlns:saxon="http://icl.com/saxon" lang="xml">		<source xmlns:saxon="http://icl.com/saxon" lang="xml">
Line 35:		Line 35:
	</source>		</source>

	The second ~~form is~~ uses an id in the Plan section to refer to the same measure, but which is now defined separately in a configuration section:		The second way uses an id in the Plan section to refer to the same measure, but which is now defined separately in a configuration section:

	<source xmlns:saxon="http://icl.com/saxon" lang="xml">		<source xmlns:saxon="http://icl.com/saxon" lang="xml">

Mnguyen: /* CrossValidation */

2012-01-27T14:13:34Z

CrossValidation

← Older revision		Revision as of 14:13, 27 January 2012
Line 159:		Line 159:
	\|description = Folds are generated from scratch for each model that is evaluated using this measure. If the same model is evaluated twice (for example, after a rebuild), new folds are used. Enabling this feature can be very costly for large sample sizes. As a rule of thumb, enable this in combination with the random partition method, or disable this when using the uniform method.		\|description = Folds are generated from scratch for each model that is evaluated using this measure. If the same model is evaluated twice (for example, after a rebuild), new folds are used. Enabling this feature can be very costly for large sample sizes. As a rule of thumb, enable this in combination with the random partition method, or disable this when using the uniform method.
	}}		}}

			Here is an example configuration for the CrossValidation measure.

	<source xmlns:saxon="http://icl.com/saxon" lang="xml">		<source xmlns:saxon="http://icl.com/saxon" lang="xml">

Mnguyen: /* CrossValidation */

2012-01-27T14:11:52Z

CrossValidation

← Older revision		Revision as of 14:11, 27 January 2012
Line 160:		Line 160:
	}}		}}

			<source xmlns:saxon="http://icl.com/saxon" lang="xml">
	<Measure type ="CrossValidation">		<Measure type ="CrossValidation">
	<Option key="use" value="on"/>		<Option key="use" value="on"/>
Line 170:		Line 170:
	<Option key="resetFolds" value="no"/>		<Option key="resetFolds" value="no"/>
	</Measure>		</Measure>
			</source>

	=== ValidationSet ===		=== ValidationSet ===