Compare commits
15 commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
044bf08e3d | ||
|
|
c4bab39245 | ||
| f3a4ef01ed | |||
|
|
54af805d4d | ||
| 79a9522ba7 | |||
|
|
c24626ff61 | ||
|
|
51696e2546 | ||
|
|
f1c5b292ed | ||
|
|
a56ad4433d | ||
|
|
f23ee21ef3 | ||
|
|
186de413ed | ||
|
|
86f6c195d7 | ||
|
|
9258f75e4e | ||
|
|
7371dab4f1 | ||
|
|
ae9a6b9a3f |
127 changed files with 3065 additions and 2272 deletions
README.md
executable
pom.xml
src
main/java/ca/joeltherrien/randomforest
Main.javaSettings.java
covariates/settings
BooleanCovariateSettings.javaCovariateSettings.javaFactorCovariateSettings.javaNumericCovariateSettings.java
loaders
utils
test/java/ca/joeltherrien/randomforest
library
pom.xml
pom.xmlsrc
main/java/ca/joeltherrien/randomforest
responses/competingrisk/combiner
utils
test/java/ca/joeltherrien/randomforest
src
main/java/ca/joeltherrien/randomforest
Bootstrapper.javaCovariateRow.javaRow.javaVisibleForTesting.java
covariates
responses
competingrisk
CompetingRiskErrorRateCalculator.javaCompetingRiskFunctions.javaCompetingRiskGraySetsImpl.javaCompetingRiskResponse.javaCompetingRiskResponseWithCensorTime.javaCompetingRiskSets.javaCompetingRiskSetsImpl.javaCompetingRiskUtils.javaIBSCalculator.java
combiner
CompetingRiskFunctionCombiner.javaCompetingRiskResponseCombiner.javaIntermediateCompetingRisksFunctionsTimesKnown.java
splitfinder
regression
tree
Forest.javaForestResponseCombiner.javaForestTrainer.javaIntermediateCombinedResponse.javaNode.javaOfflineForest.javaOnlineForest.javaResponseCombiner.javaSimpleSplitFinder.javaSplit.javaSplitAndScore.javaSplitFinder.javaSplitNode.javaTerminalNode.javaTree.javaTreeTrainer.java
vimp
utils
test/java/ca/joeltherrien/randomforest
34
README.md
34
README.md
|
|
@ -1,21 +1,18 @@
|
|||
# README
|
||||
|
||||
This repository contains two Java projects;
|
||||
This repository contains the largeRCRF Java library, containing all of the logic used for training the random forests. This provides the Jar file used in the R package [largeRCRF](https://github.com/jatherrien/largeRCRF).
|
||||
|
||||
* The first is the largeRCRF library (`library/`) containing all of the logic used for training the random forests. This part provides the Jar file used in the R package [largeRCRF](https://github.com/jatherrien/largeRCRF).
|
||||
* The second is a small executable (`executable/`) Java project that uses the library and can be run directly outside of R. It's still in its early stages and isn't polished, nor is it yet well documented; but you can take a look if you want.
|
||||
|
||||
Most users interested in training random competing risks forests should use the [R package component](https://github.com/jatherrien/largeRCRF); the content in this repository will only be useful to advanced users.
|
||||
Most users interested in training random competing risks forests should use the [R package component](https://github.com/jatherrien/largeRCRF); the content in this repository is only useful for advanced users.
|
||||
|
||||
## License
|
||||
|
||||
You're free to use / modify / redistribute either of the two projects above, as long as you follow the terms of the GPL-3 license.
|
||||
You're free to use / modify / redistribute the project, as long as you follow the terms of the GPL-3 license.
|
||||
|
||||
## Extending
|
||||
|
||||
Documentation on how to extend the library to add support for other types of random forests will eventually be added, but for now if you're interested in that I suggest you take a look at the `MeanResponseCombiner` and `WeightedVarianceSplitFinder` classes to see how some basic regression random forests were introduced.
|
||||
|
||||
If you've made a modification to the package and would like to integrate it into the R package component, build the project in Maven with `mvn clean package` (in the same directory as this `README` file), then just copy `library/target/largeRCRF-library-1.0-SNAPSHOT.jar` into the `inst/java/` directory for the R package, replacing the previous Jar file there. Then build the R package, possibly with your modifications to the code there, with `R> devtools::build()`.
|
||||
If you've made a modification to the package and would like to integrate it into the R package component, build the project in Maven with `mvn clean package`, then just copy `target/largeRCRF-library-1.0-SNAPSHOT.jar` into the `inst/java/` directory for the R package, replacing the previous Jar file there. Then build the R package, possibly with your modifications to the code there, with `R> devtools::build()`.
|
||||
|
||||
Please don't take the current lack of documentation as a sign that I oppose others extending or modifying the project; if you have any questions on running, extending, integrating with R, or anything else related to this project, please don't hesitate to either [email me](mailto:joelt@sfu.ca) or create an Issue. Most likely my answers to your questions will end up forming the basis for any documentation written.
|
||||
|
||||
|
|
@ -26,26 +23,3 @@ You need:
|
|||
* A Java runtime version 1.8 or greater
|
||||
* Maven to build the project
|
||||
|
||||
## Troubleshooting (Running `executable`)
|
||||
|
||||
Some of these Troubleshooting items can also apply if you are integrating the library classes into your own Java project.
|
||||
|
||||
### I get an `OutOfMemoryException` error but I have plenty of RAM
|
||||
|
||||
By default the Java virtual machine only uses a quarter of the available system memory. When launching the Jar file you can manually specify the memory available like below:
|
||||
```
|
||||
java -jar -Xmx15G -Xms15G largeRCRF-executable-1.0-SNAPSHOT.jar settings.yaml
|
||||
```
|
||||
|
||||
with `15G` replaced with a little less than your available system memory.
|
||||
|
||||
### I get an `OutOfMemoryException` error and I'm short on RAM
|
||||
|
||||
Try reducing the number of trees being trained simultaneously by reducing the number of threads in the settings file.
|
||||
|
||||
### Training stalls immediately at 0 trees and the CPU is idle
|
||||
|
||||
This issue has been observed before on one particular system (and only on that system) but it's not clear what causes it. If you encounter this, please open an Issue and describe what operating system you're running on, what cloud system (if relevant) you're running on, and the entire output of `java --version`.
|
||||
|
||||
From my observation this issues occurs randomly but only at 0 trees; so as an imperfect workaround you can cancel the training and try again. Another imperfect workaround is to set the number of threads to 1; this causes the code to not use Java's parallel code capabilities which will bypass the problem (at the cost of slower training).
|
||||
|
||||
|
|
|
|||
|
|
@ -1,126 +0,0 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<parent>
|
||||
<artifactId>largeRCRF</artifactId>
|
||||
<groupId>ca.joeltherrien</groupId>
|
||||
<version>1.0-SNAPSHOT</version>
|
||||
</parent>
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<artifactId>largeRCRF-executable</artifactId>
|
||||
|
||||
<properties>
|
||||
<java.version>1.8</java.version>
|
||||
<maven.compiler.target>1.8</maven.compiler.target>
|
||||
<maven.compiler.source>1.8</maven.compiler.source>
|
||||
<jackson.version>2.9.9</jackson.version>
|
||||
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<version>1.18.0</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.commons</groupId>
|
||||
<artifactId>commons-csv</artifactId>
|
||||
<version>1.5</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.fasterxml.jackson.core</groupId>
|
||||
<artifactId>jackson-databind</artifactId>
|
||||
<!--<version>${jackson.version}</version> 2.9.9.1 fixes some vulnerability; other Jackson dependencies are still on 2.9.9-->
|
||||
<version>2.9.9.1</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.fasterxml.jackson.dataformat</groupId>
|
||||
<artifactId>jackson-dataformat-yaml</artifactId>
|
||||
<version>${jackson.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-api</artifactId>
|
||||
<version>5.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-engine</artifactId>
|
||||
<version>5.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mockito</groupId>
|
||||
<artifactId>mockito-core</artifactId>
|
||||
<version>2.20.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>ca.joeltherrien</groupId>
|
||||
<artifactId>largeRCRF-library</artifactId>
|
||||
<version>1.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>shade</goal>
|
||||
</goals>
|
||||
<configuration>
|
||||
<transformers>
|
||||
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
|
||||
<mainClass>
|
||||
ca.joeltherrien.randomforest.Main
|
||||
</mainClass>
|
||||
</transformer>
|
||||
</transformers>
|
||||
<minimizeJar>true</minimizeJar>
|
||||
</configuration>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-pmd-plugin</artifactId>
|
||||
<version>3.11.0</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase> <!-- bind to the packaging phase -->
|
||||
<goals>
|
||||
<goal>check</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
<configuration>
|
||||
<rulesets>
|
||||
<!-- Custom local file system rule set -->
|
||||
<ruleset>${project.basedir}/../pmd-rules.xml</ruleset>
|
||||
</rulesets>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
|
||||
</project>
|
||||
|
|
@ -1,252 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest;
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.BooleanCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.FactorCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.NumericCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskErrorRateCalculator;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskFunctions;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskResponse;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskResponseWithCensorTime;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.combiner.CompetingRiskFunctionCombiner;
|
||||
import ca.joeltherrien.randomforest.tree.Forest;
|
||||
import ca.joeltherrien.randomforest.tree.ForestTrainer;
|
||||
import ca.joeltherrien.randomforest.tree.ResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.tree.TreeTrainer;
|
||||
import ca.joeltherrien.randomforest.utils.CSVUtils;
|
||||
import ca.joeltherrien.randomforest.utils.DataUtils;
|
||||
import ca.joeltherrien.randomforest.utils.StepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.fasterxml.jackson.databind.node.TextNode;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.io.PrintWriter;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Random;
|
||||
|
||||
public class Main {
|
||||
|
||||
public static void main(String[] args) throws IOException, ClassNotFoundException {
|
||||
if(args.length < 2){
|
||||
System.out.println("Must provide two arguments - the path to the settings.yaml file and instructions to either train or analyze.");
|
||||
System.out.println("Note that analyzing only supports competing risk data, and that you must then specify a sample size for testing errors.");
|
||||
if(args.length == 0){
|
||||
final File templateFile = new File("template.yaml");
|
||||
if(templateFile.exists()){
|
||||
System.out.println("Template file exists; not creating a new one");
|
||||
}
|
||||
else{
|
||||
System.out.println("Generating template file.");
|
||||
defaultTemplate().save(templateFile);
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
return;
|
||||
}
|
||||
final File settingsFile = new File(args[0]);
|
||||
final Settings settings = Settings.load(settingsFile);
|
||||
|
||||
final List<Covariate> covariates = settings.getCovariates();
|
||||
|
||||
if(args[1].equalsIgnoreCase("train")){
|
||||
final List<Row<Object>> dataset = CSVUtils.loadData(covariates, settings.getResponseLoader(), settings.getTrainingDataLocation());
|
||||
|
||||
final ForestTrainer forestTrainer = constructForestTrainer(settings, dataset, covariates);
|
||||
|
||||
if(settings.isSaveProgress()){
|
||||
if(settings.getNumberOfThreads() > 1){
|
||||
forestTrainer.trainParallelOnDisk(Optional.empty(), settings.getNumberOfThreads());
|
||||
} else{
|
||||
forestTrainer.trainSerialOnDisk(Optional.empty());
|
||||
}
|
||||
}
|
||||
else{
|
||||
if(settings.getNumberOfThreads() > 1){
|
||||
forestTrainer.trainParallelInMemory(Optional.empty(), settings.getNumberOfThreads());
|
||||
} else{
|
||||
forestTrainer.trainSerialInMemory(Optional.empty());
|
||||
}
|
||||
}
|
||||
}
|
||||
else if(args[1].equalsIgnoreCase("analyze")){
|
||||
// Perform different prediction measures
|
||||
|
||||
if(args.length < 3){
|
||||
System.out.println("Specify error sample size");
|
||||
return;
|
||||
}
|
||||
|
||||
final String yVarType = settings.getYVarSettings().get("type").asText();
|
||||
|
||||
if(!yVarType.equalsIgnoreCase("CompetingRiskResponse") && !yVarType.equalsIgnoreCase("CompetingRiskResponseWithCensorTime")){
|
||||
System.out.println("Analyze currently only works on competing risk data");
|
||||
return;
|
||||
}
|
||||
|
||||
final ResponseCombiner<?, CompetingRiskFunctions> responseCombiner = settings.getTreeCombiner();
|
||||
final int[] events;
|
||||
|
||||
if(responseCombiner instanceof CompetingRiskFunctionCombiner){
|
||||
events = ((CompetingRiskFunctionCombiner) responseCombiner).getEvents();
|
||||
}
|
||||
else{
|
||||
System.out.println("Unsupported tree combiner");
|
||||
return;
|
||||
}
|
||||
|
||||
final List<Row<CompetingRiskResponse>> dataset = CSVUtils.loadData(covariates, settings.getResponseLoader(), settings.getValidationDataLocation());
|
||||
|
||||
// Let's reduce this down to n
|
||||
final int n = Integer.parseInt(args[2]);
|
||||
Utils.reduceListToSize(dataset, n, new Random());
|
||||
|
||||
final File folder = new File(settings.getSaveTreeLocation());
|
||||
final Forest<?, CompetingRiskFunctions> forest = DataUtils.loadForest(folder, responseCombiner);
|
||||
|
||||
final boolean useBootstrapPredictions = settings.getTrainingDataLocation().equals(settings.getValidationDataLocation());
|
||||
|
||||
if(useBootstrapPredictions){
|
||||
System.out.println("Finished loading trees + dataset; creating calculator and evaluating OOB predictions");
|
||||
}
|
||||
else{
|
||||
System.out.println("Finished loading trees + dataset; creating calculator and evaluating predictions");
|
||||
}
|
||||
|
||||
|
||||
final CompetingRiskErrorRateCalculator errorRateCalculator = new CompetingRiskErrorRateCalculator(dataset, forest, useBootstrapPredictions);
|
||||
final PrintWriter printWriter = new PrintWriter(settings.getSaveTreeLocation() + "/errors.txt");
|
||||
|
||||
System.out.println("Running Naive Concordance");
|
||||
|
||||
final double[] naiveConcordance = errorRateCalculator.calculateConcordance(events);
|
||||
printWriter.write("Naive concordance:\n");
|
||||
for(int i=0; i<events.length; i++){
|
||||
printWriter.write('\t');
|
||||
printWriter.write(Integer.toString(events[i]));
|
||||
printWriter.write(": ");
|
||||
printWriter.write(Double.toString(naiveConcordance[i]));
|
||||
printWriter.write('\n');
|
||||
}
|
||||
|
||||
if(yVarType.equalsIgnoreCase("CompetingRiskResponseWithCensorTime")){
|
||||
System.out.println("Running IPCW Concordance - creating censor distribution");
|
||||
|
||||
final double[] censorTimes = dataset.stream()
|
||||
.mapToDouble(row -> ((CompetingRiskResponseWithCensorTime) row.getResponse()).getC())
|
||||
.toArray();
|
||||
final StepFunction censorDistribution = Utils.estimateOneMinusECDF(censorTimes);
|
||||
|
||||
System.out.println("Finished generating censor distribution - running concordance");
|
||||
|
||||
final double[] ipcwConcordance = errorRateCalculator.calculateIPCWConcordance(events, censorDistribution);
|
||||
printWriter.write("IPCW concordance:\n");
|
||||
for(int i=0; i<events.length; i++){
|
||||
printWriter.write('\t');
|
||||
printWriter.write(Integer.toString(events[i]));
|
||||
printWriter.write(": ");
|
||||
printWriter.write(Double.toString(ipcwConcordance[i]));
|
||||
printWriter.write('\n');
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
printWriter.close();
|
||||
}
|
||||
else{
|
||||
System.out.println("Invalid instruction; use either train or analyze.");
|
||||
System.out.println("Note that analyzing only supports competing risk data.");
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private static ForestTrainer constructForestTrainer(final Settings settings, final List<Row<Object>> data, final List<Covariate> covariates){
|
||||
|
||||
return ForestTrainer.builder()
|
||||
.ntree(settings.getNtree())
|
||||
.data(data)
|
||||
.displayProgress(true)
|
||||
.saveTreeLocation(settings.getSaveTreeLocation())
|
||||
.covariates(covariates)
|
||||
.treeResponseCombiner(settings.getTreeCombiner())
|
||||
.treeTrainer(constructTreeTrainer(settings, covariates))
|
||||
.randomSeed(settings.getRandomSeed() != null ? settings.getRandomSeed() : System.nanoTime())
|
||||
.build();
|
||||
|
||||
}
|
||||
|
||||
private static TreeTrainer constructTreeTrainer(final Settings settings, final List<Covariate> covariates) {
|
||||
|
||||
return TreeTrainer.builder()
|
||||
.numberOfSplits(settings.getNumberOfSplits())
|
||||
.nodeSize(settings.getNodeSize())
|
||||
.maxNodeDepth(settings.getMaxNodeDepth())
|
||||
.mtry(settings.getMtry())
|
||||
.checkNodePurity(settings.isCheckNodePurity())
|
||||
.responseCombiner(settings.getResponseCombiner())
|
||||
.splitFinder(settings.getSplitFinder())
|
||||
.covariates(covariates)
|
||||
.build();
|
||||
}
|
||||
|
||||
|
||||
private static Settings defaultTemplate(){
|
||||
|
||||
final ObjectNode splitFinderSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
splitFinderSettings.set("type", new TextNode("WeightedVarianceSplitFinder"));
|
||||
|
||||
final ObjectNode responseCombinerSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
responseCombinerSettings.set("type", new TextNode("MeanResponseCombiner"));
|
||||
|
||||
final ObjectNode treeCombinerSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
treeCombinerSettings.set("type", new TextNode("MeanResponseCombiner"));
|
||||
|
||||
final ObjectNode yVarSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
yVarSettings.set("type", new TextNode("Double"));
|
||||
yVarSettings.set("name", new TextNode("y"));
|
||||
|
||||
return Settings.builder()
|
||||
.covariateSettings(Utils.easyList(
|
||||
new NumericCovariateSettings("x1"),
|
||||
new BooleanCovariateSettings("x2"),
|
||||
new FactorCovariateSettings("x3", Utils.easyList("cat", "mouse", "dog"))
|
||||
)
|
||||
)
|
||||
.trainingDataLocation("training_data.csv")
|
||||
.validationDataLocation("validation_data.csv")
|
||||
.responseCombinerSettings(responseCombinerSettings)
|
||||
.treeCombinerSettings(treeCombinerSettings)
|
||||
.splitFinderSettings(splitFinderSettings)
|
||||
.yVarSettings(yVarSettings)
|
||||
.maxNodeDepth(100000)
|
||||
.mtry(2)
|
||||
.nodeSize(5)
|
||||
.ntree(500)
|
||||
.numberOfSplits(5)
|
||||
.numberOfThreads(1)
|
||||
.saveProgress(true)
|
||||
.saveTreeLocation("trees/")
|
||||
.build();
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -1,236 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest;
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.CovariateSettings;
|
||||
import ca.joeltherrien.randomforest.loaders.CompetingResponseLoader;
|
||||
import ca.joeltherrien.randomforest.loaders.CompetingResponseWithCensorTimeLoader;
|
||||
import ca.joeltherrien.randomforest.loaders.DoubleLoader;
|
||||
import ca.joeltherrien.randomforest.loaders.ResponseLoader;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.combiner.CompetingRiskFunctionCombiner;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.combiner.CompetingRiskResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.splitfinder.GrayLogRankSplitFinder;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.splitfinder.LogRankSplitFinder;
|
||||
import ca.joeltherrien.randomforest.responses.regression.MeanResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.responses.regression.WeightedVarianceSplitFinder;
|
||||
import ca.joeltherrien.randomforest.tree.ResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.tree.SplitFinder;
|
||||
import ca.joeltherrien.randomforest.utils.JsonUtils;
|
||||
import com.fasterxml.jackson.annotation.JsonIgnore;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Function;
|
||||
|
||||
/**
|
||||
* This class is saved & loaded using a saved configuration file. It contains all relevant settings when training a forest.
|
||||
*/
|
||||
@Data
|
||||
@Builder
|
||||
@AllArgsConstructor
|
||||
@EqualsAndHashCode
|
||||
public class Settings {
|
||||
|
||||
private static Map<String, Function<ObjectNode, ResponseLoader>> RESPONSE_LOADER_MAP = new HashMap<>();
|
||||
public static Function<ObjectNode, ResponseLoader> getResponseLoaderConstructor(final String name){
|
||||
return RESPONSE_LOADER_MAP.get(name.toLowerCase());
|
||||
}
|
||||
public static void registerResponseLoaderConstructor(final String name, final Function<ObjectNode, ResponseLoader> responseLoaderConstructor){
|
||||
RESPONSE_LOADER_MAP.put(name.toLowerCase(), responseLoaderConstructor);
|
||||
}
|
||||
|
||||
static{
|
||||
registerResponseLoaderConstructor("double",
|
||||
node -> new DoubleLoader(node)
|
||||
);
|
||||
registerResponseLoaderConstructor("CompetingRiskResponse",
|
||||
node -> new CompetingResponseLoader(node)
|
||||
);
|
||||
registerResponseLoaderConstructor("CompetingRiskResponseWithCensorTime",
|
||||
node -> new CompetingResponseWithCensorTimeLoader(node)
|
||||
);
|
||||
}
|
||||
|
||||
private static Map<String, Function<ObjectNode, SplitFinder>> SPLIT_FINDER_MAP = new HashMap<>();
|
||||
public static Function<ObjectNode, SplitFinder> getSplitFinderConstructor(final String name){
|
||||
return SPLIT_FINDER_MAP.get(name.toLowerCase());
|
||||
}
|
||||
public static void registerSplitFinderConstructor(final String name, final Function<ObjectNode, SplitFinder> splitFinderConstructor){
|
||||
SPLIT_FINDER_MAP.put(name.toLowerCase(), splitFinderConstructor);
|
||||
}
|
||||
static{
|
||||
registerSplitFinderConstructor("WeightedVarianceSplitFinder",
|
||||
(node) -> new WeightedVarianceSplitFinder()
|
||||
);
|
||||
registerSplitFinderConstructor("GrayLogRankSplitFinder",
|
||||
(objectNode) -> {
|
||||
final int[] eventsOfFocusArray = JsonUtils.jsonToIntArray(objectNode.get("eventsOfFocus"));
|
||||
final int[] eventArray = JsonUtils.jsonToIntArray(objectNode.get("events"));
|
||||
|
||||
return new GrayLogRankSplitFinder(eventsOfFocusArray, eventArray);
|
||||
}
|
||||
);
|
||||
registerSplitFinderConstructor("LogRankSplitFinder",
|
||||
(objectNode) -> {
|
||||
final int[] eventsOfFocusArray = JsonUtils.jsonToIntArray(objectNode.get("eventsOfFocus"));
|
||||
final int[] eventArray = JsonUtils.jsonToIntArray(objectNode.get("events"));
|
||||
|
||||
return new LogRankSplitFinder(eventsOfFocusArray, eventArray);
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
|
||||
private static Map<String, Function<ObjectNode, ResponseCombiner>> RESPONSE_COMBINER_MAP = new HashMap<>();
|
||||
public static Function<ObjectNode, ResponseCombiner> getResponseCombinerConstructor(final String name){
|
||||
return RESPONSE_COMBINER_MAP.get(name.toLowerCase());
|
||||
}
|
||||
public static void registerResponseCombinerConstructor(final String name, final Function<ObjectNode, ResponseCombiner> responseCombinerConstructor){
|
||||
RESPONSE_COMBINER_MAP.put(name.toLowerCase(), responseCombinerConstructor);
|
||||
}
|
||||
|
||||
static{
|
||||
|
||||
registerResponseCombinerConstructor("MeanResponseCombiner",
|
||||
(node) -> new MeanResponseCombiner()
|
||||
);
|
||||
registerResponseCombinerConstructor("CompetingRiskResponseCombiner",
|
||||
(node) -> {
|
||||
final int[] events = JsonUtils.jsonToIntArray(node.get("events"));
|
||||
|
||||
return new CompetingRiskResponseCombiner(events);
|
||||
|
||||
}
|
||||
);
|
||||
|
||||
registerResponseCombinerConstructor("CompetingRiskFunctionCombiner",
|
||||
(node) -> {
|
||||
final int[] events = JsonUtils.jsonToIntArray(node.get("events"));
|
||||
|
||||
double[] times = null;
|
||||
if(node.hasNonNull("times")){
|
||||
times = JsonUtils.jsonToDoubleArray(node.get("times"));
|
||||
}
|
||||
|
||||
return new CompetingRiskFunctionCombiner(events, times);
|
||||
|
||||
}
|
||||
);
|
||||
|
||||
|
||||
}
|
||||
|
||||
private int numberOfSplits = 5;
|
||||
private int nodeSize = 5;
|
||||
private int maxNodeDepth = 1000000; // basically no maxNodeDepth
|
||||
private boolean checkNodePurity = false;
|
||||
private Long randomSeed;
|
||||
|
||||
private ObjectNode responseCombinerSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
private ObjectNode splitFinderSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
private ObjectNode treeCombinerSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
|
||||
private List<CovariateSettings> covariateSettings = new ArrayList<>();
|
||||
private ObjectNode yVarSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
|
||||
// number of covariates to randomly try
|
||||
private int mtry = 0;
|
||||
|
||||
// number of trees to try
|
||||
private int ntree = 500;
|
||||
|
||||
private String trainingDataLocation = "data.csv";
|
||||
private String validationDataLocation = "data.csv";
|
||||
private String saveTreeLocation = "trees/";
|
||||
|
||||
private int numberOfThreads = 1;
|
||||
private boolean saveProgress = false;
|
||||
|
||||
public Settings(){
|
||||
} // required for Jackson
|
||||
|
||||
public static Settings load(File file) throws IOException {
|
||||
final ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
|
||||
//mapper.enableDefaultTyping();
|
||||
|
||||
return mapper.readValue(file, Settings.class);
|
||||
|
||||
}
|
||||
|
||||
public void save(File file) throws IOException {
|
||||
final ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
|
||||
//mapper.enableDefaultTyping();
|
||||
|
||||
// Jackson can struggle with some types of Lists, such as that returned by the useful List.of(...)
|
||||
this.covariateSettings = new ArrayList<>(this.covariateSettings);
|
||||
|
||||
mapper.writeValue(file, this);
|
||||
}
|
||||
|
||||
@JsonIgnore
|
||||
public SplitFinder getSplitFinder(){
|
||||
final String type = splitFinderSettings.get("type").asText();
|
||||
|
||||
return getSplitFinderConstructor(type).apply(splitFinderSettings);
|
||||
}
|
||||
|
||||
@JsonIgnore
|
||||
public ResponseLoader getResponseLoader(){
|
||||
final String type = yVarSettings.get("type").asText();
|
||||
|
||||
return getResponseLoaderConstructor(type).apply(yVarSettings);
|
||||
}
|
||||
|
||||
@JsonIgnore
|
||||
public ResponseCombiner getResponseCombiner(){
|
||||
final String type = responseCombinerSettings.get("type").asText();
|
||||
|
||||
return getResponseCombinerConstructor(type).apply(responseCombinerSettings);
|
||||
}
|
||||
|
||||
@JsonIgnore
|
||||
public ResponseCombiner getTreeCombiner(){
|
||||
final String type = treeCombinerSettings.get("type").asText();
|
||||
|
||||
return getResponseCombinerConstructor(type).apply(treeCombinerSettings);
|
||||
}
|
||||
|
||||
@JsonIgnore
|
||||
public List<Covariate> getCovariates(){
|
||||
final List<CovariateSettings> covariateSettingsList = this.getCovariateSettings();
|
||||
final List<Covariate> covariates = new ArrayList<>(covariateSettingsList.size());
|
||||
for(int i = 0; i < covariateSettingsList.size(); i++){
|
||||
covariates.add(covariateSettingsList.get(i).build(i));
|
||||
}
|
||||
return covariates;
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.covariates.settings;
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.bool.BooleanCovariate;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
@NoArgsConstructor // required by Jackson
|
||||
@Data
|
||||
public final class BooleanCovariateSettings extends CovariateSettings<Boolean> {
|
||||
|
||||
public BooleanCovariateSettings(String name){
|
||||
super(name);
|
||||
}
|
||||
|
||||
@Override
|
||||
public BooleanCovariate build(int index) {
|
||||
return new BooleanCovariate(name, index);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.covariates.settings;
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import com.fasterxml.jackson.annotation.JsonSubTypes;
|
||||
import com.fasterxml.jackson.annotation.JsonTypeInfo;
|
||||
import lombok.Getter;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/**
|
||||
* Nuisance class to work with Jackson for persisting settings.
|
||||
*
|
||||
* @param <V>
|
||||
*/
|
||||
@NoArgsConstructor // required for Jackson
|
||||
@Getter
|
||||
@JsonTypeInfo(
|
||||
use = JsonTypeInfo.Id.NAME,
|
||||
property = "type")
|
||||
@JsonSubTypes({
|
||||
@JsonSubTypes.Type(value = BooleanCovariateSettings.class, name = "boolean"),
|
||||
@JsonSubTypes.Type(value = NumericCovariateSettings.class, name = "numeric"),
|
||||
@JsonSubTypes.Type(value = FactorCovariateSettings.class, name = "factor")
|
||||
})
|
||||
public abstract class CovariateSettings<V> {
|
||||
|
||||
String name;
|
||||
|
||||
CovariateSettings(String name){
|
||||
this.name = name;
|
||||
}
|
||||
|
||||
public abstract Covariate<V> build(int index);
|
||||
}
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.covariates.settings;
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.factor.FactorCovariate;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
@NoArgsConstructor // required by Jackson
|
||||
@Data
|
||||
public final class FactorCovariateSettings extends CovariateSettings<String> {
|
||||
|
||||
private List<String> levels;
|
||||
|
||||
public FactorCovariateSettings(String name, List<String> levels){
|
||||
super(name);
|
||||
this.levels = new ArrayList<>(levels); // Jackson struggles with List.of(...)
|
||||
}
|
||||
|
||||
@Override
|
||||
public FactorCovariate build(int index) {
|
||||
return new FactorCovariate(name, index, levels);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.covariates.settings;
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.numeric.NumericCovariate;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
@NoArgsConstructor // required by Jackson
|
||||
@Data
|
||||
public final class NumericCovariateSettings extends CovariateSettings<Double> {
|
||||
|
||||
public NumericCovariateSettings(String name){
|
||||
super(name);
|
||||
}
|
||||
|
||||
@Override
|
||||
public NumericCovariate build(int index) {
|
||||
return new NumericCovariate(name, index);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.loaders;
|
||||
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskResponse;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import org.apache.commons.csv.CSVRecord;
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public class CompetingResponseLoader implements ResponseLoader<CompetingRiskResponse> {
|
||||
|
||||
private final String deltaName;
|
||||
private final String uName;
|
||||
|
||||
public CompetingResponseLoader(ObjectNode node){
|
||||
this.deltaName = node.get("delta").asText();
|
||||
this.uName = node.get("u").asText();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompetingRiskResponse parse(CSVRecord record) {
|
||||
final int delta = Integer.parseInt(record.get(deltaName));
|
||||
final double u = Double.parseDouble(record.get(uName));
|
||||
|
||||
return new CompetingRiskResponse(delta, u);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.loaders;
|
||||
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskResponseWithCensorTime;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import org.apache.commons.csv.CSVRecord;
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public class CompetingResponseWithCensorTimeLoader implements ResponseLoader<CompetingRiskResponseWithCensorTime>{
|
||||
|
||||
private final String deltaName;
|
||||
private final String uName;
|
||||
private final String cName;
|
||||
|
||||
public CompetingResponseWithCensorTimeLoader(ObjectNode node){
|
||||
this.deltaName = node.get("delta").asText();
|
||||
this.uName = node.get("u").asText();
|
||||
this.cName = node.get("c").asText();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompetingRiskResponseWithCensorTime parse(CSVRecord record) {
|
||||
final int delta = Integer.parseInt(record.get(deltaName));
|
||||
final double u = Double.parseDouble(record.get(uName));
|
||||
final double c = Double.parseDouble(record.get(cName));
|
||||
|
||||
return new CompetingRiskResponseWithCensorTime(delta, u, c);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.loaders;
|
||||
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import org.apache.commons.csv.CSVRecord;
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public class DoubleLoader implements ResponseLoader<Double> {
|
||||
|
||||
private final String yName;
|
||||
|
||||
public DoubleLoader(final ObjectNode node){
|
||||
this.yName = node.get("name").asText();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Double parse(CSVRecord record) {
|
||||
return Double.parseDouble(record.get(yName));
|
||||
}
|
||||
}
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
import ca.joeltherrien.randomforest.Row;
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.loaders.ResponseLoader;
|
||||
import org.apache.commons.csv.CSVFormat;
|
||||
import org.apache.commons.csv.CSVParser;
|
||||
import org.apache.commons.csv.CSVRecord;
|
||||
|
||||
import java.io.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.zip.GZIPInputStream;
|
||||
|
||||
public class CSVUtils {
|
||||
|
||||
public static <Y> List<Row<Y>> loadData(final List<Covariate> covariates, final ResponseLoader<Y> responseLoader, String filename) throws IOException {
|
||||
|
||||
final List<Row<Y>> dataset = new ArrayList<>();
|
||||
|
||||
final Reader input;
|
||||
if(filename.endsWith(".gz")){
|
||||
final FileInputStream inputStream = new FileInputStream(filename);
|
||||
final GZIPInputStream gzipInputStream = new GZIPInputStream(inputStream);
|
||||
|
||||
input = new InputStreamReader(gzipInputStream);
|
||||
}
|
||||
else{
|
||||
input = new FileReader(filename);
|
||||
}
|
||||
|
||||
|
||||
final CSVParser parser = CSVFormat.RFC4180.withFirstRecordAsHeader().parse(input);
|
||||
|
||||
|
||||
int id = 1;
|
||||
for(final CSVRecord record : parser){
|
||||
final Covariate.Value[] valueArray = new Covariate.Value[covariates.size()];
|
||||
|
||||
for(final Covariate<?> covariate : covariates){
|
||||
valueArray[covariate.getIndex()] = covariate.createValue(record.get(covariate.getName()));
|
||||
}
|
||||
|
||||
final Y y = responseLoader.parse(record);
|
||||
|
||||
dataset.add(new Row<>(valueArray, id++, y));
|
||||
|
||||
}
|
||||
|
||||
return dataset;
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
|
||||
public class JsonUtils {
|
||||
|
||||
public static int[] jsonToIntArray(final JsonNode node){
|
||||
final Iterator<JsonNode> elements = node.elements();
|
||||
final List<JsonNode> elementList = new ArrayList<>();
|
||||
elements.forEachRemaining(n -> elementList.add(n));
|
||||
|
||||
final int[] array = elementList.stream().mapToInt(n -> n.asInt()).toArray();
|
||||
return array;
|
||||
}
|
||||
|
||||
public static double[] jsonToDoubleArray(final JsonNode node){
|
||||
final Iterator<JsonNode> elements = node.elements();
|
||||
final List<JsonNode> elementList = new ArrayList<>();
|
||||
elements.forEachRemaining(n -> elementList.add(n));
|
||||
|
||||
final double[] array = elementList.stream().mapToDouble(n -> n.asDouble()).toArray();
|
||||
return array;
|
||||
}
|
||||
}
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.settings.BooleanCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.FactorCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.NumericCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.fasterxml.jackson.databind.node.TextNode;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
|
||||
public class TestPersistence {
|
||||
|
||||
@Test
|
||||
public void testSaving() throws IOException {
|
||||
final ObjectNode splitFinderSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
splitFinderSettings.set("type", new TextNode("WeightedVarianceSplitFinder"));
|
||||
|
||||
final ObjectNode responseCombinerSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
responseCombinerSettings.set("type", new TextNode("MeanResponseCombiner"));
|
||||
|
||||
final ObjectNode treeCombinerSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
treeCombinerSettings.set("type", new TextNode("MeanResponseCombiner"));
|
||||
|
||||
final ObjectNode yVarSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
yVarSettings.set("type", new TextNode("Double"));
|
||||
yVarSettings.set("name", new TextNode("y"));
|
||||
|
||||
final Settings settingsOriginal = Settings.builder()
|
||||
.covariateSettings(Utils.easyList(
|
||||
new NumericCovariateSettings("x1"),
|
||||
new BooleanCovariateSettings("x2"),
|
||||
new FactorCovariateSettings("x3", Utils.easyList("cat", "mouse", "dog"))
|
||||
)
|
||||
)
|
||||
.trainingDataLocation("training_data.csv")
|
||||
.validationDataLocation("validation_data.csv")
|
||||
.responseCombinerSettings(responseCombinerSettings)
|
||||
.treeCombinerSettings(treeCombinerSettings)
|
||||
.splitFinderSettings(splitFinderSettings)
|
||||
.yVarSettings(yVarSettings)
|
||||
.maxNodeDepth(100000)
|
||||
.mtry(2)
|
||||
.nodeSize(5)
|
||||
.ntree(500)
|
||||
.numberOfSplits(5)
|
||||
.numberOfThreads(1)
|
||||
.saveProgress(true)
|
||||
.saveTreeLocation("trees/")
|
||||
.build();
|
||||
|
||||
final File templateFile = new File("template.yaml");
|
||||
settingsOriginal.save(templateFile);
|
||||
|
||||
final Settings reloadedSettings = Settings.load(templateFile);
|
||||
|
||||
assertEquals(settingsOriginal, reloadedSettings);
|
||||
|
||||
//templateFile.delete();
|
||||
|
||||
|
||||
}
|
||||
}
|
||||
|
|
@ -1,120 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.csv;
|
||||
|
||||
import ca.joeltherrien.randomforest.Row;
|
||||
import ca.joeltherrien.randomforest.Settings;
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.BooleanCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.FactorCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.covariates.settings.NumericCovariateSettings;
|
||||
import ca.joeltherrien.randomforest.loaders.ResponseLoader;
|
||||
import ca.joeltherrien.randomforest.utils.CSVUtils;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.fasterxml.jackson.databind.node.TextNode;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
public class TestLoadingCSV {
|
||||
|
||||
/*
|
||||
y,x1,x2,x3
|
||||
5,3.0,"mouse",true
|
||||
2,1.0,"dog",false
|
||||
9,1.5,"cat",true
|
||||
-3,NA,NA,NA
|
||||
*/
|
||||
|
||||
|
||||
public List<Row<Double>> loadData(String filename) throws IOException {
|
||||
final ObjectNode yVarSettings = new ObjectNode(JsonNodeFactory.instance);
|
||||
yVarSettings.set("type", new TextNode("Double"));
|
||||
yVarSettings.set("name", new TextNode("y"));
|
||||
|
||||
final Settings settings = Settings.builder()
|
||||
.trainingDataLocation(filename)
|
||||
.covariateSettings(
|
||||
Utils.easyList(new NumericCovariateSettings("x1"),
|
||||
new FactorCovariateSettings("x2", Utils.easyList("dog", "cat", "mouse")),
|
||||
new BooleanCovariateSettings("x3"))
|
||||
)
|
||||
.yVarSettings(yVarSettings)
|
||||
.build();
|
||||
|
||||
final List<Covariate> covariates = settings.getCovariates();
|
||||
|
||||
|
||||
final ResponseLoader loader = settings.getResponseLoader();
|
||||
|
||||
return CSVUtils.loadData(covariates, loader, settings.getTrainingDataLocation());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void verifyLoadingNormal(final List<Covariate> covariates) throws IOException {
|
||||
final List<Row<Double>> data = loadData("src/test/resources/testCSV.csv");
|
||||
|
||||
assertData(data, covariates);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void verifyLoadingGz(final List<Covariate> covariates) throws IOException {
|
||||
final List<Row<Double>> data = loadData("src/test/resources/testCSV.csv.gz");
|
||||
|
||||
assertData(data, covariates);
|
||||
}
|
||||
|
||||
|
||||
private void assertData(final List<Row<Double>> data, final List<Covariate> covariates){
|
||||
final Covariate x1 = covariates.get(0);
|
||||
final Covariate x2 = covariates.get(0);
|
||||
final Covariate x3 = covariates.get(0);
|
||||
|
||||
assertEquals(4, data.size());
|
||||
|
||||
Row<Double> row = data.get(0);
|
||||
assertEquals(5.0, (double)row.getResponse());
|
||||
assertEquals(3.0, row.getCovariateValue(x1).getValue());
|
||||
assertEquals("mouse", row.getCovariateValue(x2).getValue());
|
||||
assertEquals(true, row.getCovariateValue(x3).getValue());
|
||||
|
||||
row = data.get(1);
|
||||
assertEquals(2.0, (double)row.getResponse());
|
||||
assertEquals(1.0, row.getCovariateValue(x1).getValue());
|
||||
assertEquals("dog", row.getCovariateValue(x2).getValue());
|
||||
assertEquals(false, row.getCovariateValue(x3).getValue());
|
||||
|
||||
row = data.get(2);
|
||||
assertEquals(9.0, (double)row.getResponse());
|
||||
assertEquals(1.5, row.getCovariateValue(x1).getValue());
|
||||
assertEquals("cat", row.getCovariateValue(x2).getValue());
|
||||
assertEquals(true, row.getCovariateValue(x3).getValue());
|
||||
|
||||
row = data.get(3);
|
||||
assertEquals(-3.0, (double)row.getResponse());
|
||||
assertTrue(row.getCovariateValue(x1).isNA());
|
||||
assertTrue(row.getCovariateValue(x2).isNA());
|
||||
assertTrue(row.getCovariateValue(x3).isNA());
|
||||
}
|
||||
|
||||
}
|
||||
102
library/pom.xml
102
library/pom.xml
|
|
@ -1,102 +0,0 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<artifactId>largeRCRF-library</artifactId>
|
||||
|
||||
<!-- parent pom -->
|
||||
<parent>
|
||||
<groupId>ca.joeltherrien</groupId>
|
||||
<artifactId>largeRCRF</artifactId>
|
||||
<version>1.0-SNAPSHOT</version>
|
||||
</parent>
|
||||
|
||||
<properties>
|
||||
<java.version>1.8</java.version>
|
||||
<maven.compiler.target>1.8</maven.compiler.target>
|
||||
<maven.compiler.source>1.8</maven.compiler.source>
|
||||
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
|
||||
</properties>
|
||||
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<version>1.18.0</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.commons</groupId>
|
||||
<artifactId>commons-csv</artifactId>
|
||||
<version>1.5</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-api</artifactId>
|
||||
<version>5.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-engine</artifactId>
|
||||
<version>5.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mockito</groupId>
|
||||
<artifactId>mockito-core</artifactId>
|
||||
<version>2.20.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>shade</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-pmd-plugin</artifactId>
|
||||
<version>3.11.0</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase> <!-- bind to the packaging phase -->
|
||||
<goals>
|
||||
<goal>check</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
<configuration>
|
||||
<rulesets>
|
||||
<!-- Custom local file system rule set -->
|
||||
<ruleset>${project.basedir}/../pmd-rules.xml</ruleset>
|
||||
</rulesets>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
|
||||
|
||||
</project>
|
||||
|
|
@ -1,126 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.responses.competingrisk.combiner;
|
||||
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskFunctions;
|
||||
import ca.joeltherrien.randomforest.tree.ResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public class CompetingRiskFunctionCombiner implements ResponseCombiner<CompetingRiskFunctions, CompetingRiskFunctions> {
|
||||
|
||||
private final int[] events;
|
||||
private final double[] times; // We may restrict ourselves to specific times.
|
||||
|
||||
public int[] getEvents(){
|
||||
return events.clone();
|
||||
}
|
||||
|
||||
public double[] getTimes(){
|
||||
return times.clone();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompetingRiskFunctions combine(List<CompetingRiskFunctions> responses) {
|
||||
|
||||
final double[] timesToUse;
|
||||
if(times != null){
|
||||
timesToUse = times;
|
||||
}
|
||||
else{
|
||||
timesToUse = responses.stream()
|
||||
.map(functions -> functions.getSurvivalCurve())
|
||||
.flatMapToDouble(
|
||||
function -> Arrays.stream(function.getX())
|
||||
).sorted().distinct().toArray();
|
||||
}
|
||||
|
||||
final double n = responses.size();
|
||||
|
||||
final double[] survivalY = new double[timesToUse.length];
|
||||
final double[][] csCHFY = new double[events.length][timesToUse.length];
|
||||
final double[][] cifY = new double[events.length][timesToUse.length];
|
||||
|
||||
/*
|
||||
We're going to try to efficiently put our predictions together -
|
||||
Assumptions - for each event on a response, the hazard and CIF functions share the same x points
|
||||
|
||||
Plan - go through the time on each response and make use of that so that when we search for a time index
|
||||
to evaluate the function at, we don't need to re-search the earlier times.
|
||||
|
||||
*/
|
||||
|
||||
|
||||
for(final CompetingRiskFunctions currentFunctions : responses){
|
||||
final double[] survivalXPoints = currentFunctions.getSurvivalCurve().getX();
|
||||
final double[][] eventSpecificXPoints = new double[events.length][];
|
||||
|
||||
for(final int event : events){
|
||||
eventSpecificXPoints[event-1] = currentFunctions.getCumulativeIncidenceFunction(event)
|
||||
.getX();
|
||||
}
|
||||
|
||||
int previousSurvivalIndex = 0;
|
||||
final int[] previousEventSpecificIndex = new int[events.length]; // relying on 0 being default value
|
||||
|
||||
for(int i=0; i<timesToUse.length; i++){
|
||||
final double time = timesToUse[i];
|
||||
|
||||
// Survival curve
|
||||
final int survivalTimeIndex = Utils.binarySearchLessThan(previousSurvivalIndex, survivalXPoints.length, survivalXPoints, time);
|
||||
survivalY[i] = survivalY[i] + currentFunctions.getSurvivalCurve().evaluateByIndex(survivalTimeIndex) / n;
|
||||
previousSurvivalIndex = Math.max(survivalTimeIndex, 0); // if our current time is less than the smallest time in xPoints then binarySearchLessThan returned a -1.
|
||||
// -1's not an issue for evaluateByIndex, but it is an issue for the next time binarySearchLessThan is called.
|
||||
|
||||
// CHFs and CIFs
|
||||
for(final int event : events){
|
||||
final double[] xPoints = eventSpecificXPoints[event-1];
|
||||
final int eventTimeIndex = Utils.binarySearchLessThan(previousEventSpecificIndex[event-1], xPoints.length,
|
||||
xPoints, time);
|
||||
csCHFY[event-1][i] = csCHFY[event-1][i] + currentFunctions.getCauseSpecificHazardFunction(event)
|
||||
.evaluateByIndex(eventTimeIndex) / n;
|
||||
cifY[event-1][i] = cifY[event-1][i] + currentFunctions.getCumulativeIncidenceFunction(event)
|
||||
.evaluateByIndex(eventTimeIndex) / n;
|
||||
|
||||
previousEventSpecificIndex[event-1] = Math.max(eventTimeIndex, 0);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
final RightContinuousStepFunction survivalFunction = new RightContinuousStepFunction(timesToUse, survivalY, 1.0);
|
||||
final List<RightContinuousStepFunction> causeSpecificCumulativeHazardFunctionList = new ArrayList<>(events.length);
|
||||
final List<RightContinuousStepFunction> cumulativeIncidenceFunctionList = new ArrayList<>(events.length);
|
||||
|
||||
for(final int event : events){
|
||||
causeSpecificCumulativeHazardFunctionList.add(event-1, new RightContinuousStepFunction(timesToUse, csCHFY[event-1], 0));
|
||||
cumulativeIncidenceFunctionList.add(event-1, new RightContinuousStepFunction(timesToUse, cifY[event-1], 0));
|
||||
}
|
||||
|
||||
return CompetingRiskFunctions.builder()
|
||||
.causeSpecificHazards(causeSpecificCumulativeHazardFunctionList)
|
||||
.cumulativeIncidenceCurves(cumulativeIncidenceFunctionList)
|
||||
.survivalCurve(survivalFunction)
|
||||
.build();
|
||||
}
|
||||
}
|
||||
|
|
@ -1,113 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
/**
|
||||
* Represents a function represented by discrete points. However, the function may be right-continuous or left-continuous
|
||||
* at a given point, with no consistency. This function tracks that.
|
||||
*/
|
||||
public final class DiscontinuousStepFunction extends StepFunction {
|
||||
|
||||
private final double[] y;
|
||||
private final boolean[] isLeftContinuous;
|
||||
|
||||
/**
|
||||
* Represents the value that should be returned by evaluate if there are points prior to the time the function is being evaluated at.
|
||||
*
|
||||
* Map be null.
|
||||
*/
|
||||
private final double defaultY;
|
||||
|
||||
public DiscontinuousStepFunction(double[] x, double[] y, boolean[] isLeftContinuous, double defaultY) {
|
||||
super(x);
|
||||
this.y = y;
|
||||
this.isLeftContinuous = isLeftContinuous;
|
||||
this.defaultY = defaultY;
|
||||
}
|
||||
|
||||
@Override
|
||||
public double evaluate(double time){
|
||||
int index = Utils.binarySearchLessThan(0, x.length, x, time);
|
||||
if(index < 0){
|
||||
return defaultY;
|
||||
}
|
||||
else{
|
||||
if(x[index] == time){
|
||||
return evaluateByIndex(index);
|
||||
}
|
||||
else{
|
||||
return y[index];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public double evaluatePrevious(double time){
|
||||
int index = Utils.binarySearchLessThan(0, x.length, x, time) - 1;
|
||||
if(index < 0){
|
||||
return defaultY;
|
||||
}
|
||||
else{
|
||||
if(x[index] == time){
|
||||
return evaluateByIndex(index);
|
||||
}
|
||||
else{
|
||||
return y[index];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public double evaluateByIndex(int i) {
|
||||
if(isLeftContinuous[i]){
|
||||
i -= 1;
|
||||
}
|
||||
|
||||
if(i < 0){
|
||||
return defaultY;
|
||||
}
|
||||
|
||||
return y[i];
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString(){
|
||||
final StringBuilder builder = new StringBuilder();
|
||||
builder.append("Default point: ");
|
||||
builder.append(defaultY);
|
||||
builder.append("\n");
|
||||
|
||||
for(int i=0; i<x.length; i++){
|
||||
builder.append("x:");
|
||||
builder.append(x[i]);
|
||||
builder.append('\t');
|
||||
|
||||
if(isLeftContinuous[i]){
|
||||
builder.append("*y:");
|
||||
}
|
||||
else{
|
||||
builder.append("y*:");
|
||||
}
|
||||
builder.append(y[i]);
|
||||
builder.append("\n");
|
||||
}
|
||||
|
||||
return builder.toString();
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -1,129 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.ListIterator;
|
||||
|
||||
/**
|
||||
* Represents a function represented by discrete points. We assume that the function is a stepwise left-continuous
|
||||
* function, constant at the value of the previous encountered point.
|
||||
*
|
||||
*/
|
||||
public final class LeftContinuousStepFunction extends StepFunction {
|
||||
|
||||
private final double[] y;
|
||||
|
||||
/**
|
||||
* Represents the value that should be returned by evaluate if there are points prior to the time the function is being evaluated at.
|
||||
*
|
||||
* Map be null.
|
||||
*/
|
||||
private final double defaultY;
|
||||
|
||||
public LeftContinuousStepFunction(double[] x, double[] y, double defaultY) {
|
||||
super(x);
|
||||
this.y = y;
|
||||
this.defaultY = defaultY;
|
||||
}
|
||||
|
||||
/**
|
||||
* This isn't a formal constructor because of limitations with abstract classes.
|
||||
*
|
||||
* @param pointList
|
||||
* @param defaultY
|
||||
* @return
|
||||
*/
|
||||
public static LeftContinuousStepFunction constructFromPoints(final List<Point> pointList, final double defaultY){
|
||||
|
||||
final double[] x = new double[pointList.size()];
|
||||
final double[] y = new double[pointList.size()];
|
||||
|
||||
final ListIterator<Point> pointIterator = pointList.listIterator();
|
||||
while(pointIterator.hasNext()){
|
||||
final int index = pointIterator.nextIndex();
|
||||
final Point currentPoint = pointIterator.next();
|
||||
|
||||
x[index] = currentPoint.getTime();
|
||||
y[index] = currentPoint.getY();
|
||||
}
|
||||
|
||||
return new LeftContinuousStepFunction(x, y, defaultY);
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public double evaluate(double time){
|
||||
int index = Utils.binarySearchLessThan(0, x.length, x, time);
|
||||
if(index < 0){
|
||||
return defaultY;
|
||||
}
|
||||
else{
|
||||
if(x[index] == time){
|
||||
return evaluateByIndex(index-1);
|
||||
}
|
||||
else{
|
||||
return y[index];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public double evaluatePrevious(double time){
|
||||
int index = Utils.binarySearchLessThan(0, x.length, x, time) - 1;
|
||||
if(index < 0){
|
||||
return defaultY;
|
||||
}
|
||||
else{
|
||||
if(x[index] == time){
|
||||
return evaluateByIndex(index-1);
|
||||
}
|
||||
else{
|
||||
return y[index];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public double evaluateByIndex(int i) {
|
||||
if(i < 0){
|
||||
return defaultY;
|
||||
}
|
||||
|
||||
return y[i];
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public String toString(){
|
||||
final StringBuilder builder = new StringBuilder();
|
||||
builder.append("Default point: ");
|
||||
builder.append(defaultY);
|
||||
builder.append("\n");
|
||||
|
||||
for(int i=0; i<x.length; i++){
|
||||
builder.append("x:");
|
||||
builder.append(x[i]);
|
||||
builder.append("\ty:");
|
||||
builder.append(y[i]);
|
||||
builder.append("\n");
|
||||
}
|
||||
|
||||
return builder.toString();
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
/**
|
||||
* Represents a step function represented by discrete points. However, there may be individual time values that has
|
||||
* a y value that doesn't belong to a particular 'step'.
|
||||
*/
|
||||
public final class VeryDiscontinuousStepFunction implements MathFunction {
|
||||
|
||||
private final double[] x;
|
||||
private final double[] yAt;
|
||||
private final double[] yRight;
|
||||
|
||||
/**
|
||||
* Represents the value that should be returned by evaluate if there are points prior to the time the function is being evaluated at.
|
||||
*
|
||||
* Map be null.
|
||||
*/
|
||||
private final double defaultY;
|
||||
|
||||
public VeryDiscontinuousStepFunction(double[] x, double[] yAt, double[] yRight, double defaultY) {
|
||||
this.x = x;
|
||||
this.yAt = yAt;
|
||||
this.yRight = yRight;
|
||||
this.defaultY = defaultY;
|
||||
}
|
||||
|
||||
@Override
|
||||
public double evaluate(double time){
|
||||
int index = Utils.binarySearchLessThan(0, x.length, x, time);
|
||||
if(index < 0){
|
||||
return defaultY;
|
||||
}
|
||||
else{
|
||||
if(x[index] == time){
|
||||
return yAt[index];
|
||||
}
|
||||
else{ // time > x[index]
|
||||
return yRight[index];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public String toString(){
|
||||
final StringBuilder builder = new StringBuilder();
|
||||
builder.append("Default point: ");
|
||||
builder.append(defaultY);
|
||||
builder.append("\n");
|
||||
|
||||
for(int i=0; i<x.length; i++){
|
||||
builder.append("x:");
|
||||
builder.append(x[i]);
|
||||
builder.append("\tyAt:");
|
||||
builder.append(yAt[i]);
|
||||
builder.append("\tyRight:");
|
||||
builder.append(yRight[i]);
|
||||
builder.append("\n");
|
||||
}
|
||||
|
||||
return builder.toString();
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -1,86 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.covariates;
|
||||
|
||||
|
||||
import ca.joeltherrien.randomforest.covariates.factor.FactorCovariate;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.function.Executable;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Random;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
public class FactorCovariateTest {
|
||||
|
||||
@Test
|
||||
void verifyEqualLevels() {
|
||||
final FactorCovariate petCovariate = createTestCovariate();
|
||||
|
||||
final FactorCovariate.FactorValue dog1 = petCovariate.createValue("DOG");
|
||||
final FactorCovariate.FactorValue dog2 = petCovariate.createValue("DO" + "G");
|
||||
|
||||
assertSame(dog1, dog2);
|
||||
|
||||
final FactorCovariate.FactorValue cat1 = petCovariate.createValue("CAT");
|
||||
final FactorCovariate.FactorValue cat2 = petCovariate.createValue("CA" + "T");
|
||||
|
||||
assertSame(cat1, cat2);
|
||||
|
||||
final FactorCovariate.FactorValue mouse1 = petCovariate.createValue("MOUSE");
|
||||
final FactorCovariate.FactorValue mouse2 = petCovariate.createValue("MOUS" + "E");
|
||||
|
||||
assertSame(mouse1, mouse2);
|
||||
|
||||
|
||||
}
|
||||
|
||||
@Test
|
||||
void verifyBadLevelException(){
|
||||
final FactorCovariate petCovariate = createTestCovariate();
|
||||
final Executable badCode = () -> petCovariate.createValue("vulcan");
|
||||
|
||||
assertThrows(IllegalArgumentException.class, badCode, "vulcan is not a level in FactorCovariate pet");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testAllSubsets(){
|
||||
final FactorCovariate petCovariate = createTestCovariate();
|
||||
|
||||
final List<SplitRule<String>> splitRules = new ArrayList<>();
|
||||
|
||||
petCovariate.generateSplitRuleUpdater(null, 100, new Random())
|
||||
.forEachRemaining(split -> splitRules.add(split.getSplitRule()));
|
||||
|
||||
assertEquals(splitRules.size(), 3);
|
||||
|
||||
// TODO verify the contents of the split rules
|
||||
|
||||
}
|
||||
|
||||
|
||||
private FactorCovariate createTestCovariate(){
|
||||
final List<String> levels = Utils.easyList("DOG", "CAT", "MOUSE");
|
||||
|
||||
return new FactorCovariate("pet", 0, levels);
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -1,77 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.nas;
|
||||
|
||||
import ca.joeltherrien.randomforest.Row;
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.covariates.numeric.NumericCovariate;
|
||||
import ca.joeltherrien.randomforest.responses.regression.MeanResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.responses.regression.WeightedVarianceSplitFinder;
|
||||
import ca.joeltherrien.randomforest.tree.TreeTrainer;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Random;
|
||||
|
||||
public class TestNAs {
|
||||
|
||||
private List<Row<Double>> generateData(List<Covariate> covariates){
|
||||
final List<Row<Double>> dataList = new ArrayList<>();
|
||||
|
||||
|
||||
// We must include an NA for one of the values
|
||||
dataList.add(Row.createSimple(Utils.easyMap("x", "NA"), covariates, 1, 5.0));
|
||||
dataList.add(Row.createSimple(Utils.easyMap("x", "1"), covariates, 1, 6.0));
|
||||
dataList.add(Row.createSimple(Utils.easyMap("x", "2"), covariates, 1, 5.5));
|
||||
dataList.add(Row.createSimple(Utils.easyMap("x", "7"), covariates, 1, 0.0));
|
||||
dataList.add(Row.createSimple(Utils.easyMap("x", "8"), covariates, 1, 1.0));
|
||||
dataList.add(Row.createSimple(Utils.easyMap("x", "8.4"), covariates, 1, 1.0));
|
||||
|
||||
|
||||
return dataList;
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testException(){
|
||||
// There was a bug with NAs where when we tried to randomly assign NAs during a split to the best split produced by NumericSplitRuleUpdater,
|
||||
// but NumericSplitRuleUpdater had unmodifiable lists when creating the split.
|
||||
// This bug verifies that this no longer causes a crash
|
||||
|
||||
final List<Covariate> covariates = Collections.singletonList(new NumericCovariate("x", 0));
|
||||
final List<Row<Double>> dataset = generateData(covariates);
|
||||
|
||||
final TreeTrainer<Double, Double> treeTrainer = TreeTrainer.<Double, Double>builder()
|
||||
.checkNodePurity(false)
|
||||
.covariates(covariates)
|
||||
.numberOfSplits(0)
|
||||
.nodeSize(1)
|
||||
.maxNodeDepth(1000)
|
||||
.splitFinder(new WeightedVarianceSplitFinder())
|
||||
.responseCombiner(new MeanResponseCombiner())
|
||||
.build();
|
||||
|
||||
treeTrainer.growTree(dataset, new Random(123));
|
||||
|
||||
// As long as no exception occurs, we passed
|
||||
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -1,79 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
import ca.joeltherrien.randomforest.TestUtils;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
public class RightContinuousStepFunctionIntegrationTest {
|
||||
|
||||
private RightContinuousStepFunction createTestFunction(){
|
||||
|
||||
final double defaultY = 1;
|
||||
final double[] x = new double[]{1.0, 2.0, 3.0, 4.0, 5.0};
|
||||
final double[] y = new double[]{2.0, 3.0, -0.5, -1.0, 3.0};
|
||||
|
||||
return new RightContinuousStepFunction(x, y, defaultY);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testIntegration(){
|
||||
final RightContinuousStepFunction function = createTestFunction();
|
||||
|
||||
// Test whether it can handle both from and to being before all the x-values
|
||||
TestUtils.closeEnough(0.9,
|
||||
function.integrate(0, 0.9)
|
||||
, 0.00000001);
|
||||
|
||||
// Test whether it can handle both from and to being after all the x-values
|
||||
TestUtils.closeEnough(3.0,
|
||||
function.integrate(6, 7)
|
||||
, 0.00000001);
|
||||
|
||||
TestUtils.closeEnough(6.0,
|
||||
function.integrate(0, 3)
|
||||
, 0.00000001);
|
||||
|
||||
TestUtils.closeEnough(4.5,
|
||||
function.integrate(0, 2.5)
|
||||
, 0.00000001);
|
||||
|
||||
TestUtils.closeEnough(7.5,
|
||||
function.integrate(0, 6)
|
||||
, 0.00000001);
|
||||
|
||||
TestUtils.closeEnough(-0.5,
|
||||
function.integrate(3, 4)
|
||||
, 0.00000001);
|
||||
|
||||
TestUtils.closeEnough(-0.5*0.8,
|
||||
function.integrate(3.1, 3.9)
|
||||
, 0.00000001);
|
||||
|
||||
TestUtils.closeEnough(1.5,
|
||||
function.integrate(3, 6)
|
||||
, 0.00000001);
|
||||
|
||||
TestUtils.closeEnough(3.0,
|
||||
function.integrate(2.5, 6)
|
||||
, 0.00000001);
|
||||
|
||||
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
105
pom.xml
105
pom.xml
|
|
@ -1,21 +1,98 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
|
||||
http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<!-- parent pom -->
|
||||
<groupId>ca.joeltherrien</groupId>
|
||||
<artifactId>largeRCRF</artifactId>
|
||||
<groupId>ca.joeltherrien.ca</groupId>
|
||||
<artifactId>largeRCRF-library</artifactId>
|
||||
<version>1.0-SNAPSHOT</version>
|
||||
<packaging>pom</packaging>
|
||||
|
||||
<!-- sub modules -->
|
||||
<modules>
|
||||
<module>library</module>
|
||||
<module>executable</module>
|
||||
</modules>
|
||||
<properties>
|
||||
<java.version>1.8</java.version>
|
||||
<maven.compiler.target>1.8</maven.compiler.target>
|
||||
<maven.compiler.source>1.8</maven.compiler.source>
|
||||
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
|
||||
</properties>
|
||||
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<version>1.18.0</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.commons</groupId>
|
||||
<artifactId>commons-csv</artifactId>
|
||||
<version>1.5</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-api</artifactId>
|
||||
<version>5.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-engine</artifactId>
|
||||
<version>5.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mockito</groupId>
|
||||
<artifactId>mockito-core</artifactId>
|
||||
<version>2.20.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
<version>3.2.1</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>shade</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-pmd-plugin</artifactId>
|
||||
<version>3.11.0</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase> <!-- bind to the packaging phase -->
|
||||
<goals>
|
||||
<goal>check</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
<configuration>
|
||||
<rulesets>
|
||||
<!-- Custom local file system rule set -->
|
||||
<ruleset>pmd-rules.xml</ruleset>
|
||||
</rulesets>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
|
||||
|
||||
</project>
|
||||
|
||||
|
|
|
|||
|
|
@ -21,12 +21,13 @@ import lombok.Getter;
|
|||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.*;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public class CovariateRow implements Serializable {
|
||||
public class CovariateRow implements Serializable, Cloneable {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final Covariate.Value[] valueArray;
|
||||
|
||||
|
|
@ -46,6 +47,14 @@ public class CovariateRow implements Serializable {
|
|||
return "CovariateRow " + this.id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CovariateRow clone() {
|
||||
// shallow clone, which is fine. I want a new array, but the values don't need to be copied
|
||||
final Covariate.Value[] copyValueArray = this.valueArray.clone();
|
||||
|
||||
return new CovariateRow(copyValueArray, this.id);
|
||||
}
|
||||
|
||||
public static CovariateRow createSimple(Map<String, String> simpleMap, List<Covariate> covariateList, int id){
|
||||
final Covariate.Value[] valueArray = new Covariate.Value[covariateList.size()];
|
||||
final Map<String, Covariate> covariateMap = new HashMap<>();
|
||||
|
|
@ -64,4 +73,27 @@ public class CovariateRow implements Serializable {
|
|||
return new CovariateRow(valueArray, id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Used for variable importance; takes a List of CovariateRows and permute one of the Covariates.
|
||||
*
|
||||
* @param covariateRows The List of CovariateRows to scramble. Note that the originals won't be modified.
|
||||
* @param covariateToScramble The Covariate to scramble on.
|
||||
* @param random The source of randomness to use. If not present, one will be created.
|
||||
* @return A List of CovariateRows where the specified covariate was scrambled. These are different objects from the ones provided.
|
||||
*/
|
||||
public static List<CovariateRow> scrambleCovariateValues(List<? extends CovariateRow> covariateRows, Covariate covariateToScramble, Optional<Random> random){
|
||||
final List<CovariateRow> permutedCovariateRowList = new ArrayList<>(covariateRows);
|
||||
Collections.shuffle(permutedCovariateRowList, random.orElse(new Random())); // without replacement
|
||||
|
||||
final List<CovariateRow> clonedRowList = covariateRows.stream().map(CovariateRow::clone).collect(Collectors.toList());
|
||||
|
||||
final int covariateToScrambleIndex = covariateToScramble.getIndex();
|
||||
for(int i=0; i < covariateRows.size(); i++){
|
||||
clonedRowList.get(i).valueArray[covariateToScrambleIndex] = permutedCovariateRowList.get(i).valueArray[covariateToScrambleIndex];
|
||||
}
|
||||
|
||||
return clonedRowList;
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -32,9 +32,6 @@ public class Row<Y> extends CovariateRow {
|
|||
this.response = response;
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
public Y getResponse() {
|
||||
return this.response;
|
||||
}
|
||||
|
|
@ -49,6 +49,8 @@ public interface Covariate<V> extends Serializable, Comparable<Covariate> {
|
|||
return getIndex() - other.getIndex();
|
||||
}
|
||||
|
||||
boolean haveNASplitPenalty();
|
||||
|
||||
interface Value<V> extends Serializable{
|
||||
|
||||
Covariate<V> getParent();
|
||||
|
|
@ -25,9 +25,12 @@ import lombok.Getter;
|
|||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Random;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
public final class BooleanCovariate implements Covariate<Boolean> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
@Getter
|
||||
private final String name;
|
||||
|
||||
|
|
@ -38,14 +41,26 @@ public final class BooleanCovariate implements Covariate<Boolean> {
|
|||
|
||||
private final BooleanSplitRule splitRule; // there's only one possible rule for BooleanCovariates.
|
||||
|
||||
public BooleanCovariate(String name, int index){
|
||||
private final boolean haveNASplitPenalty;
|
||||
@Override
|
||||
public boolean haveNASplitPenalty(){
|
||||
// penalty would add worthless computational time if there are no NAs
|
||||
return hasNAs && haveNASplitPenalty;
|
||||
}
|
||||
|
||||
public BooleanCovariate(String name, int index, boolean haveNASplitPenalty){
|
||||
this.name = name;
|
||||
this.index = index;
|
||||
splitRule = new BooleanSplitRule(this);
|
||||
this.splitRule = new BooleanSplitRule(this);
|
||||
this.haveNASplitPenalty = haveNASplitPenalty;
|
||||
}
|
||||
|
||||
@Override
|
||||
public <Y> Iterator<Split<Y, Boolean>> generateSplitRuleUpdater(List<Row<Y>> data, int number, Random random) {
|
||||
if(hasNAs){
|
||||
data = data.stream().filter(row -> !row.getValueByIndex(index).isNA()).collect(Collectors.toList());
|
||||
}
|
||||
|
||||
return new SingletonIterator<>(this.splitRule.applyRule(data));
|
||||
}
|
||||
|
||||
|
|
@ -84,6 +99,8 @@ public final class BooleanCovariate implements Covariate<Boolean> {
|
|||
|
||||
public class BooleanValue implements Value<Boolean>{
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final Boolean value;
|
||||
|
||||
private BooleanValue(final Boolean value){
|
||||
|
|
@ -21,6 +21,8 @@ import ca.joeltherrien.randomforest.covariates.SplitRule;
|
|||
|
||||
public class BooleanSplitRule implements SplitRule<Boolean> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int parentCovariateIndex;
|
||||
|
||||
public BooleanSplitRule(BooleanCovariate parent){
|
||||
|
|
@ -23,9 +23,12 @@ import lombok.EqualsAndHashCode;
|
|||
import lombok.Getter;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
public final class FactorCovariate implements Covariate<String> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
@Getter
|
||||
private final String name;
|
||||
|
||||
|
|
@ -38,8 +41,15 @@ public final class FactorCovariate implements Covariate<String> {
|
|||
|
||||
private boolean hasNAs;
|
||||
|
||||
private final boolean haveNASplitPenalty;
|
||||
@Override
|
||||
public boolean haveNASplitPenalty(){
|
||||
// penalty would add worthless computational time if there are no NAs
|
||||
return hasNAs && haveNASplitPenalty;
|
||||
}
|
||||
|
||||
public FactorCovariate(final String name, final int index, List<String> levels){
|
||||
|
||||
public FactorCovariate(final String name, final int index, List<String> levels, final boolean haveNASplitPenalty){
|
||||
this.name = name;
|
||||
this.index = index;
|
||||
this.factorLevels = new HashMap<>();
|
||||
|
|
@ -61,12 +71,22 @@ public final class FactorCovariate implements Covariate<String> {
|
|||
this.numberOfPossiblePairings = numberOfPossiblePairingsTemp-1;
|
||||
|
||||
this.naValue = new FactorValue(null);
|
||||
|
||||
this.haveNASplitPenalty = haveNASplitPenalty;
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Override
|
||||
public <Y> Iterator<Split<Y, String>> generateSplitRuleUpdater(List<Row<Y>> data, int number, Random random) {
|
||||
if(hasNAs()){
|
||||
data = data.stream().filter(row -> !row.getCovariateValue(this).isNA()).collect(Collectors.toList());
|
||||
}
|
||||
|
||||
if(number == 0){ // nsplit = 0 => try every possibility, although we limit it to the number of observations.
|
||||
number = data.size();
|
||||
}
|
||||
|
||||
final Set<Split<Y, String>> splits = new HashSet<>();
|
||||
|
||||
// This is to ensure we don't get stuck in an infinite loop for small factors
|
||||
|
|
@ -122,6 +142,8 @@ public final class FactorCovariate implements Covariate<String> {
|
|||
@EqualsAndHashCode
|
||||
public final class FactorValue implements Covariate.Value<String>{
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final String value;
|
||||
|
||||
private FactorValue(final String value){
|
||||
|
|
@ -25,6 +25,8 @@ import java.util.Set;
|
|||
@EqualsAndHashCode
|
||||
public final class FactorSplitRule implements SplitRule<String> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int parentCovariateIndex;
|
||||
private final Set<String> leftSideValues;
|
||||
|
||||
|
|
@ -37,6 +37,8 @@ import java.util.stream.Stream;
|
|||
@ToString
|
||||
public final class NumericCovariate implements Covariate<Double> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
@Getter
|
||||
private final String name;
|
||||
|
||||
|
|
@ -45,6 +47,13 @@ public final class NumericCovariate implements Covariate<Double> {
|
|||
|
||||
private boolean hasNAs = false;
|
||||
|
||||
private final boolean haveNASplitPenalty;
|
||||
@Override
|
||||
public boolean haveNASplitPenalty(){
|
||||
// penalty would add worthless computational time if there are no NAs
|
||||
return hasNAs && haveNASplitPenalty;
|
||||
}
|
||||
|
||||
@Override
|
||||
public <Y> NumericSplitRuleUpdater<Y> generateSplitRuleUpdater(List<Row<Y>> data, int number, Random random) {
|
||||
Stream<Row<Y>> stream = data.stream();
|
||||
|
|
@ -122,6 +131,8 @@ public final class NumericCovariate implements Covariate<Double> {
|
|||
@EqualsAndHashCode
|
||||
public class NumericValue implements Covariate.Value<Double>{
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final Double value; // may be null
|
||||
|
||||
private NumericValue(final Double value){
|
||||
|
|
@ -23,10 +23,12 @@ import lombok.EqualsAndHashCode;
|
|||
@EqualsAndHashCode
|
||||
public class NumericSplitRule implements SplitRule<Double> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int parentCovariateIndex;
|
||||
private final double threshold;
|
||||
|
||||
NumericSplitRule(NumericCovariate parent, final double threshold){
|
||||
public NumericSplitRule(NumericCovariate parent, final double threshold){
|
||||
this.parentCovariateIndex = parent.getIndex();
|
||||
this.threshold = threshold;
|
||||
}
|
||||
|
|
@ -26,6 +26,8 @@ import java.util.List;
|
|||
@Builder
|
||||
public class CompetingRiskFunctions implements Serializable {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final List<RightContinuousStepFunction> causeSpecificHazards;
|
||||
private final List<RightContinuousStepFunction> cumulativeIncidenceCurves;
|
||||
|
||||
|
|
@ -23,6 +23,8 @@ import java.io.Serializable;
|
|||
@Data
|
||||
public class CompetingRiskResponse implements Serializable {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int delta;
|
||||
private final double u;
|
||||
|
||||
|
|
@ -26,6 +26,9 @@ import lombok.EqualsAndHashCode;
|
|||
@EqualsAndHashCode(callSuper = true)
|
||||
@Data
|
||||
public final class CompetingRiskResponseWithCensorTime extends CompetingRiskResponse {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final double c;
|
||||
|
||||
public CompetingRiskResponseWithCensorTime(int delta, double u, double c) {
|
||||
|
|
@ -16,9 +16,11 @@
|
|||
|
||||
package ca.joeltherrien.randomforest.responses.competingrisk;
|
||||
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.StepFunction;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.stream.IntStream;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
public class CompetingRiskUtils {
|
||||
|
|
@ -116,6 +118,44 @@ public class CompetingRiskUtils {
|
|||
|
||||
}
|
||||
|
||||
/**
|
||||
* Calculate the Integrated Brier Score error on a list of responses and predictions.
|
||||
*
|
||||
* @param responses A List of responses
|
||||
* @param predictions The corresponding List of predictions.
|
||||
* @param censoringDistribution The censoring distribution.
|
||||
* @param eventOfFocus The event we are calculating the error for.
|
||||
* @param integrationUpperBound The upper bound to integrate to.
|
||||
* @param isParallel Whether we should use parallel streams or not (provided because of bugs on a particular system).
|
||||
* @return
|
||||
*/
|
||||
public static double[] calculateIBSError(final List<CompetingRiskResponse> responses,
|
||||
List<CompetingRiskFunctions> predictions,
|
||||
Optional<RightContinuousStepFunction> censoringDistribution,
|
||||
int eventOfFocus,
|
||||
double integrationUpperBound,
|
||||
boolean isParallel){
|
||||
|
||||
if(responses.size() != predictions.size()){
|
||||
throw new IllegalArgumentException("Length of responses and predictions must be equal.");
|
||||
}
|
||||
|
||||
final IBSCalculator calculator = new IBSCalculator(censoringDistribution);
|
||||
|
||||
IntStream stream = IntStream.range(0, responses.size());
|
||||
|
||||
if(isParallel){
|
||||
stream = stream.parallel();
|
||||
}
|
||||
|
||||
return stream.mapToDouble(i -> {
|
||||
CompetingRiskResponse response = responses.get(i);
|
||||
RightContinuousStepFunction cif = predictions.get(i).getCumulativeIncidenceFunction(eventOfFocus);
|
||||
|
||||
return calculator.calculateError(response, cif, eventOfFocus, integrationUpperBound);
|
||||
}).toArray();
|
||||
}
|
||||
|
||||
|
||||
public static CompetingRiskSetsImpl calculateSetsEfficiently(final List<CompetingRiskResponse> initialLeftHand,
|
||||
final List<CompetingRiskResponse> initialRightHand,
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
package ca.joeltherrien.randomforest.responses.competingrisk;
|
||||
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
|
||||
import java.util.Optional;
|
||||
|
||||
/**
|
||||
* Used to calculate the Integrated Brier Score. See Section 4.2 of "Random survival forests for competing risks" by Ishwaran.
|
||||
*
|
||||
*/
|
||||
public class IBSCalculator {
|
||||
|
||||
private final Optional<RightContinuousStepFunction> censoringDistribution;
|
||||
|
||||
public IBSCalculator(RightContinuousStepFunction censoringDistribution){
|
||||
this.censoringDistribution = Optional.of(censoringDistribution);
|
||||
}
|
||||
|
||||
public IBSCalculator(){
|
||||
this.censoringDistribution = Optional.empty();
|
||||
}
|
||||
|
||||
public IBSCalculator(Optional<RightContinuousStepFunction> censoringDistribution){
|
||||
this.censoringDistribution = censoringDistribution;
|
||||
}
|
||||
|
||||
public double calculateError(CompetingRiskResponse response, RightContinuousStepFunction cif, int eventOfInterest, double integrationUpperBound){
|
||||
|
||||
// return integral of weights*(I(response.getU() <= times & response.getDelta() == eventOfInterest) - cif(times))^2
|
||||
// Note that if we don't have weights, just treat them all as one (i.e. don't bother multiplying)
|
||||
|
||||
RightContinuousStepFunction functionToIntegrate = cif;
|
||||
|
||||
if(response.getDelta() == eventOfInterest){
|
||||
final RightContinuousStepFunction observedFunction = new RightContinuousStepFunction(new double[]{response.getU()}, new double[]{1.0}, 0.0);
|
||||
functionToIntegrate = RightContinuousStepFunction.biOperation(observedFunction, functionToIntegrate, (a, b) -> (a - b) * (a - b));
|
||||
} else{
|
||||
functionToIntegrate = functionToIntegrate.unaryOperation(a -> a*a);
|
||||
}
|
||||
|
||||
if(censoringDistribution.isPresent()){
|
||||
final RightContinuousStepFunction weights = calculateWeights(response, censoringDistribution.get());
|
||||
functionToIntegrate = RightContinuousStepFunction.biOperation(weights, functionToIntegrate, (a, b) -> a*b);
|
||||
|
||||
// the censoring weights go to 0 after the response is censored, so we can speed up results by only integrating
|
||||
// prior to the censor times
|
||||
if(response.isCensored()){
|
||||
integrationUpperBound = Math.min(integrationUpperBound, response.getU());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
return functionToIntegrate.integrate(0.0, integrationUpperBound);
|
||||
}
|
||||
|
||||
private RightContinuousStepFunction calculateWeights(CompetingRiskResponse response, RightContinuousStepFunction censoringDistribution){
|
||||
final double recordedTime = response.getU();
|
||||
|
||||
// Function(t) = firstPart(t) + secondPart(t)/thirdPart(t) where:
|
||||
// firstPart(t) = I(recordedTime <= t & !response.isCensored()) / censoringDistribution.evaluate(recordedTime);
|
||||
// secondPart(t) = I(recordedTime > t) = 1 - I(recordedTime <= t)
|
||||
// thirdPart(t) = censoringDistribution.evaluate(t)
|
||||
|
||||
final RightContinuousStepFunction secondPart = new RightContinuousStepFunction(new double[]{recordedTime}, new double[]{0.0}, 1.0);
|
||||
RightContinuousStepFunction result = RightContinuousStepFunction.biOperation(secondPart, censoringDistribution,
|
||||
(second, third) -> second / third);
|
||||
|
||||
if(!response.isCensored()){
|
||||
final RightContinuousStepFunction firstPart = new RightContinuousStepFunction(
|
||||
new double[]{recordedTime},
|
||||
new double[]{1.0 / censoringDistribution.evaluate(recordedTime)},
|
||||
0.0);
|
||||
|
||||
result = RightContinuousStepFunction.biOperation(firstPart, result, Double::sum);
|
||||
}
|
||||
|
||||
return result;
|
||||
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.responses.competingrisk.combiner;
|
||||
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskFunctions;
|
||||
import ca.joeltherrien.randomforest.tree.ForestResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.tree.IntermediateCombinedResponse;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public class CompetingRiskFunctionCombiner implements ForestResponseCombiner<CompetingRiskFunctions, CompetingRiskFunctions> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int[] events;
|
||||
private final double[] times; // We may restrict ourselves to specific times.
|
||||
|
||||
public int[] getEvents(){
|
||||
return events.clone();
|
||||
}
|
||||
|
||||
public double[] getTimes(){
|
||||
return times.clone();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompetingRiskFunctions combine(List<CompetingRiskFunctions> responses) {
|
||||
|
||||
final double[] timesToUse;
|
||||
if(times != null){
|
||||
timesToUse = times;
|
||||
}
|
||||
else{
|
||||
timesToUse = responses.stream()
|
||||
.map(functions -> functions.getSurvivalCurve())
|
||||
.flatMapToDouble(
|
||||
function -> Arrays.stream(function.getX())
|
||||
).sorted().distinct().toArray();
|
||||
}
|
||||
|
||||
final IntermediateCompetingRisksFunctionsTimesKnown intermediateResult = new IntermediateCompetingRisksFunctionsTimesKnown(responses.size(), this.events, timesToUse);
|
||||
|
||||
for(CompetingRiskFunctions input : responses){
|
||||
intermediateResult.processNewInput(input);
|
||||
}
|
||||
|
||||
return intermediateResult.transformToOutput();
|
||||
}
|
||||
|
||||
@Override
|
||||
public IntermediateCombinedResponse<CompetingRiskFunctions, CompetingRiskFunctions> startIntermediateCombinedResponse(int countInputs) {
|
||||
if(this.times != null){
|
||||
return new IntermediateCompetingRisksFunctionsTimesKnown(countInputs, this.events, this.times);
|
||||
}
|
||||
|
||||
// TODO - implement
|
||||
throw new RuntimeException("startIntermediateCombinedResponse when times is unknown is not yet implemented");
|
||||
}
|
||||
}
|
||||
|
|
@ -35,6 +35,8 @@ import java.util.List;
|
|||
*/
|
||||
public class CompetingRiskResponseCombiner implements ResponseCombiner<CompetingRiskResponse, CompetingRiskFunctions> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int[] events;
|
||||
|
||||
public CompetingRiskResponseCombiner(final int[] events){
|
||||
|
|
@ -0,0 +1,118 @@
|
|||
package ca.joeltherrien.randomforest.responses.competingrisk.combiner;
|
||||
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskFunctions;
|
||||
import ca.joeltherrien.randomforest.tree.IntermediateCombinedResponse;
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
public class IntermediateCompetingRisksFunctionsTimesKnown implements IntermediateCombinedResponse<CompetingRiskFunctions, CompetingRiskFunctions> {
|
||||
|
||||
private double expectedN;
|
||||
private final int[] events;
|
||||
private final double[] timesToUse;
|
||||
private int actualN;
|
||||
|
||||
private final double[] survivalY;
|
||||
private final double[][] csCHFY;
|
||||
private final double[][] cifY;
|
||||
|
||||
public IntermediateCompetingRisksFunctionsTimesKnown(int n, int[] events, double[] timesToUse){
|
||||
this.expectedN = n;
|
||||
this.events = events;
|
||||
this.timesToUse = timesToUse;
|
||||
this.actualN = 0;
|
||||
|
||||
this.survivalY = new double[timesToUse.length];
|
||||
this.csCHFY = new double[events.length][timesToUse.length];
|
||||
this.cifY = new double[events.length][timesToUse.length];
|
||||
}
|
||||
|
||||
@Override
|
||||
public void processNewInput(CompetingRiskFunctions input) {
|
||||
/*
|
||||
We're going to try to efficiently put our predictions together -
|
||||
Assumptions - for each event on a response, the hazard and CIF functions share the same x points
|
||||
|
||||
Plan - go through the time on each response and make use of that so that when we search for a time index
|
||||
to evaluate the function at, we don't need to re-search the earlier times.
|
||||
|
||||
*/
|
||||
|
||||
this.actualN++;
|
||||
|
||||
final double[] survivalXPoints = input.getSurvivalCurve().getX();
|
||||
final double[][] eventSpecificXPoints = new double[events.length][];
|
||||
|
||||
for(final int event : events){
|
||||
eventSpecificXPoints[event-1] = input.getCumulativeIncidenceFunction(event)
|
||||
.getX();
|
||||
}
|
||||
|
||||
int previousSurvivalIndex = 0;
|
||||
final int[] previousEventSpecificIndex = new int[events.length]; // relying on 0 being default value
|
||||
|
||||
for(int i=0; i<timesToUse.length; i++){
|
||||
final double time = timesToUse[i];
|
||||
|
||||
// Survival curve
|
||||
final int survivalTimeIndex = Utils.binarySearchLessThan(previousSurvivalIndex, survivalXPoints.length, survivalXPoints, time);
|
||||
survivalY[i] = survivalY[i] + input.getSurvivalCurve().evaluateByIndex(survivalTimeIndex) / expectedN;
|
||||
previousSurvivalIndex = Math.max(survivalTimeIndex, 0); // if our current time is less than the smallest time in xPoints then binarySearchLessThan returned a -1.
|
||||
// -1's not an issue for evaluateByIndex, but it is an issue for the next time binarySearchLessThan is called.
|
||||
|
||||
// CHFs and CIFs
|
||||
for(final int event : events){
|
||||
final double[] xPoints = eventSpecificXPoints[event-1];
|
||||
final int eventTimeIndex = Utils.binarySearchLessThan(previousEventSpecificIndex[event-1], xPoints.length,
|
||||
xPoints, time);
|
||||
csCHFY[event-1][i] = csCHFY[event-1][i] + input.getCauseSpecificHazardFunction(event)
|
||||
.evaluateByIndex(eventTimeIndex) / expectedN;
|
||||
cifY[event-1][i] = cifY[event-1][i] + input.getCumulativeIncidenceFunction(event)
|
||||
.evaluateByIndex(eventTimeIndex) / expectedN;
|
||||
|
||||
previousEventSpecificIndex[event-1] = Math.max(eventTimeIndex, 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompetingRiskFunctions transformToOutput() {
|
||||
rescaleOutput();
|
||||
|
||||
final RightContinuousStepFunction survivalFunction = new RightContinuousStepFunction(timesToUse, survivalY, 1.0);
|
||||
final List<RightContinuousStepFunction> causeSpecificCumulativeHazardFunctionList = new ArrayList<>(events.length);
|
||||
final List<RightContinuousStepFunction> cumulativeIncidenceFunctionList = new ArrayList<>(events.length);
|
||||
|
||||
for(final int event : events){
|
||||
causeSpecificCumulativeHazardFunctionList.add(event-1, new RightContinuousStepFunction(timesToUse, csCHFY[event-1], 0));
|
||||
cumulativeIncidenceFunctionList.add(event-1, new RightContinuousStepFunction(timesToUse, cifY[event-1], 0));
|
||||
}
|
||||
|
||||
return CompetingRiskFunctions.builder()
|
||||
.causeSpecificHazards(causeSpecificCumulativeHazardFunctionList)
|
||||
.cumulativeIncidenceCurves(cumulativeIncidenceFunctionList)
|
||||
.survivalCurve(survivalFunction)
|
||||
.build();
|
||||
}
|
||||
|
||||
private void rescaleOutput() {
|
||||
rescaleArray(actualN, this.survivalY);
|
||||
|
||||
for(int event : events){
|
||||
rescaleArray(actualN, this.cifY[event - 1]);
|
||||
rescaleArray(actualN, this.csCHFY[event - 1]);
|
||||
}
|
||||
|
||||
this.expectedN = actualN;
|
||||
|
||||
}
|
||||
|
||||
private void rescaleArray(double newN, double[] array){
|
||||
for(int i=0; i<array.length; i++){
|
||||
array[i] = array[i] * (this.expectedN / newN);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -28,6 +28,7 @@ import java.util.List;
|
|||
*
|
||||
*/
|
||||
public class GrayLogRankSplitFinder extends CompetingRiskSplitFinder<CompetingRiskResponseWithCensorTime> {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int[] eventsOfFocus;
|
||||
private final int[] events;
|
||||
|
|
@ -28,6 +28,7 @@ import java.util.List;
|
|||
*
|
||||
*/
|
||||
public class LogRankSplitFinder extends CompetingRiskSplitFinder<CompetingRiskResponse> {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int[] eventsOfFocus;
|
||||
private final int[] events;
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.responses.regression;
|
||||
|
||||
import ca.joeltherrien.randomforest.tree.ForestResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.tree.IntermediateCombinedResponse;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Returns the Mean value of a group of Doubles.
|
||||
*
|
||||
*/
|
||||
public class MeanResponseCombiner implements ForestResponseCombiner<Double, Double> {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
@Override
|
||||
public Double combine(List<Double> responses) {
|
||||
final double size = responses.size();
|
||||
|
||||
return responses.stream().mapToDouble(db -> db/size).sum();
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public IntermediateCombinedResponse<Double, Double> startIntermediateCombinedResponse(int countInputs) {
|
||||
return new MeanIntermediateCombinedResponse(countInputs);
|
||||
}
|
||||
|
||||
public static class MeanIntermediateCombinedResponse implements IntermediateCombinedResponse<Double, Double>{
|
||||
|
||||
private double expectedN;
|
||||
private int actualN;
|
||||
private double currentMean;
|
||||
|
||||
public MeanIntermediateCombinedResponse(int n){
|
||||
this.expectedN = n;
|
||||
this.actualN = 0;
|
||||
this.currentMean = 0.0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void processNewInput(Double input) {
|
||||
this.currentMean = this.currentMean + input / expectedN;
|
||||
this.actualN ++;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Double transformToOutput() {
|
||||
// rescale if necessary
|
||||
this.currentMean = this.currentMean * (this.expectedN / (double) actualN);
|
||||
this.expectedN = actualN;
|
||||
|
||||
return currentMean;
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -27,6 +27,7 @@ import java.util.List;
|
|||
import java.util.stream.Collectors;
|
||||
|
||||
public class WeightedVarianceSplitFinder implements SplitFinder<Double> {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private Double getScore(Set leftHand, Set rightHand) {
|
||||
|
||||
|
|
@ -17,31 +17,18 @@
|
|||
package ca.joeltherrien.randomforest.tree;
|
||||
|
||||
import ca.joeltherrien.randomforest.CovariateRow;
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import lombok.Builder;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@Builder
|
||||
public class Forest<O, FO> { // O = output of trees, FO = forest output. In practice O == FO, even in competing risk & survival settings
|
||||
public abstract class Forest<O, FO> {
|
||||
|
||||
private final List<Tree<O>> trees;
|
||||
private final ResponseCombiner<O, FO> treeResponseCombiner;
|
||||
private final List<Covariate> covariateList;
|
||||
|
||||
public FO evaluate(CovariateRow row){
|
||||
|
||||
return treeResponseCombiner.combine(
|
||||
trees.stream()
|
||||
.map(node -> node.evaluate(row))
|
||||
.collect(Collectors.toList())
|
||||
);
|
||||
|
||||
}
|
||||
public abstract FO evaluate(CovariateRow row);
|
||||
public abstract FO evaluateOOB(CovariateRow row);
|
||||
public abstract Iterable<Tree<O>> getTrees();
|
||||
public abstract int getNumberOfTrees();
|
||||
|
||||
/**
|
||||
* Used primarily in the R package interface to avoid R loops; and for easier parallelization.
|
||||
|
|
@ -93,21 +80,6 @@ public class Forest<O, FO> { // O = output of trees, FO = forest output. In prac
|
|||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
public FO evaluateOOB(CovariateRow row){
|
||||
|
||||
return treeResponseCombiner.combine(
|
||||
trees.stream()
|
||||
.filter(tree -> !tree.idInBootstrapSample(row.getId()))
|
||||
.map(node -> node.evaluate(row))
|
||||
.collect(Collectors.toList())
|
||||
);
|
||||
|
||||
}
|
||||
|
||||
public List<Tree<O>> getTrees(){
|
||||
return Collections.unmodifiableList(trees);
|
||||
}
|
||||
|
||||
public Map<Integer, Integer> findSplitsByCovariate(){
|
||||
final Map<Integer, Integer> countMap = new TreeMap<>();
|
||||
|
||||
|
|
@ -158,4 +130,5 @@ public class Forest<O, FO> { // O = output of trees, FO = forest output. In prac
|
|||
return countTerminalNodes;
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -14,13 +14,10 @@
|
|||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.loaders;
|
||||
package ca.joeltherrien.randomforest.tree;
|
||||
|
||||
import org.apache.commons.csv.CSVRecord;
|
||||
public interface ForestResponseCombiner<I, O> extends ResponseCombiner<I, O>{
|
||||
|
||||
@FunctionalInterface
|
||||
public interface ResponseLoader<Y> {
|
||||
|
||||
Y parse(CSVRecord record);
|
||||
IntermediateCombinedResponse<I, O> startIntermediateCombinedResponse(int countInputs);
|
||||
|
||||
}
|
||||
|
|
@ -38,7 +38,7 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
|
||||
private final TreeTrainer<Y, TO> treeTrainer;
|
||||
private final List<Covariate> covariates;
|
||||
private final ResponseCombiner<TO, FO> treeResponseCombiner;
|
||||
private final ForestResponseCombiner<TO, FO> treeResponseCombiner;
|
||||
private final List<Row<Y>> data;
|
||||
|
||||
// number of trees to try
|
||||
|
|
@ -57,10 +57,10 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
* in which case its trees are combined with the new one.
|
||||
* @return A trained forest.
|
||||
*/
|
||||
public Forest<TO, FO> trainSerialInMemory(Optional<Forest<TO, FO>> initialForest){
|
||||
public OnlineForest<TO, FO> trainSerialInMemory(Optional<Forest<TO, FO>> initialForest){
|
||||
|
||||
final List<Tree<TO>> trees = new ArrayList<>(ntree);
|
||||
initialForest.ifPresent(forest -> trees.addAll(forest.getTrees()));
|
||||
initialForest.ifPresent(forest -> forest.getTrees().forEach(trees::add));
|
||||
|
||||
final Bootstrapper<Row<Y>> bootstrapper = new Bootstrapper<>(data);
|
||||
|
||||
|
|
@ -77,11 +77,9 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
System.out.println("Finished");
|
||||
}
|
||||
|
||||
|
||||
return Forest.<TO, FO>builder()
|
||||
return OnlineForest.<TO, FO>builder()
|
||||
.treeResponseCombiner(treeResponseCombiner)
|
||||
.trees(trees)
|
||||
.covariateList(covariates)
|
||||
.build();
|
||||
|
||||
}
|
||||
|
|
@ -94,7 +92,7 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
* There cannot be existing trees if the initial forest is
|
||||
* specified.
|
||||
*/
|
||||
public void trainSerialOnDisk(Optional<Forest<TO, FO>> initialForest){
|
||||
public OfflineForest<TO, FO> trainSerialOnDisk(Optional<Forest<TO, FO>> initialForest){
|
||||
// First we need to see how many trees there currently are
|
||||
final File folder = new File(saveTreeLocation);
|
||||
if(!folder.exists()){
|
||||
|
|
@ -115,17 +113,14 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
final AtomicInteger treeCount; // tracks how many trees are finished
|
||||
// Using an AtomicInteger is overkill for serial code, but this lets us reuse TreeSavedWorker
|
||||
if(initialForest.isPresent()){
|
||||
final List<Tree<TO>> initialTrees = initialForest.get().getTrees();
|
||||
|
||||
for(int j=0; j<initialTrees.size(); j++){
|
||||
int j=0;
|
||||
for(final Tree<TO> tree : initialForest.get().getTrees()){
|
||||
final String filename = "tree-" + (j+1) + ".tree";
|
||||
final Tree<TO> tree = initialTrees.get(j);
|
||||
|
||||
saveTree(tree, filename);
|
||||
|
||||
j++;
|
||||
}
|
||||
|
||||
treeCount = new AtomicInteger(initialTrees.size());
|
||||
treeCount = new AtomicInteger(j);
|
||||
} else{
|
||||
treeCount = new AtomicInteger(treeFiles.length);
|
||||
}
|
||||
|
|
@ -153,6 +148,8 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
System.out.println("Finished");
|
||||
}
|
||||
|
||||
return new OfflineForest<>(folder, treeResponseCombiner);
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -162,7 +159,7 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
* in which case its trees are combined with the new one.
|
||||
* @param threads The number of trees to train at once.
|
||||
*/
|
||||
public Forest<TO, FO> trainParallelInMemory(Optional<Forest<TO, FO>> initialForest, int threads){
|
||||
public OnlineForest<TO, FO> trainParallelInMemory(Optional<Forest<TO, FO>> initialForest, int threads){
|
||||
|
||||
// create a list that is pre-specified in size (I can call the .set method at any index < ntree without
|
||||
// the earlier indexes being filled.
|
||||
|
|
@ -170,11 +167,12 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
|
||||
final int startingCount;
|
||||
if(initialForest.isPresent()){
|
||||
final List<Tree<TO>> initialTrees = initialForest.get().getTrees();
|
||||
for(int j=0; j<initialTrees.size(); j++) {
|
||||
trees.set(j, initialTrees.get(j));
|
||||
int j = 0;
|
||||
for(final Tree<TO> tree : initialForest.get().getTrees()){
|
||||
trees.set(j, tree);
|
||||
j++;
|
||||
}
|
||||
startingCount = initialTrees.size();
|
||||
startingCount = initialForest.get().getNumberOfTrees();
|
||||
}
|
||||
else{
|
||||
startingCount = 0;
|
||||
|
|
@ -219,7 +217,7 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
System.out.println("\nFinished");
|
||||
}
|
||||
|
||||
return Forest.<TO, FO>builder()
|
||||
return OnlineForest.<TO, FO>builder()
|
||||
.treeResponseCombiner(treeResponseCombiner)
|
||||
.trees(trees)
|
||||
.build();
|
||||
|
|
@ -235,7 +233,7 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
* specified.
|
||||
* @param threads The number of trees to train at once.
|
||||
*/
|
||||
public void trainParallelOnDisk(Optional<Forest<TO, FO>> initialForest, int threads){
|
||||
public OfflineForest<TO, FO> trainParallelOnDisk(Optional<Forest<TO, FO>> initialForest, int threads){
|
||||
// First we need to see how many trees there currently are
|
||||
final File folder = new File(saveTreeLocation);
|
||||
if(!folder.exists()){
|
||||
|
|
@ -255,17 +253,14 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
|
||||
final AtomicInteger treeCount; // tracks how many trees are finished
|
||||
if(initialForest.isPresent()){
|
||||
final List<Tree<TO>> initialTrees = initialForest.get().getTrees();
|
||||
|
||||
for(int j=0; j<initialTrees.size(); j++){
|
||||
int j=0;
|
||||
for(final Tree<TO> tree : initialForest.get().getTrees()){
|
||||
final String filename = "tree-" + (j+1) + ".tree";
|
||||
final Tree<TO> tree = initialTrees.get(j);
|
||||
|
||||
saveTree(tree, filename);
|
||||
|
||||
j++;
|
||||
}
|
||||
|
||||
treeCount = new AtomicInteger(initialTrees.size());
|
||||
treeCount = new AtomicInteger(j);
|
||||
} else{
|
||||
treeCount = new AtomicInteger(treeFiles.length);
|
||||
}
|
||||
|
|
@ -309,6 +304,8 @@ public class ForestTrainer<Y, TO, FO> {
|
|||
System.out.println("\nFinished");
|
||||
}
|
||||
|
||||
return new OfflineForest<>(folder, treeResponseCombiner);
|
||||
|
||||
}
|
||||
|
||||
private Tree<TO> trainTree(final Bootstrapper<Row<Y>> bootstrapper, Random random){
|
||||
|
|
@ -14,25 +14,17 @@
|
|||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.responses.regression;
|
||||
|
||||
import ca.joeltherrien.randomforest.tree.ResponseCombiner;
|
||||
|
||||
import java.util.List;
|
||||
package ca.joeltherrien.randomforest.tree;
|
||||
|
||||
/**
|
||||
* Returns the Mean value of a group of Doubles.
|
||||
* Similar to ResponseCombiner, but an IntermediateCombinedResponse represents the intermediate state of a single output in the process of being combined.
|
||||
* This class is only used in OfflineForests where we can only load one Tree in memory at a time.
|
||||
*
|
||||
*/
|
||||
public class MeanResponseCombiner implements ResponseCombiner<Double, Double> {
|
||||
public interface IntermediateCombinedResponse<I, O> {
|
||||
|
||||
@Override
|
||||
public Double combine(List<Double> responses) {
|
||||
final double size = responses.size();
|
||||
|
||||
return responses.stream().mapToDouble(db -> db/size).sum();
|
||||
|
||||
}
|
||||
void processNewInput(I input);
|
||||
|
||||
O transformToOutput();
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,198 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.tree;
|
||||
|
||||
import ca.joeltherrien.randomforest.CovariateRow;
|
||||
import ca.joeltherrien.randomforest.utils.IterableOfflineTree;
|
||||
import lombok.AllArgsConstructor;
|
||||
|
||||
import java.io.File;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.stream.Collectors;
|
||||
import java.util.stream.IntStream;
|
||||
|
||||
@AllArgsConstructor
|
||||
public class OfflineForest<O, FO> extends Forest<O, FO> {
|
||||
|
||||
private final File[] treeFiles;
|
||||
private final ForestResponseCombiner<O, FO> treeResponseCombiner;
|
||||
|
||||
public OfflineForest(File treeDirectoryPath, ForestResponseCombiner<O, FO> treeResponseCombiner){
|
||||
this.treeResponseCombiner = treeResponseCombiner;
|
||||
|
||||
if(!treeDirectoryPath.isDirectory()){
|
||||
throw new IllegalArgumentException("treeDirectoryPath must point to a directory!");
|
||||
}
|
||||
|
||||
this.treeFiles = treeDirectoryPath.listFiles((file, s) -> s.endsWith(".tree"));
|
||||
}
|
||||
|
||||
@Override
|
||||
public FO evaluate(CovariateRow row) {
|
||||
final List<O> predictedOutputs = new ArrayList<>(treeFiles.length);
|
||||
for(final Tree<O> tree : getTrees()){
|
||||
final O prediction = tree.evaluate(row);
|
||||
predictedOutputs.add(prediction);
|
||||
}
|
||||
|
||||
return treeResponseCombiner.combine(predictedOutputs);
|
||||
}
|
||||
|
||||
@Override
|
||||
public FO evaluateOOB(CovariateRow row) {
|
||||
final List<O> predictedOutputs = new ArrayList<>(treeFiles.length);
|
||||
for(final Tree<O> tree : getTrees()){
|
||||
if(!tree.idInBootstrapSample(row.getId())){
|
||||
final O prediction = tree.evaluate(row);
|
||||
predictedOutputs.add(prediction);
|
||||
}
|
||||
}
|
||||
|
||||
return treeResponseCombiner.combine(predictedOutputs);
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public List<FO> evaluate(List<? extends CovariateRow> rowList){
|
||||
final List<IntermediateCombinedResponse<O, FO>> intermediatePredictions =
|
||||
IntStream.range(0, rowList.size())
|
||||
.mapToObj(i -> treeResponseCombiner.startIntermediateCombinedResponse(treeFiles.length))
|
||||
.collect(Collectors.toList());
|
||||
|
||||
final Iterator<Tree<O>> treeIterator = getTrees().iterator();
|
||||
for(int treeId = 0; treeId < treeFiles.length; treeId++){
|
||||
final Tree<O> currentTree = treeIterator.next();
|
||||
|
||||
IntStream.range(0, rowList.size()).parallel().forEach(
|
||||
rowId -> {
|
||||
final CovariateRow row = rowList.get(rowId);
|
||||
final O prediction = currentTree.evaluate(row);
|
||||
intermediatePredictions.get(rowId).processNewInput(prediction);
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
return intermediatePredictions.stream().parallel()
|
||||
.map(intPred -> intPred.transformToOutput())
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<FO> evaluateSerial(List<? extends CovariateRow> rowList){
|
||||
final List<IntermediateCombinedResponse<O, FO>> intermediatePredictions =
|
||||
IntStream.range(0, rowList.size())
|
||||
.mapToObj(i -> treeResponseCombiner.startIntermediateCombinedResponse(treeFiles.length))
|
||||
.collect(Collectors.toList());
|
||||
|
||||
final Iterator<Tree<O>> treeIterator = getTrees().iterator();
|
||||
for(int treeId = 0; treeId < treeFiles.length; treeId++){
|
||||
final Tree<O> currentTree = treeIterator.next();
|
||||
|
||||
IntStream.range(0, rowList.size()).sequential().forEach(
|
||||
rowId -> {
|
||||
final CovariateRow row = rowList.get(rowId);
|
||||
final O prediction = currentTree.evaluate(row);
|
||||
intermediatePredictions.get(rowId).processNewInput(prediction);
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
return intermediatePredictions.stream().sequential()
|
||||
.map(intPred -> intPred.transformToOutput())
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public List<FO> evaluateOOB(List<? extends CovariateRow> rowList){
|
||||
final List<IntermediateCombinedResponse<O, FO>> intermediatePredictions =
|
||||
IntStream.range(0, rowList.size())
|
||||
.mapToObj(i -> treeResponseCombiner.startIntermediateCombinedResponse(treeFiles.length))
|
||||
.collect(Collectors.toList());
|
||||
|
||||
final Iterator<Tree<O>> treeIterator = getTrees().iterator();
|
||||
for(int treeId = 0; treeId < treeFiles.length; treeId++){
|
||||
final Tree<O> currentTree = treeIterator.next();
|
||||
|
||||
IntStream.range(0, rowList.size()).parallel().forEach(
|
||||
rowId -> {
|
||||
final CovariateRow row = rowList.get(rowId);
|
||||
if(!currentTree.idInBootstrapSample(row.getId())){
|
||||
final O prediction = currentTree.evaluate(row);
|
||||
intermediatePredictions.get(rowId).processNewInput(prediction);
|
||||
}
|
||||
// else do nothing; when we get the final output it will get scaled for the smaller N
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
return intermediatePredictions.stream().parallel()
|
||||
.map(intPred -> intPred.transformToOutput())
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<FO> evaluateSerialOOB(List<? extends CovariateRow> rowList){
|
||||
final List<IntermediateCombinedResponse<O, FO>> intermediatePredictions =
|
||||
IntStream.range(0, rowList.size())
|
||||
.mapToObj(i -> treeResponseCombiner.startIntermediateCombinedResponse(treeFiles.length))
|
||||
.collect(Collectors.toList());
|
||||
|
||||
final Iterator<Tree<O>> treeIterator = getTrees().iterator();
|
||||
for(int treeId = 0; treeId < treeFiles.length; treeId++){
|
||||
final Tree<O> currentTree = treeIterator.next();
|
||||
|
||||
IntStream.range(0, rowList.size()).sequential().forEach(
|
||||
rowId -> {
|
||||
final CovariateRow row = rowList.get(rowId);
|
||||
if(!currentTree.idInBootstrapSample(row.getId())){
|
||||
final O prediction = currentTree.evaluate(row);
|
||||
intermediatePredictions.get(rowId).processNewInput(prediction);
|
||||
}
|
||||
// else do nothing; when we get the final output it will get scaled for the smaller N
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
return intermediatePredictions.stream().sequential()
|
||||
.map(intPred -> intPred.transformToOutput())
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Tree<O>> getTrees() {
|
||||
return new IterableOfflineTree<>(treeFiles);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getNumberOfTrees() {
|
||||
return treeFiles.length;
|
||||
}
|
||||
|
||||
public OnlineForest<O, FO> createOnlineCopy(){
|
||||
final List<Tree<O>> allTrees = new ArrayList<>(getNumberOfTrees());
|
||||
getTrees().forEach(allTrees::add);
|
||||
|
||||
return OnlineForest.<O, FO>builder()
|
||||
.trees(allTrees)
|
||||
.treeResponseCombiner(treeResponseCombiner)
|
||||
.build();
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.tree;
|
||||
|
||||
import ca.joeltherrien.randomforest.CovariateRow;
|
||||
import lombok.Builder;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@Builder
|
||||
public class OnlineForest<O, FO> extends Forest<O, FO> { // O = output of trees, FO = forest output. In practice O == FO, even in competing risk & survival settings
|
||||
|
||||
private final List<Tree<O>> trees;
|
||||
private final ResponseCombiner<O, FO> treeResponseCombiner;
|
||||
|
||||
@Override
|
||||
public FO evaluate(CovariateRow row){
|
||||
|
||||
return treeResponseCombiner.combine(
|
||||
trees.stream()
|
||||
.map(node -> node.evaluate(row))
|
||||
.collect(Collectors.toList())
|
||||
);
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public FO evaluateOOB(CovariateRow row){
|
||||
|
||||
return treeResponseCombiner.combine(
|
||||
trees.stream()
|
||||
.filter(tree -> !tree.idInBootstrapSample(row.getId()))
|
||||
.map(node -> node.evaluate(row))
|
||||
.collect(Collectors.toList())
|
||||
);
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Tree<O>> getTrees(){
|
||||
return Collections.unmodifiableList(trees);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getNumberOfTrees() {
|
||||
return trees.size();
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -17,15 +17,13 @@
|
|||
package ca.joeltherrien.randomforest.tree;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Getter;
|
||||
import lombok.Data;
|
||||
|
||||
@AllArgsConstructor
|
||||
@Data
|
||||
public class SplitAndScore<Y, V> {
|
||||
|
||||
@Getter
|
||||
private final Split<Y, V> split;
|
||||
|
||||
@Getter
|
||||
private final Double score;
|
||||
private Split<Y, V> split;
|
||||
private Double score;
|
||||
|
||||
}
|
||||
|
|
@ -29,6 +29,7 @@ import java.util.List;
|
|||
@ToString
|
||||
@Getter
|
||||
public class SplitNode<Y> implements Node<Y> {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final Node<Y> leftHand;
|
||||
private final Node<Y> rightHand;
|
||||
|
|
@ -27,6 +27,7 @@ import java.util.List;
|
|||
@RequiredArgsConstructor
|
||||
@ToString
|
||||
public class TerminalNode<Y> implements Node<Y> {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final Y responseValue;
|
||||
|
||||
|
|
@ -23,6 +23,7 @@ import java.util.Arrays;
|
|||
import java.util.List;
|
||||
|
||||
public class Tree<Y> implements Node<Y> {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
@Getter
|
||||
private final Node<Y> rootNode;
|
||||
|
|
@ -17,7 +17,9 @@
|
|||
package ca.joeltherrien.randomforest.tree;
|
||||
|
||||
import ca.joeltherrien.randomforest.Row;
|
||||
import ca.joeltherrien.randomforest.VisibleForTesting;
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.utils.SingletonIterator;
|
||||
import lombok.AccessLevel;
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
|
|
@ -72,31 +74,12 @@ public class TreeTrainer<Y, O> {
|
|||
|
||||
}
|
||||
|
||||
|
||||
// Now that we have the best split; we need to handle any NAs that were dropped off
|
||||
final double probabilityLeftHand = (double) bestSplit.leftHand.size() /
|
||||
(double) (bestSplit.leftHand.size() + bestSplit.rightHand.size());
|
||||
|
||||
// Assign missing values to the split if necessary
|
||||
if(covariates.get(bestSplit.getSplitRule().getParentCovariateIndex()).hasNAs()){
|
||||
bestSplit = bestSplit.modifiableClone(); // the lists in bestSplit are otherwise usually unmodifiable lists
|
||||
|
||||
for(Row<Y> row : data) {
|
||||
final int covariateIndex = bestSplit.getSplitRule().getParentCovariateIndex();
|
||||
|
||||
if(row.getValueByIndex(covariateIndex).isNA()) {
|
||||
final boolean randomDecision = random.nextDouble() <= probabilityLeftHand;
|
||||
|
||||
if(randomDecision){
|
||||
bestSplit.getLeftHand().add(row);
|
||||
}
|
||||
else{
|
||||
bestSplit.getRightHand().add(row);
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
bestSplit = randomlyAssignNAs(data, bestSplit, random);
|
||||
|
||||
final Node<O> leftNode;
|
||||
final Node<O> rightNode;
|
||||
|
|
@ -144,7 +127,8 @@ public class TreeTrainer<Y, O> {
|
|||
return splitCovariates;
|
||||
}
|
||||
|
||||
private Split<Y, ?> findBestSplitRule(List<Row<Y>> data, List<Covariate> covariatesToTry, Random random){
|
||||
@VisibleForTesting
|
||||
public Split<Y, ?> findBestSplitRule(List<Row<Y>> data, List<Covariate> covariatesToTry, Random random){
|
||||
|
||||
SplitAndScore<Y, ?> bestSplitAndScore = null;
|
||||
final SplitFinder noGenericSplitFinder = splitFinder; // cause Java generics are sometimes too frustrating
|
||||
|
|
@ -157,10 +141,32 @@ public class TreeTrainer<Y, O> {
|
|||
continue;
|
||||
}
|
||||
|
||||
final SplitAndScore<Y, ?> candidateSplitAndScore = noGenericSplitFinder.findBestSplit(iterator);
|
||||
SplitAndScore<Y, ?> candidateSplitAndScore = noGenericSplitFinder.findBestSplit(iterator);
|
||||
|
||||
if(candidateSplitAndScore != null && (bestSplitAndScore == null ||
|
||||
candidateSplitAndScore.getScore() > bestSplitAndScore.getScore())) {
|
||||
|
||||
if(candidateSplitAndScore == null){
|
||||
continue;
|
||||
}
|
||||
|
||||
// This score was based on splitting only non-NA values. However, there might be a similar covariate we are also considering
|
||||
// that is just as good at splitting but has less NAs; we should thus penalize the split score for variables with NAs
|
||||
// We do this by randomly assigning the NAs and then recalculating the split score on the best split we already have.
|
||||
//
|
||||
// We only have to penalize the score though if we know it's possible that this might be the best split. If it's not,
|
||||
// then we can skip the computations.
|
||||
final boolean mayBeGoodSplit = bestSplitAndScore == null ||
|
||||
candidateSplitAndScore.getScore() > bestSplitAndScore.getScore();
|
||||
if(mayBeGoodSplit && covariate.haveNASplitPenalty()){
|
||||
Split<Y, ?> candiateSplitWithNAs = randomlyAssignNAs(data, candidateSplitAndScore.getSplit(), random);
|
||||
final Iterator<Split<Y, ?>> newSplitWithRandomNAs = new SingletonIterator<>(candiateSplitWithNAs);
|
||||
final double newScore = splitFinder.findBestSplit(newSplitWithRandomNAs).getScore();
|
||||
|
||||
// There's a chance that NAs might add noise to *improve* the score; but we want to ensure we penalize it.
|
||||
// Thus we only change the score if its worse.
|
||||
candidateSplitAndScore.setScore(Math.min(newScore, candidateSplitAndScore.getScore()));
|
||||
}
|
||||
|
||||
if(bestSplitAndScore == null || candidateSplitAndScore.getScore() > bestSplitAndScore.getScore()) {
|
||||
bestSplitAndScore = candidateSplitAndScore;
|
||||
}
|
||||
|
||||
|
|
@ -174,6 +180,38 @@ public class TreeTrainer<Y, O> {
|
|||
|
||||
}
|
||||
|
||||
private <V> Split<Y, V> randomlyAssignNAs(List<Row<Y>> data, Split<Y, V> existingSplit, Random random){
|
||||
|
||||
// Now that we have the best split; we need to handle any NAs that were dropped off
|
||||
final double probabilityLeftHand = (double) existingSplit.leftHand.size() /
|
||||
(double) (existingSplit.leftHand.size() + existingSplit.rightHand.size());
|
||||
|
||||
|
||||
final int covariateIndex = existingSplit.getSplitRule().getParentCovariateIndex();
|
||||
|
||||
// Assign missing values to the split if necessary
|
||||
if(covariates.get(existingSplit.getSplitRule().getParentCovariateIndex()).hasNAs()){
|
||||
existingSplit = existingSplit.modifiableClone(); // the lists in bestSplit are otherwise usually unmodifiable lists
|
||||
|
||||
for(Row<Y> row : data) {
|
||||
if(row.getValueByIndex(covariateIndex).isNA()) {
|
||||
final boolean randomDecision = random.nextDouble() <= probabilityLeftHand;
|
||||
|
||||
if(randomDecision){
|
||||
existingSplit.getLeftHand().add(row);
|
||||
}
|
||||
else{
|
||||
existingSplit.getRightHand().add(row);
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return existingSplit;
|
||||
|
||||
}
|
||||
|
||||
private boolean nodeIsPure(List<Row<Y>> data){
|
||||
if(!checkNodePurity){
|
||||
return false;
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
package ca.joeltherrien.randomforest.tree.vimp;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Simple interface for VariableImportanceCalculator; takes in a List of observed responses and a List of predictions
|
||||
* and produces an average error measure.
|
||||
*
|
||||
* @param <Y> The class of the responses.
|
||||
* @param <P> The class of the predictions.
|
||||
*/
|
||||
public interface ErrorCalculator<Y, P>{
|
||||
|
||||
/**
|
||||
* Compares the observed responses with the predictions to produce an average error measure.
|
||||
* Lower errors should indicate a better model fit.
|
||||
*
|
||||
* @param responses
|
||||
* @param predictions
|
||||
* @return
|
||||
*/
|
||||
double averageError(List<Y> responses, List<P> predictions);
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
package ca.joeltherrien.randomforest.tree.vimp;
|
||||
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskFunctions;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskResponse;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.IBSCalculator;
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Implements ErrorCalculator; essentially just wraps around IBSCalculator to fit into VariableImportanceCalculator.
|
||||
*
|
||||
*/
|
||||
public class IBSErrorCalculatorWrapper implements ErrorCalculator<CompetingRiskResponse, CompetingRiskFunctions> {
|
||||
|
||||
private final IBSCalculator calculator;
|
||||
private final int[] events;
|
||||
private final double integrationUpperBound;
|
||||
private final double[] eventWeights;
|
||||
|
||||
public IBSErrorCalculatorWrapper(IBSCalculator calculator, int[] events, double integrationUpperBound, double[] eventWeights) {
|
||||
this.calculator = calculator;
|
||||
this.events = events;
|
||||
this.integrationUpperBound = integrationUpperBound;
|
||||
this.eventWeights = eventWeights;
|
||||
}
|
||||
|
||||
public IBSErrorCalculatorWrapper(IBSCalculator calculator, int[] events, double integrationUpperBound) {
|
||||
this.calculator = calculator;
|
||||
this.events = events;
|
||||
this.integrationUpperBound = integrationUpperBound;
|
||||
this.eventWeights = new double[events.length];
|
||||
|
||||
Arrays.fill(this.eventWeights, 1.0); // default is to just sum all errors together
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public double averageError(List<CompetingRiskResponse> responses, List<CompetingRiskFunctions> predictions) {
|
||||
final double[] errors = new double[events.length];
|
||||
final double n = responses.size();
|
||||
|
||||
for(int i=0; i < responses.size(); i++){
|
||||
final CompetingRiskResponse response = responses.get(i);
|
||||
final CompetingRiskFunctions prediction = predictions.get(i);
|
||||
|
||||
for(int k=0; k < this.events.length; k++){
|
||||
final int event = this.events[k];
|
||||
final RightContinuousStepFunction cif = prediction.getCumulativeIncidenceFunction(event);
|
||||
errors[k] += calculator.calculateError(response, cif, event, integrationUpperBound) / n;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
double totalError = 0.0;
|
||||
for(int k=0; k < this.events.length; k++){
|
||||
totalError += this.eventWeights[k] * errors[k];
|
||||
}
|
||||
|
||||
return totalError;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
package ca.joeltherrien.randomforest.tree.vimp;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public class RegressionErrorCalculator implements ErrorCalculator<Double, Double>{
|
||||
|
||||
@Override
|
||||
public double averageError(List<Double> responses, List<Double> predictions) {
|
||||
double mean = 0.0;
|
||||
final double n = responses.size();
|
||||
|
||||
for(int i=0; i<responses.size(); i++){
|
||||
final double response = responses.get(i);
|
||||
final double prediction = predictions.get(i);
|
||||
|
||||
final double difference = response - prediction;
|
||||
|
||||
mean += difference * difference / n;
|
||||
}
|
||||
|
||||
return mean;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,117 @@
|
|||
package ca.joeltherrien.randomforest.tree.vimp;
|
||||
|
||||
import ca.joeltherrien.randomforest.CovariateRow;
|
||||
import ca.joeltherrien.randomforest.Row;
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.tree.Tree;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Random;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
|
||||
public class VariableImportanceCalculator<Y, P> {
|
||||
|
||||
private final ErrorCalculator<Y, P> errorCalculator;
|
||||
private final List<Tree<P>> trees;
|
||||
private final List<Row<Y>> observations;
|
||||
|
||||
private final boolean isTrainingSet; // If true, then we use out-of-bag predictions
|
||||
private final double[] baselineErrors;
|
||||
|
||||
public VariableImportanceCalculator(
|
||||
ErrorCalculator<Y, P> errorCalculator,
|
||||
List<Tree<P>> trees,
|
||||
List<Row<Y>> observations,
|
||||
boolean isTrainingSet
|
||||
){
|
||||
this.errorCalculator = errorCalculator;
|
||||
this.trees = trees;
|
||||
this.observations = observations;
|
||||
this.isTrainingSet = isTrainingSet;
|
||||
|
||||
|
||||
try {
|
||||
|
||||
this.baselineErrors = new double[trees.size()];
|
||||
for (int i = 0; i < baselineErrors.length; i++) {
|
||||
final Tree<P> tree = trees.get(i);
|
||||
final List<Row<Y>> oobSubset = getAppropriateSubset(observations, tree); // may not actually be OOB depending on isTrainingSet
|
||||
final List<Y> responses = oobSubset.stream().map(Row::getResponse).collect(Collectors.toList());
|
||||
|
||||
this.baselineErrors[i] = errorCalculator.averageError(responses, makePredictions(oobSubset, tree));
|
||||
}
|
||||
|
||||
} catch(Exception e){
|
||||
e.printStackTrace();
|
||||
throw e;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns an array of importance values for every Tree for the given Covariate.
|
||||
*
|
||||
* @param covariate The Covariate to scramble.
|
||||
* @param random
|
||||
* @return
|
||||
*/
|
||||
public double[] calculateVariableImportanceRaw(Covariate covariate, Optional<Random> random){
|
||||
|
||||
final double[] vimp = new double[trees.size()];
|
||||
for(int i = 0; i < vimp.length; i++){
|
||||
final Tree<P> tree = trees.get(i);
|
||||
final List<Row<Y>> oobSubset = getAppropriateSubset(observations, tree); // may not actually be OOB depending on isTrainingSet
|
||||
final List<Y> responses = oobSubset.stream().map(Row::getResponse).collect(Collectors.toList());
|
||||
final List<CovariateRow> scrambledValues = CovariateRow.scrambleCovariateValues(oobSubset, covariate, random);
|
||||
|
||||
final double error = errorCalculator.averageError(responses, makePredictions(scrambledValues, tree));
|
||||
|
||||
vimp[i] = error - this.baselineErrors[i];
|
||||
}
|
||||
|
||||
return vimp;
|
||||
}
|
||||
|
||||
public double calculateVariableImportanceZScore(Covariate covariate, Optional<Random> random){
|
||||
final double[] vimpArray = calculateVariableImportanceRaw(covariate, random);
|
||||
|
||||
double mean = 0.0;
|
||||
double variance = 0.0;
|
||||
final double numTrees = vimpArray.length;
|
||||
|
||||
for(double vimp : vimpArray){
|
||||
mean += vimp / numTrees;
|
||||
}
|
||||
for(double vimp : vimpArray){
|
||||
variance += (vimp - mean)*(vimp - mean) / (numTrees - 1.0);
|
||||
}
|
||||
|
||||
final double standardError = Math.sqrt(variance / numTrees);
|
||||
|
||||
return mean / standardError;
|
||||
}
|
||||
|
||||
|
||||
|
||||
// Assume rowList has already been filtered for OOB
|
||||
private List<P> makePredictions(List<? extends CovariateRow> rowList, Tree<P> tree){
|
||||
return rowList.stream()
|
||||
.map(tree::evaluate)
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
private List<Row<Y>> getAppropriateSubset(List<Row<Y>> initialList, Tree<P> tree){
|
||||
if(!isTrainingSet){
|
||||
return initialList; // no need to make any subsets
|
||||
}
|
||||
|
||||
return initialList.stream()
|
||||
.filter(row -> !tree.idInBootstrapSample(row.getId()))
|
||||
.collect(Collectors.toList());
|
||||
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -16,9 +16,7 @@
|
|||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
import ca.joeltherrien.randomforest.tree.Forest;
|
||||
import ca.joeltherrien.randomforest.tree.ResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.tree.Tree;
|
||||
import ca.joeltherrien.randomforest.tree.*;
|
||||
|
||||
import java.io.*;
|
||||
import java.util.*;
|
||||
|
|
@ -27,12 +25,17 @@ import java.util.zip.GZIPOutputStream;
|
|||
|
||||
public class DataUtils {
|
||||
|
||||
public static <O, FO> Forest<O, FO> loadForest(File folder, ResponseCombiner<O, FO> treeResponseCombiner) throws IOException, ClassNotFoundException {
|
||||
public static <O, FO> OnlineForest<O, FO> loadOnlineForest(File folder, ResponseCombiner<O, FO> treeResponseCombiner) throws IOException, ClassNotFoundException {
|
||||
if(!folder.isDirectory()){
|
||||
throw new IllegalArgumentException("Tree directory must be a directory!");
|
||||
}
|
||||
|
||||
final File[] treeFiles = folder.listFiles((file, s) -> s.endsWith(".tree"));
|
||||
|
||||
return loadOnlineForest(treeFiles, treeResponseCombiner);
|
||||
}
|
||||
|
||||
public static <O, FO> OnlineForest<O, FO> loadOnlineForest(File[] treeFiles, ResponseCombiner<O, FO> treeResponseCombiner) throws IOException, ClassNotFoundException {
|
||||
final List<File> treeFileList = Arrays.asList(treeFiles);
|
||||
|
||||
Collections.sort(treeFileList, Comparator.comparing(File::getName));
|
||||
|
|
@ -48,16 +51,16 @@ public class DataUtils {
|
|||
|
||||
}
|
||||
|
||||
return Forest.<O, FO>builder()
|
||||
return OnlineForest.<O, FO>builder()
|
||||
.trees(treeList)
|
||||
.treeResponseCombiner(treeResponseCombiner)
|
||||
.build();
|
||||
|
||||
}
|
||||
|
||||
public static <O, FO> Forest<O, FO> loadForest(String folder, ResponseCombiner<O, FO> treeResponseCombiner) throws IOException, ClassNotFoundException {
|
||||
public static <O, FO> OnlineForest<O, FO> loadOnlineForest(String folder, ResponseCombiner<O, FO> treeResponseCombiner) throws IOException, ClassNotFoundException {
|
||||
final File directory = new File(folder);
|
||||
return loadForest(directory, treeResponseCombiner);
|
||||
return loadOnlineForest(directory, treeResponseCombiner);
|
||||
}
|
||||
|
||||
public static void saveObject(Serializable object, String filename) throws IOException {
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
/*
|
||||
* Copyright (c) 2019 Joel Therrien.
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
import ca.joeltherrien.randomforest.tree.Tree;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.FileInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.ObjectInputStream;
|
||||
import java.util.Iterator;
|
||||
import java.util.zip.GZIPInputStream;
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public class IterableOfflineTree<Y> implements Iterable<Tree<Y>> {
|
||||
|
||||
private final File[] treeFiles;
|
||||
|
||||
@Override
|
||||
public Iterator<Tree<Y>> iterator() {
|
||||
return new OfflineTreeIterator<>(treeFiles);
|
||||
}
|
||||
|
||||
@RequiredArgsConstructor
|
||||
public static class OfflineTreeIterator<Y> implements Iterator<Tree<Y>>{
|
||||
private final File[] treeFiles;
|
||||
private int position = 0;
|
||||
|
||||
@Override
|
||||
public boolean hasNext() {
|
||||
return position < treeFiles.length;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Tree<Y> next() {
|
||||
final File treeFile = treeFiles[position];
|
||||
position++;
|
||||
|
||||
|
||||
try {
|
||||
final ObjectInputStream inputStream= new ObjectInputStream(new GZIPInputStream(new FileInputStream(treeFile)));
|
||||
final Tree<Y> tree = (Tree) inputStream.readObject();
|
||||
return tree;
|
||||
} catch (IOException | ClassNotFoundException e) {
|
||||
e.printStackTrace();
|
||||
throw new RuntimeException("Failed to load tree for " + treeFile.toString());
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -26,6 +26,8 @@ import java.io.Serializable;
|
|||
*/
|
||||
@Data
|
||||
public class Point implements Serializable {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final double time;
|
||||
private final double y;
|
||||
}
|
||||
|
|
@ -188,4 +188,24 @@ public final class RUtils {
|
|||
return responses;
|
||||
}
|
||||
|
||||
public static List<Object> produceSublist(List<Object> initialList, int[] indices){
|
||||
final List<Object> newList = new ArrayList<>(indices.length);
|
||||
|
||||
for(int i : indices){
|
||||
newList.add(initialList.get(i));
|
||||
}
|
||||
|
||||
return newList;
|
||||
}
|
||||
|
||||
public static File[] getTreeFileArray(String folderPath, int endingId){
|
||||
final File[] fileArray = new File[endingId];
|
||||
|
||||
for(int i = 1; i <= endingId; i++){
|
||||
fileArray[i-1] = new File(folderPath + "/tree-" + i + ".tree");
|
||||
}
|
||||
|
||||
return fileArray;
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -16,8 +16,14 @@
|
|||
|
||||
package ca.joeltherrien.randomforest.utils;
|
||||
|
||||
import lombok.Getter;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.ListIterator;
|
||||
import java.util.function.DoubleBinaryOperator;
|
||||
import java.util.function.DoubleUnaryOperator;
|
||||
|
||||
/**
|
||||
* Represents a function represented by discrete points. We assume that the function is a stepwise right-continuous
|
||||
|
|
@ -26,13 +32,16 @@ import java.util.ListIterator;
|
|||
*/
|
||||
public final class RightContinuousStepFunction extends StepFunction {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final double[] y;
|
||||
|
||||
/**
|
||||
* Represents the value that should be returned by evaluate if there are points prior to the time the function is being evaluated at.
|
||||
* Represents the value that should be returned by evaluate if there are no points prior to the time the function is being evaluated at.
|
||||
*
|
||||
* Map be null.
|
||||
* May not be null.
|
||||
*/
|
||||
@Getter
|
||||
private final double defaultY;
|
||||
|
||||
public RightContinuousStepFunction(double[] x, double[] y, double defaultY) {
|
||||
|
|
@ -127,7 +136,12 @@ public final class RightContinuousStepFunction extends StepFunction {
|
|||
}
|
||||
|
||||
if(to < from){
|
||||
return integrate(to, from);
|
||||
return -integrate(to, from);
|
||||
}
|
||||
|
||||
// Edge case - no points; just defaultY
|
||||
if(this.x.length == 0){
|
||||
return (to - from) * this.defaultY;
|
||||
}
|
||||
|
||||
double summation = 0.0;
|
||||
|
|
@ -170,7 +184,7 @@ public final class RightContinuousStepFunction extends StepFunction {
|
|||
final double currentTime = xPoints[i];
|
||||
final double currentHeight = evaluateByIndex(i);
|
||||
|
||||
if(i == xPoints.length-1 || xPoints[i+1] > to){
|
||||
if(i == xPoints.length-1 || xPoints[i+1] >= to){
|
||||
summation += currentHeight * (to - currentTime);
|
||||
return summation;
|
||||
}
|
||||
|
|
@ -186,5 +200,76 @@ public final class RightContinuousStepFunction extends StepFunction {
|
|||
|
||||
}
|
||||
|
||||
public RightContinuousStepFunction unaryOperation(DoubleUnaryOperator operator){
|
||||
final double newDefaultY = operator.applyAsDouble(this.defaultY);
|
||||
final double[] newY = Arrays.stream(this.getY()).map(operator).toArray();
|
||||
|
||||
return new RightContinuousStepFunction(this.getX(), newY, newDefaultY);
|
||||
}
|
||||
|
||||
|
||||
public static RightContinuousStepFunction biOperation(RightContinuousStepFunction funLeft,
|
||||
RightContinuousStepFunction funRight,
|
||||
DoubleBinaryOperator operator){
|
||||
|
||||
final double newDefaultY = operator.applyAsDouble(funLeft.defaultY, funRight.defaultY);
|
||||
final double[] leftX = funLeft.x;
|
||||
final double[] rightX = funRight.x;
|
||||
|
||||
final List<Point> combinedPoints = new ArrayList<>(leftX.length + rightX.length);
|
||||
|
||||
// These indexes represent the times that have *already* been processed.
|
||||
// They start at -1 because we already processed the defaultY values.
|
||||
int indexLeft = -1;
|
||||
int indexRight = -1;
|
||||
|
||||
// This while-loop will keep going until one of the functions reaches the ends of its points
|
||||
while(indexLeft < leftX.length-1 && indexRight < rightX.length-1){
|
||||
final double time;
|
||||
if(leftX[indexLeft+1] < rightX[indexRight+1]){
|
||||
indexLeft += 1;
|
||||
|
||||
time = leftX[indexLeft];
|
||||
combinedPoints.add(new Point(time, operator.applyAsDouble(funLeft.evaluateByIndex(indexLeft), funRight.evaluateByIndex(indexRight))));
|
||||
}
|
||||
else if(leftX[indexLeft+1] > rightX[indexRight+1]){
|
||||
indexRight += 1;
|
||||
|
||||
time = rightX[indexRight];
|
||||
combinedPoints.add(new Point(time, operator.applyAsDouble(funLeft.evaluateByIndex(indexLeft), funRight.evaluateByIndex(indexRight))));
|
||||
}
|
||||
else{ // equal times
|
||||
indexLeft += 1;
|
||||
indexRight += 1;
|
||||
|
||||
time = leftX[indexLeft];
|
||||
combinedPoints.add(new Point(time, operator.applyAsDouble(funLeft.evaluateByIndex(indexLeft), funRight.evaluateByIndex(indexRight))));
|
||||
}
|
||||
}
|
||||
|
||||
// At this point, at least one of function left or function right has reached the end of its points
|
||||
|
||||
// This while-loop occurring implies that functionRight can not move forward anymore
|
||||
while(indexLeft < leftX.length-1){
|
||||
indexLeft += 1;
|
||||
|
||||
final double time = leftX[indexLeft];
|
||||
combinedPoints.add(new Point(time, operator.applyAsDouble(funLeft.evaluateByIndex(indexLeft), funRight.evaluateByIndex(indexRight))));
|
||||
|
||||
}
|
||||
|
||||
// This while-loop occurring implies that functionLeft can not move forward anymore
|
||||
while(indexRight < rightX.length-1){
|
||||
indexRight += 1;
|
||||
|
||||
final double time = rightX[indexRight];
|
||||
combinedPoints.add(new Point(time, operator.applyAsDouble(funLeft.evaluateByIndex(indexLeft), funRight.evaluateByIndex(indexRight))));
|
||||
|
||||
}
|
||||
|
||||
|
||||
return RightContinuousStepFunction.constructFromPoints(combinedPoints, newDefaultY);
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -20,7 +20,7 @@ import java.util.*;
|
|||
|
||||
public final class Utils {
|
||||
|
||||
public static StepFunction estimateOneMinusECDF(final double[] times){
|
||||
public static RightContinuousStepFunction estimateOneMinusECDF(final double[] times){
|
||||
Arrays.sort(times);
|
||||
|
||||
final Map<Double, Integer> timeCounterMap = new HashMap<>();
|
||||
|
|
@ -45,20 +45,20 @@ public class TestDeterministicForests {
|
|||
|
||||
int index = 0;
|
||||
for(int j=0; j<5; j++){
|
||||
final NumericCovariate numericCovariate = new NumericCovariate("numeric"+j, index);
|
||||
final NumericCovariate numericCovariate = new NumericCovariate("numeric"+j, index, false);
|
||||
covariateList.add(numericCovariate);
|
||||
index++;
|
||||
}
|
||||
|
||||
for(int j=0; j<5; j++){
|
||||
final BooleanCovariate booleanCovariate = new BooleanCovariate("boolean"+j, index);
|
||||
final BooleanCovariate booleanCovariate = new BooleanCovariate("boolean"+j, index, false);
|
||||
covariateList.add(booleanCovariate);
|
||||
index++;
|
||||
}
|
||||
|
||||
final List<String> levels = Utils.easyList("cat", "dog", "mouse");
|
||||
for(int j=0; j<5; j++){
|
||||
final FactorCovariate factorCovariate = new FactorCovariate("factor"+j, index, levels);
|
||||
final FactorCovariate factorCovariate = new FactorCovariate("factor"+j, index, levels, false);
|
||||
covariateList.add(factorCovariate);
|
||||
index++;
|
||||
}
|
||||
|
|
@ -214,14 +214,14 @@ public class TestDeterministicForests {
|
|||
|
||||
forestTrainer5Trees.trainSerialOnDisk(Optional.empty());
|
||||
forestTrainer10Trees.trainSerialOnDisk(Optional.empty());
|
||||
final Forest<Double, Double> forestSerial = DataUtils.loadForest(saveTreeFile, new MeanResponseCombiner());
|
||||
final Forest<Double, Double> forestSerial = DataUtils.loadOnlineForest(saveTreeFile, new MeanResponseCombiner());
|
||||
TestUtils.removeFolder(saveTreeFile);
|
||||
verifyTwoForestsEqual(testData, referenceForest, forestSerial);
|
||||
|
||||
|
||||
forestTrainer5Trees.trainParallelOnDisk(Optional.empty(), 4);
|
||||
forestTrainer10Trees.trainParallelOnDisk(Optional.empty(), 4);
|
||||
final Forest<Double, Double> forestParallel = DataUtils.loadForest(saveTreeFile, new MeanResponseCombiner());
|
||||
final Forest<Double, Double> forestParallel = DataUtils.loadOnlineForest(saveTreeFile, new MeanResponseCombiner());
|
||||
TestUtils.removeFolder(saveTreeFile);
|
||||
verifyTwoForestsEqual(testData, referenceForest, forestParallel);
|
||||
|
||||
|
|
@ -259,7 +259,7 @@ public class TestDeterministicForests {
|
|||
|
||||
for(int k=0; k<3; k++){
|
||||
forestTrainer.trainSerialOnDisk(Optional.empty());
|
||||
final Forest<Double, Double> replicantForest = DataUtils.loadForest(saveTreeFile, responseCombiner);
|
||||
final Forest<Double, Double> replicantForest = DataUtils.loadOnlineForest(saveTreeFile, responseCombiner);
|
||||
TestUtils.removeFolder(saveTreeFile);
|
||||
verifyTwoForestsEqual(testData, referenceForest, replicantForest);
|
||||
}
|
||||
|
|
@ -274,7 +274,7 @@ public class TestDeterministicForests {
|
|||
|
||||
for(int k=0; k<3; k++){
|
||||
forestTrainer.trainParallelOnDisk(Optional.empty(), 4);
|
||||
final Forest<Double, Double> replicantForest = DataUtils.loadForest(saveTreeFile, responseCombiner);
|
||||
final Forest<Double, Double> replicantForest = DataUtils.loadOnlineForest(saveTreeFile, responseCombiner);
|
||||
TestUtils.removeFolder(saveTreeFile);
|
||||
verifyTwoForestsEqual(testData, referenceForest, replicantForest);
|
||||
}
|
||||
|
|
@ -20,8 +20,8 @@ import ca.joeltherrien.randomforest.covariates.Covariate;
|
|||
import ca.joeltherrien.randomforest.covariates.numeric.NumericCovariate;
|
||||
import ca.joeltherrien.randomforest.responses.regression.MeanResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.responses.regression.WeightedVarianceSplitFinder;
|
||||
import ca.joeltherrien.randomforest.tree.Forest;
|
||||
import ca.joeltherrien.randomforest.tree.ForestTrainer;
|
||||
import ca.joeltherrien.randomforest.tree.OnlineForest;
|
||||
import ca.joeltherrien.randomforest.tree.Tree;
|
||||
import ca.joeltherrien.randomforest.tree.TreeTrainer;
|
||||
import ca.joeltherrien.randomforest.utils.DataUtils;
|
||||
|
|
@ -39,12 +39,12 @@ import static org.junit.jupiter.api.Assertions.*;
|
|||
|
||||
public class TestProvidingInitialForest {
|
||||
|
||||
private Forest<Double, Double> initialForest;
|
||||
private OnlineForest<Double, Double> initialForest;
|
||||
private List<Covariate> covariateList;
|
||||
private List<Row<Double>> data;
|
||||
|
||||
public TestProvidingInitialForest(){
|
||||
covariateList = Collections.singletonList(new NumericCovariate("x", 0));
|
||||
covariateList = Collections.singletonList(new NumericCovariate("x", 0, false));
|
||||
|
||||
data = Utils.easyList(
|
||||
Row.createSimple(Utils.easyMap("x", "1.0"), covariateList, 1, 1.0),
|
||||
|
|
@ -107,8 +107,8 @@ public class TestProvidingInitialForest {
|
|||
public void testSerialInMemory(){
|
||||
final ForestTrainer<Double, Double, Double> forestTrainer = getForestTrainer(null, 20);
|
||||
|
||||
final Forest<Double, Double> newForest = forestTrainer.trainSerialInMemory(Optional.of(initialForest));
|
||||
assertEquals(20, newForest.getTrees().size());
|
||||
final OnlineForest<Double, Double> newForest = forestTrainer.trainSerialInMemory(Optional.of(initialForest));
|
||||
assertEquals(20, newForest.getNumberOfTrees());
|
||||
|
||||
for(Tree<Double> initialTree : initialForest.getTrees()){
|
||||
assertTrue(newForest.getTrees().contains(initialTree));
|
||||
|
|
@ -124,8 +124,8 @@ public class TestProvidingInitialForest {
|
|||
public void testParallelInMemory(){
|
||||
final ForestTrainer<Double, Double, Double> forestTrainer = getForestTrainer(null, 20);
|
||||
|
||||
final Forest<Double, Double> newForest = forestTrainer.trainParallelInMemory(Optional.of(initialForest), 2);
|
||||
assertEquals(20, newForest.getTrees().size());
|
||||
final OnlineForest<Double, Double> newForest = forestTrainer.trainParallelInMemory(Optional.of(initialForest), 2);
|
||||
assertEquals(20, newForest.getNumberOfTrees());
|
||||
|
||||
for(Tree<Double> initialTree : initialForest.getTrees()){
|
||||
assertTrue(newForest.getTrees().contains(initialTree));
|
||||
|
|
@ -149,11 +149,11 @@ public class TestProvidingInitialForest {
|
|||
forestTrainer.trainParallelOnDisk(Optional.of(initialForest), 2);
|
||||
|
||||
assertEquals(20, directory.listFiles().length);
|
||||
final Forest<Double, Double> newForest = DataUtils.loadForest(directory, new MeanResponseCombiner());
|
||||
final OnlineForest<Double, Double> newForest = DataUtils.loadOnlineForest(directory, new MeanResponseCombiner());
|
||||
|
||||
|
||||
|
||||
assertEquals(20, newForest.getTrees().size());
|
||||
assertEquals(20, newForest.getNumberOfTrees());
|
||||
|
||||
final List<String> newForestTreesAsStrings = newForest.getTrees().stream()
|
||||
.map(tree -> tree.toString()).collect(Collectors.toList());
|
||||
|
|
@ -179,9 +179,9 @@ public class TestProvidingInitialForest {
|
|||
|
||||
assertEquals(20, directory.listFiles().length);
|
||||
|
||||
final Forest<Double, Double> newForest = DataUtils.loadForest(directory, new MeanResponseCombiner());
|
||||
final OnlineForest<Double, Double> newForest = DataUtils.loadOnlineForest(directory, new MeanResponseCombiner());
|
||||
|
||||
assertEquals(20, newForest.getTrees().size());
|
||||
assertEquals(20, newForest.getNumberOfTrees());
|
||||
|
||||
final List<String> newForestTreesAsStrings = newForest.getTrees().stream()
|
||||
.map(tree -> tree.toString()).collect(Collectors.toList());
|
||||
|
|
@ -198,7 +198,7 @@ public class TestProvidingInitialForest {
|
|||
it's not clear if the forest being provided is the same one that trees were saved from.
|
||||
*/
|
||||
@Test
|
||||
public void verifyExceptions(){
|
||||
public void testExceptions(){
|
||||
final String filePath = "src/test/resources/trees/";
|
||||
final File directory = new File(filePath);
|
||||
if(directory.exists()){
|
||||
|
|
@ -24,11 +24,10 @@ import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskRespons
|
|||
import ca.joeltherrien.randomforest.responses.competingrisk.combiner.CompetingRiskFunctionCombiner;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.combiner.CompetingRiskResponseCombiner;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.splitfinder.LogRankSplitFinder;
|
||||
import ca.joeltherrien.randomforest.tree.Forest;
|
||||
import ca.joeltherrien.randomforest.tree.ForestTrainer;
|
||||
import ca.joeltherrien.randomforest.tree.TreeTrainer;
|
||||
import ca.joeltherrien.randomforest.tree.*;
|
||||
import ca.joeltherrien.randomforest.utils.DataUtils;
|
||||
import ca.joeltherrien.randomforest.utils.ResponseLoader;
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
|
|
@ -47,10 +46,10 @@ public class TestSavingLoading {
|
|||
|
||||
public List<Covariate> getCovariates(){
|
||||
return Utils.easyList(
|
||||
new NumericCovariate("ageatfda", 0),
|
||||
new BooleanCovariate("idu", 1),
|
||||
new BooleanCovariate("black", 2),
|
||||
new NumericCovariate("cd4nadir", 3)
|
||||
new NumericCovariate("ageatfda", 0, false),
|
||||
new BooleanCovariate("idu", 1, false),
|
||||
new BooleanCovariate("black", 2, false),
|
||||
new NumericCovariate("cd4nadir", 3, false)
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -119,16 +118,21 @@ public class TestSavingLoading {
|
|||
assertTrue(directory.isDirectory());
|
||||
assertEquals(NTREE, directory.listFiles().length);
|
||||
|
||||
final Forest<CompetingRiskFunctions, CompetingRiskFunctions> forest = DataUtils.loadForest(directory, new CompetingRiskFunctionCombiner(new int[]{1,2}, null));
|
||||
final CompetingRiskFunctionCombiner treeResponseCombiner = new CompetingRiskFunctionCombiner(new int[]{1,2}, null);
|
||||
final OnlineForest<CompetingRiskFunctions, CompetingRiskFunctions> onlineForest = DataUtils.loadOnlineForest(directory, treeResponseCombiner);
|
||||
final OfflineForest<CompetingRiskFunctions, CompetingRiskFunctions> offlineForest = new OfflineForest<>(directory, treeResponseCombiner);
|
||||
|
||||
final CovariateRow predictionRow = getPredictionRow(covariates);
|
||||
|
||||
final CompetingRiskFunctions functions = forest.evaluate(predictionRow);
|
||||
assertNotNull(functions);
|
||||
assertTrue(functions.getCumulativeIncidenceFunction(1).getX().length > 2);
|
||||
final CompetingRiskFunctions functionsOnline = onlineForest.evaluate(predictionRow);
|
||||
assertNotNull(functionsOnline);
|
||||
assertTrue(functionsOnline.getCumulativeIncidenceFunction(1).getX().length > 2);
|
||||
|
||||
final CompetingRiskFunctions functionsOffline = offlineForest.evaluate(predictionRow);
|
||||
assertTrue(competingFunctionsEqual(functionsOffline, functionsOnline));
|
||||
|
||||
|
||||
assertEquals(NTREE, forest.getTrees().size());
|
||||
assertEquals(NTREE, onlineForest.getTrees().size());
|
||||
|
||||
TestUtils.removeFolder(directory);
|
||||
|
||||
|
|
@ -159,17 +163,22 @@ public class TestSavingLoading {
|
|||
assertEquals(NTREE, directory.listFiles().length);
|
||||
|
||||
|
||||
|
||||
final Forest<CompetingRiskFunctions, CompetingRiskFunctions> forest = DataUtils.loadForest(directory, new CompetingRiskFunctionCombiner(new int[]{1,2}, null));
|
||||
final CompetingRiskFunctionCombiner treeResponseCombiner = new CompetingRiskFunctionCombiner(new int[]{1,2}, null);
|
||||
final OnlineForest<CompetingRiskFunctions, CompetingRiskFunctions> onlineForest = DataUtils.loadOnlineForest(directory, treeResponseCombiner);
|
||||
final OfflineForest<CompetingRiskFunctions, CompetingRiskFunctions> offlineForest = new OfflineForest<>(directory, treeResponseCombiner);
|
||||
|
||||
final CovariateRow predictionRow = getPredictionRow(covariates);
|
||||
|
||||
final CompetingRiskFunctions functions = forest.evaluate(predictionRow);
|
||||
assertNotNull(functions);
|
||||
assertTrue(functions.getCumulativeIncidenceFunction(1).getX().length > 2);
|
||||
final CompetingRiskFunctions functionsOnline = onlineForest.evaluate(predictionRow);
|
||||
assertNotNull(functionsOnline);
|
||||
assertTrue(functionsOnline.getCumulativeIncidenceFunction(1).getX().length > 2);
|
||||
|
||||
|
||||
assertEquals(NTREE, forest.getTrees().size());
|
||||
final CompetingRiskFunctions functionsOffline = offlineForest.evaluate(predictionRow);
|
||||
assertTrue(competingFunctionsEqual(functionsOffline, functionsOnline));
|
||||
|
||||
|
||||
assertEquals(NTREE, onlineForest.getTrees().size());
|
||||
|
||||
TestUtils.removeFolder(directory);
|
||||
|
||||
|
|
@ -177,6 +186,64 @@ public class TestSavingLoading {
|
|||
|
||||
}
|
||||
|
||||
/*
|
||||
We don't implement equals() methods on the below mentioned classes because then we'd need to implement an
|
||||
appropriate hashCode() method that's consistent with the equals(), and we only need plain equals() for
|
||||
these tests.
|
||||
*/
|
||||
|
||||
private boolean competingFunctionsEqual(CompetingRiskFunctions f1 ,CompetingRiskFunctions f2){
|
||||
if(!functionsEqual(f1.getSurvivalCurve(), f2.getSurvivalCurve())){
|
||||
return false;
|
||||
}
|
||||
|
||||
for(int i=1; i<=2; i++){
|
||||
if(!functionsEqual(f1.getCauseSpecificHazardFunction(i), f2.getCauseSpecificHazardFunction(i))){
|
||||
return false;
|
||||
}
|
||||
if(!functionsEqual(f1.getCumulativeIncidenceFunction(i), f2.getCumulativeIncidenceFunction(i))){
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private boolean functionsEqual(RightContinuousStepFunction f1, RightContinuousStepFunction f2){
|
||||
|
||||
final double[] f1X = f1.getX();
|
||||
final double[] f2X = f2.getX();
|
||||
|
||||
final double[] f1Y = f1.getY();
|
||||
final double[] f2Y = f2.getY();
|
||||
|
||||
// first compare array lengths
|
||||
if(f1X.length != f2X.length){
|
||||
return false;
|
||||
}
|
||||
if(f1Y.length != f2Y.length){
|
||||
return false;
|
||||
}
|
||||
|
||||
// TODO - better comparisons of doubles. I don't really care too much though as this equals method is only being used in tests
|
||||
final double delta = 0.000001;
|
||||
|
||||
if(Math.abs(f1.getDefaultY() - f2.getDefaultY()) > delta){
|
||||
return false;
|
||||
}
|
||||
|
||||
for(int i=0; i < f1X.length; i++){
|
||||
if(Math.abs(f1X[i] - f2X[i]) > delta){
|
||||
return false;
|
||||
}
|
||||
if(Math.abs(f1Y[i] - f2Y[i]) > delta){
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -156,7 +156,7 @@ public class TestUtils {
|
|||
}
|
||||
|
||||
@Test
|
||||
public void reduceListToSize(){
|
||||
public void testReduceListToSize(){
|
||||
final List<Integer> testList = Utils.easyList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
|
||||
final Random random = new Random();
|
||||
for(int i=0; i<100; i++) { // want to test many times to ensure it doesn't work just due to randomness
|
||||
|
|
@ -0,0 +1,164 @@
|
|||
package ca.joeltherrien.randomforest.competingrisk;
|
||||
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskFunctions;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskResponse;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskUtils;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.IBSCalculator;
|
||||
import ca.joeltherrien.randomforest.utils.Point;
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
public class IBSCalculatorTest {
|
||||
|
||||
private final RightContinuousStepFunction cif;
|
||||
|
||||
public IBSCalculatorTest(){
|
||||
this.cif = RightContinuousStepFunction.constructFromPoints(
|
||||
Utils.easyList(
|
||||
new Point(1.0, 0.1),
|
||||
new Point(2.0, 0.2),
|
||||
new Point(3.0, 0.3),
|
||||
new Point(4.0, 0.8)
|
||||
), 0.0
|
||||
);
|
||||
}
|
||||
|
||||
/*
|
||||
R code to get these results:
|
||||
|
||||
predicted_cif <- stepfun(1:4, c(0, 0.1, 0.2, 0.3, 0.8))
|
||||
weights <- 1
|
||||
recorded_time <- 2.0
|
||||
recorded_status <- 1.0
|
||||
event_of_interest <- 2
|
||||
times <- 0:4
|
||||
|
||||
errors <- weights * ( as.integer(recorded_time <= times & recorded_status == event_of_interest) - predicted_cif(times))^2
|
||||
sum(errors)
|
||||
|
||||
|
||||
and run again with event_of_interest <- 1
|
||||
|
||||
|
||||
Note that in the R code I only evaluate up to 4, while in the Java code I integrate up to 5
|
||||
This is because the evaluation at 4 is giving the area of the rectangle from 4 to 5.
|
||||
|
||||
*/
|
||||
|
||||
@Test
|
||||
public void testResultsWithoutCensoringDistribution(){
|
||||
final IBSCalculator calculator = new IBSCalculator();
|
||||
|
||||
final double errorDifferentEvent = calculator.calculateError(
|
||||
new CompetingRiskResponse(1, 2.0),
|
||||
this.cif,
|
||||
2,
|
||||
5.0);
|
||||
|
||||
assertEquals(0.78, errorDifferentEvent, 0.000001);
|
||||
|
||||
final double errorSameEvent = calculator.calculateError(
|
||||
new CompetingRiskResponse(1, 2.0),
|
||||
this.cif,
|
||||
1,
|
||||
5.0);
|
||||
|
||||
assertEquals(1.18, errorSameEvent, 0.000001);
|
||||
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testResultsWithCensoringDistribution(){
|
||||
final RightContinuousStepFunction censorSurvivalFunction = RightContinuousStepFunction.constructFromPoints(
|
||||
Utils.easyList(
|
||||
new Point(0.0, 0.75),
|
||||
new Point(1.0, 0.5),
|
||||
new Point(3.0, 0.25),
|
||||
new Point(5.0, 0)
|
||||
), 1.0
|
||||
);
|
||||
|
||||
final IBSCalculator calculator = new IBSCalculator(censorSurvivalFunction);
|
||||
|
||||
final double errorDifferentEvent = calculator.calculateError(
|
||||
new CompetingRiskResponse(1, 2.0),
|
||||
this.cif,
|
||||
2,
|
||||
5.0);
|
||||
|
||||
assertEquals(1.56, errorDifferentEvent, 0.000001);
|
||||
|
||||
final double errorSameEvent = calculator.calculateError(
|
||||
new CompetingRiskResponse(1, 2.0),
|
||||
this.cif,
|
||||
1,
|
||||
5.0);
|
||||
|
||||
assertEquals(2.36, errorSameEvent, 0.000001);
|
||||
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testStaticFunction(){
|
||||
final RightContinuousStepFunction censorSurvivalFunction = RightContinuousStepFunction.constructFromPoints(
|
||||
Utils.easyList(
|
||||
new Point(0.0, 0.75),
|
||||
new Point(1.0, 0.5),
|
||||
new Point(3.0, 0.25),
|
||||
new Point(5.0, 0)
|
||||
), 1.0
|
||||
);
|
||||
|
||||
final List<CompetingRiskResponse> responseList = Utils.easyList(
|
||||
new CompetingRiskResponse(1, 2.0),
|
||||
new CompetingRiskResponse(1, 2.0));
|
||||
|
||||
// for predictions; we'll construct an improper CompetingRisksFunctions
|
||||
final RightContinuousStepFunction trivialFunction = RightContinuousStepFunction.constructFromPoints(
|
||||
Utils.easyList(new Point(1.0, 0.0)),
|
||||
1.0);
|
||||
|
||||
final CompetingRiskFunctions prediction = CompetingRiskFunctions.builder()
|
||||
.survivalCurve(trivialFunction)
|
||||
.causeSpecificHazards(Utils.easyList(trivialFunction, trivialFunction))
|
||||
.cumulativeIncidenceCurves(Utils.easyList(this.cif, trivialFunction))
|
||||
.build();
|
||||
|
||||
final List<CompetingRiskFunctions> predictionList = Utils.easyList(prediction, prediction);
|
||||
|
||||
double[] errorParallel = CompetingRiskUtils.calculateIBSError(
|
||||
responseList,
|
||||
predictionList,
|
||||
Optional.of(censorSurvivalFunction),
|
||||
1,
|
||||
5.0,
|
||||
true);
|
||||
|
||||
double[] errorSerial = CompetingRiskUtils.calculateIBSError(
|
||||
responseList,
|
||||
predictionList,
|
||||
Optional.of(censorSurvivalFunction),
|
||||
1,
|
||||
5.0,
|
||||
false);
|
||||
|
||||
assertEquals(responseList.size(), errorParallel.length);
|
||||
assertEquals(responseList.size(), errorSerial.length);
|
||||
|
||||
assertEquals(2.36, errorParallel[0], 0.000001);
|
||||
assertEquals(2.36, errorParallel[1], 0.000001);
|
||||
|
||||
assertEquals(2.36, errorSerial[0], 0.000001);
|
||||
assertEquals(2.36, errorSerial[1], 0.000001);
|
||||
|
||||
}
|
||||
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -53,10 +53,10 @@ public class TestCompetingRisk {
|
|||
|
||||
public List<Covariate> getCovariates(){
|
||||
return Utils.easyList(
|
||||
new NumericCovariate("ageatfda", 0),
|
||||
new BooleanCovariate("idu", 1),
|
||||
new BooleanCovariate("black", 2),
|
||||
new NumericCovariate("cd4nadir", 3)
|
||||
new NumericCovariate("ageatfda", 0, false),
|
||||
new BooleanCovariate("idu", 1, false),
|
||||
new BooleanCovariate("black", 2, false),
|
||||
new NumericCovariate("cd4nadir", 3, false)
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -109,8 +109,8 @@ public class TestCompetingRisk {
|
|||
|
||||
// by only using BooleanCovariates (only one split rule) we can guarantee identical results with randomForestSRC on one tree.
|
||||
final List<Covariate> covariates = Utils.easyList(
|
||||
new BooleanCovariate("idu", 0),
|
||||
new BooleanCovariate("black", 1)
|
||||
new BooleanCovariate("idu", 0, false),
|
||||
new BooleanCovariate("black", 1, false)
|
||||
);
|
||||
|
||||
final List<Row<CompetingRiskResponse>> dataset = getData(covariates, "src/test/resources/wihs.bootstrapped.csv");
|
||||
|
|
@ -210,8 +210,8 @@ public class TestCompetingRisk {
|
|||
public void testLogRankSplitFinderTwoBooleans() throws IOException {
|
||||
// by only using BooleanCovariates (only one split rule) we can guarantee identical results with randomForestSRC on one tree.
|
||||
final List<Covariate> covariates = Utils.easyList(
|
||||
new BooleanCovariate("idu", 0),
|
||||
new BooleanCovariate("black", 1)
|
||||
new BooleanCovariate("idu", 0, false),
|
||||
new BooleanCovariate("black", 1, false)
|
||||
);
|
||||
|
||||
|
||||
|
|
@ -259,7 +259,7 @@ public class TestCompetingRisk {
|
|||
}
|
||||
|
||||
@Test
|
||||
public void verifyDataset() throws IOException {
|
||||
public void testDataset() throws IOException {
|
||||
final List<Covariate> covariates = getCovariates();
|
||||
|
||||
final List<Row<CompetingRiskResponse>> dataset = getData(covariates, DEFAULT_FILEPATH);
|
||||
|
|
@ -16,10 +16,11 @@
|
|||
|
||||
package ca.joeltherrien.randomforest.competingrisk;
|
||||
|
||||
import ca.joeltherrien.randomforest.Row;
|
||||
import ca.joeltherrien.randomforest.covariates.Covariate;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.*;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskFunctions;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskResponse;
|
||||
import ca.joeltherrien.randomforest.responses.competingrisk.CompetingRiskUtils;
|
||||
import ca.joeltherrien.randomforest.tree.Forest;
|
||||
import ca.joeltherrien.randomforest.tree.OnlineForest;
|
||||
import ca.joeltherrien.randomforest.utils.RightContinuousStepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.StepFunction;
|
||||
import ca.joeltherrien.randomforest.utils.Utils;
|
||||
|
|
@ -30,8 +31,6 @@ import java.util.List;
|
|||
|
||||
import static ca.joeltherrien.randomforest.TestUtils.closeEnough;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
public class TestCompetingRiskErrorRateCalculator {
|
||||
|
||||
|
|
@ -48,7 +47,7 @@ public class TestCompetingRiskErrorRateCalculator {
|
|||
|
||||
final int event = 1;
|
||||
|
||||
final Forest<CompetingRiskFunctions, CompetingRiskFunctions> fakeForest = Forest.<CompetingRiskFunctions, CompetingRiskFunctions>builder().build();
|
||||
final Forest<CompetingRiskFunctions, CompetingRiskFunctions> fakeForest = OnlineForest.<CompetingRiskFunctions, CompetingRiskFunctions>builder().build();
|
||||
|
||||
final double naiveConcordance = CompetingRiskUtils.calculateConcordance(responseList, mortalityArray, event);
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue