This example script is intended to illustrate how to use the 'ddmore' R package to perform a M&S workflow using the DDMoRe Standalone Execution Environment (SEE).
The following steps are implemented in this workflow:
To run a task, select wiht the coursor any code lines you wish to execute and press CTRL+R+R in your keyboard. An HTML file containing the commands in this file and associated output will be provided to allow the user to compare the results
Clear workspace and set working directory under 'UsesCasesDemo' project
rm(list=ls(all=F))
mydir <- file.path(Sys.getenv("MDLIDE_WORKSPACE_HOME"),"UseCasesDemo")
setwd(mydir)
Create a working directory under 'models' folder where results are stored
uc<-"UseCase14"
datafile <- "warfarin_TTE_exact.csv"
mdlfile <- paste0(uc,".mdl")
Create a new folder to be used as working directory and where results will be stored
wd <- file.path(mydir,uc)
dir.create(wd)
Copy the dataset and the .mdl file available under “models” into the working directory
file.copy(file.path(mydir,"models", datafile),wd)
## [1] TRUE
file.copy(file.path(mydir,"models",mdlfile),wd)
## [1] TRUE
Set the working directory.
setwd(file.path(mydir,uc))
The working directory needs to be set differently when knitr R package is used to spin the file
library(knitr)
opts_knit$set(root.dir = file.path(mydir,uc))
List files available in working directory
list.files()
## [1] "UseCase14.mdl" "warfarin_TTE_exact.csv"
Use 'ddmore' function getMDLObjects() to retrieve model object(s) from an existing .mdl file. This function reads the MDL in an .mdl file and parses the MDL code for each MDL Object into an R list of objects of appropriate types with names corresponding to the MDL Object names given in the file.
myMDLObj <- getMDLObjects(mdlfile)
length(myMDLObj)
## [1] 4
names(myMDLObj)
## [1] "warfarin_T2E_exact_dat" "warfarin_T2E_exact_par"
## [3] "warfarin_T2E_exact_mdl" "warfarin_T2E_exact_task"
Use 'ddmore' function getDataObjects() to retrieve only data object(s) from an existing .mdl file This function returns a list of Parameter Object(s) from which we select the first element Hover over the variable name to see its structure
myDataObj <- getDataObjects(mdlfile)[[1]]
Use 'ddmore' function getParameterObjects() to retrieve only parameter object(s) from an existing .mdl file
myParObj <- getParameterObjects(mdlfile)[[1]]
Use 'ddmore' function getModelObjects() to retrieve only model object(s) from an existing .mdl file.
myModObj <- getModelObjects(mdlfile)[[1]]
Use 'ddmore' function getTaskPropertiesObjects() to retrieve only task properties object(s) from an existing .mdl file
myTaskObj <- getTaskPropertiesObjects(mdlfile)[[1]]
Recall that getDataObjects only reads the MDL code from the .mdl file. Use 'ddmore' function readDataObj() to create an R object from the MDL data object.
myData <- readDataObj(myDataObj)
Let's look at the first 6 lines of the data set
head(myData)
## ID TIME TRT AMT RATE ADDL II WT DVID DV MDV REP CPX PCAX
## 1 1 0 0 1 -2 27 1 72.065 3 0 0 1 0.00000 96.572
## 2 1 28 0 0 0 0 0 72.065 3 0 0 1 0.00000 96.572
## 3 2 0 1 1 -2 27 1 71.471 3 0 0 1 0.00000 95.398
## 4 2 28 1 0 0 0 0 71.471 3 0 0 1 0.77845 75.223
## 5 3 0 1 1 -2 27 1 69.330 3 0 0 1 0.00000 94.227
## 6 3 4 1 0 0 0 0 69.330 3 1 0 1 0.51216 58.072
## INRX ICL IV ITABS ITLAG IF1 IPCA0 IEMAX IC50 ITEQ
## 1 1.0000 2.9581 9.075 0.014635 0.109280 0.0 96.572 -1 1.39470 0.45693
## 2 1.0000 2.9581 9.075 0.014635 0.109280 0.0 96.572 -1 1.39470 0.45693
## 3 1.0000 2.7709 8.709 0.030355 0.034677 2.5 95.398 -1 3.27190 0.53506
## 4 1.2682 2.7709 8.709 0.030355 0.034677 2.5 95.398 -1 3.27190 0.53506
## 5 1.0000 2.9963 10.192 0.042503 0.085843 2.5 94.227 -1 0.83837 0.51051
## 6 1.6226 2.9963 10.192 0.042503 0.085843 2.5 94.227 -1 0.83837 0.51051
## NEVT
## 1 1
## 2 1
## 3 1
## 4 1
## 5 1
## 6 1
Extract only observation records
myEDAData<-myData[myData$MDV==0,]
head(myEDAData)
## ID TIME TRT AMT RATE ADDL II WT DVID DV MDV REP CPX PCAX
## 1 1 0 0 1 -2 27 1 72.065 3 0 0 1 0.00000 96.572
## 2 1 28 0 0 0 0 0 72.065 3 0 0 1 0.00000 96.572
## 3 2 0 1 1 -2 27 1 71.471 3 0 0 1 0.00000 95.398
## 4 2 28 1 0 0 0 0 71.471 3 0 0 1 0.77845 75.223
## 5 3 0 1 1 -2 27 1 69.330 3 0 0 1 0.00000 94.227
## 6 3 4 1 0 0 0 0 69.330 3 1 0 1 0.51216 58.072
## INRX ICL IV ITABS ITLAG IF1 IPCA0 IEMAX IC50 ITEQ
## 1 1.0000 2.9581 9.075 0.014635 0.109280 0.0 96.572 -1 1.39470 0.45693
## 2 1.0000 2.9581 9.075 0.014635 0.109280 0.0 96.572 -1 1.39470 0.45693
## 3 1.0000 2.7709 8.709 0.030355 0.034677 2.5 95.398 -1 3.27190 0.53506
## 4 1.2682 2.7709 8.709 0.030355 0.034677 2.5 95.398 -1 3.27190 0.53506
## 5 1.0000 2.9963 10.192 0.042503 0.085843 2.5 94.227 -1 0.83837 0.51051
## 6 1.6226 2.9963 10.192 0.042503 0.085843 2.5 94.227 -1 0.83837 0.51051
## NEVT
## 1 1
## 2 1
## 3 1
## 4 1
## 5 1
## 6 1
Open an R window to record and access all your plots
windows(record=TRUE)
Plot Kaplan-Meier survival curve
library(survival)
kmfit <- survfit(formula=Surv(TIME,DV==1)~1, data=myData)
plot1 <- plot(kmfit, xlab="Time(days)", ylab="Survival Probability", conf.int=F)
print(plot1)
## $x
## [1] 28
##
## $y
## [1] 0.44
Plot Kaplan-Meier survival curve stratified by Treatment
kmfit.TRT <- survfit(formula=Surv(TIME,DV==1)~TRT, data=myData)
trt <- unique(myData$TRT)
plot2 <- plot(kmfit.TRT, xlab="Time(days)", ylab="Survival Probability", conf.int=F,
col=c(1:length(myData$TRT)))
legend("bottomleft", legend= paste("TRT",unique(myData$TRT)) ,
text.col=c(1:length(myData$TRT)),
bty="n")
print(plot2)
## $x
## [1] 28 28 28 28
##
## $y
## [1] 0.4782609 0.4838710 0.3500000 0.4230769
Export results to a PDF file
pdf(paste0(uc,"_EGA.pdf"))
print(plot1)
## $x
## [1] 28
##
## $y
## [1] 0.44
print(plot2)
## $x
## [1] 28 28 28 28
##
## $y
## [1] 0.4782609 0.4838710 0.3500000 0.4230769
dev.off()
## rj.GD
## 2
The ddmore “estimate” function translates the contents of the .mdl file to a target language and then estimates parameters using the target software. After estimation, the output is converted to a Standardised Output object which is saved in a .SO.xml file.
Translated files and Monolix output will be returned in the ./Monolix subfolder. The Standardised Output object (.SO.xml) is read and parsed into an R object called “mlx” of (S4) class “StandardOutputObject”.
mlx <- estimate(mdlfile, target="MONOLIX", subfolder="Monolix")
## -- Fri Dec 11 02:38:26 2015
## New
## Submitted
## Job d90613a6-9338-447a-90f7-203c29053076 progress:
## Running [ ........ ]
## Importing Results
## Copying the result data back to the local machine for job ID d90613a6-9338-447a-90f7-203c29053076...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd03a6c47ad to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase14/Monolix
## Done.
##
## Failed
## -- Fri Dec 11 02:41:10 2015
## Error in DDMORE.performExecutionWorkflow(submission, fisServer = fisServer, : Execution of model D:\SEE-1.2-SNAPSHOT-10DEC2015\MDL_IDE\workspace\UseCasesDemo\UseCase14\UseCase14.mdl failed.
## The contents of the working directory D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase14/Monolix may be useful for tracking down the cause of the failure.
## Additionally, the submission object has been saved to the hidden variable .ddmoreFailedSubmission in the workspace; internal errors should have been captured in .ddmoreFailedSubmission$error$message.
There is a bug in the execution process and the SO is not generated, so the execution fails, however, the raw results from Monolix execution can be seen under the Monolix subfolderslotNames(mlx) The ddmore “LoadSOObj” reads and parsed existing Standardise Output Objects mlx <- LoadSOObject(“Monolix/UseCase14.SO.xml”) The ddmore function “getPopulationParameters” extracts the Population Parameter values from an R object of (S4) class “StandardOutputObject” and returns the estimates. See documentation for getPopulationParameters to see other arguments and settings for this function.
#parameters_mlx <- getPopulationParameters(mlx, what="estimates")$MLE
#print(parameters_mlx)
#print(getPopulationParameters(mlx, what="precisions"))
#print(getEstimationInfo(mlx))
The ddmore “estimate” function translates the contents of the .mdl file to a target language and then estimates parameters using the target software. After estimation, the output is converted to a Standardised Output object which is saved in a .SO.xml file.
Translated files and Monolix output will be returned in the ./Monolix subfolder. The Standardised Output object (.SO.xml) is read and parsed into an R object called “mlx” of (S4) class “StandardOutputObject”.
NM <- estimate(mdlfile, target="NONMEM", subfolder="NONMEM")
## -- Fri Dec 11 02:41:10 2015
## New
## Submitted
## Job 675cedd7-4420-4d91-aa5a-73004c5f571a progress:
## Running [ ..... ]
## Importing Results
## Copying the result data back to the local machine for job ID 675cedd7-4420-4d91-aa5a-73004c5f571a...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd05953727a to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase14/NONMEM
## Done.
##
##
## The following elements were parsed successfully:
## RawResults
## TaskInformation
## Estimation:PopulationEstimates
## Estimation:IndividualEstimates
## Estimation:Likelihood
##
## The following MESSAGEs were raised during the job execution:
## minimization_successful: 1
## covariance_step_run: 0
## rounding_errors: 0
## estimate_near_boundary: 0
## s_matrix_singular: 0
## nmoutput2so_version: This SOBlock was created with nmoutput2so version 4.5.11
##
## Completed
## -- Fri Dec 11 02:42:54 2015
Load previous results
NM <- LoadSOObject(“NONMEM/UseCase14.SO.xml”)
The ddmore function “getPopulationParameters” extracts the Population Parameter values from
an R object of (S4) class “StandardOutputObject” and returns the estimates.
See documentation for getPopulationParameters to see other arguments and settings for this function.
Results from NONMEM should be comparable with results from MONOLIX
parameters_nm <- getPopulationParameters(NM, what="estimates")
print(parameters_nm)
## $MLE
## POP_HBASE POP_BTATRT
## 9.8717300 0.0305249
#print(parameters_mlx)
Covariance step cannot be requested in the current version of the Framework
#print(getPopulationParameters(NM, what="precisions"))
print(getEstimationInfo(NM))
## $Likelihood
## $Likelihood$Deviance
## [1] 510.33
##
##
## $Messages
## $Messages$Info
## $Messages$Info$minimization_successful
## [1] "1"
##
## $Messages$Info$covariance_step_run
## [1] "0"
##
## $Messages$Info$rounding_errors
## [1] "0"
##
## $Messages$Info$estimate_near_boundary
## [1] "0"
##
## $Messages$Info$s_matrix_singular
## [1] "0"
##
## $Messages$Info$nmoutput2so_version
## [1] "This SOBlock was created with nmoutput2so version 4.5.11"
Xpose diagnostics using NONMEM output
There is a bug in the generation of xpose database
nm.xpdb<-as.xpdb(NM,datafile)
## Warning in as.data(SOObject, inputDataPath): No Estimation::Predictions
## found in the SO; the resulting data frame will not contain these
## Warning in as.data(SOObject, inputDataPath): No Estimation::Residuals
## found in the SO; the resulting data frame will not contain these
## Warning in as.data(SOObject, inputDataPath): No
## Estimation::IndividualEstimates::RandomEffects::EffectMean found in the
## SO; the resulting data frame will not contain these
Perform some basic goodness of fit. There is no IIV in the model
print(parm.hist(nm.xpdb, inclZeroWRES=TRUE))
MDL Objects can be manipulated from R to change for example the estimation algorithm
myTaskProperties <- getTaskPropertiesObjects(mdlfile)[[1]]
myNewTaskProperties <- myTaskProperties
myNewTaskProperties@ESTIMATE$algo <- "foce"
Assembling the new Modelling Object Group (MOG). Note that we reuse the data, parameters and model from the MOG.
myNewerMOG <- createMogObj(dataObj = getDataObjects(mdlfile)[[1]],
parObj = getParameterObjects(mdlfile)[[1]],
mdlObj = getModelObjects(mdlfile)[[1]],
taskObj = myNewTaskProperties)
We can then write the MOG back out to an .mdl file.
mdlfile.FOCE <- paste0(uc,"_FOCE.mdl")
writeMogObj(myNewerMOG,mdlfile.FOCE)
Test estimation using this new MOG in NONMEM via PsN
NM.FOCE <- estimate(mdlfile.FOCE, target="PsN", subfolder="NONMEM_FOCE")
## -- Fri Dec 11 02:43:09 2015
## New
## Submitted
## Job b6a6defa-433c-416b-b4a5-540c6e1ae762 progress:
## Running [ ... ]
## Importing Results
## Copying the result data back to the local machine for job ID b6a6defa-433c-416b-b4a5-540c6e1ae762...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd02aef46fa to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase14/NONMEM_FOCE
## Done.
##
##
## The following elements were parsed successfully:
## RawResults
## TaskInformation
## Estimation:PopulationEstimates
## Estimation:IndividualEstimates
## Estimation:Likelihood
##
## The following MESSAGEs were raised during the job execution:
## minimization_successful: 1
## covariance_step_run: 0
## rounding_errors: 0
## hessian_reset: 0
## zero_gradients: 0
## final_zero_gradients: 0
## estimate_near_boundary: 0
## s_matrix_singular: 0
## significant_digits: 4.2
## nmoutput2so_version: This SOBlock was created with nmoutput2so version 4.5.11
##
## Completed
## -- Fri Dec 11 02:44:13 2015
The ddmore “LoadSOObj” reads and parsed existing Standardise Output Objects NM.FOCE <- LoadSOObject(“NONMEM_FOCE/UseCase14_FOCE.SO.xml”) Results from NONMEM should be comparable to previous results
print(getPopulationParameters(NM.FOCE, what="estimates")$MLE)
## POP_HBASE POP_BTATRT
## 9.8839000 0.0300856
print(parameters_nm)
## $MLE
## POP_HBASE POP_BTATRT
## 9.8717300 0.0305249
#print(parameters_mlx)
Covariance step cannot be requested in the current version of the Framework
#print(getPopulationParameters(NM, what="precisions"))
nmfoce.xpdb<-as.xpdb(NM.FOCE,datafile)
## Warning in as.data(SOObject, inputDataPath): No Estimation::Predictions
## found in the SO; the resulting data frame will not contain these
## Warning in as.data(SOObject, inputDataPath): No Estimation::Residuals
## found in the SO; the resulting data frame will not contain these
## Warning in as.data(SOObject, inputDataPath): No
## Estimation::IndividualEstimates::RandomEffects::EffectMean found in the
## SO; the resulting data frame will not contain these
Basic diagnostics for NONMEM fit.
print(parm.hist(nmfoce.xpdb, inclZeroWRES=TRUE))
The ddmore “bootstrap.PsN” function is a wrap up function that calls Bootstrap PsN functionality using as input an MDL file that will be translated to NMTRAN as first step. Additional PsN arguments can be specified under the “bootstrapOptions” attribute. After task execution, the output from PsN is converted to a Standardised Output object which is saved in a .SO.xml file. Translated files and PsN output will be returned in the ./Bootstrap subfolder
bootstrapResults <- bootstrap.PsN(mdlfile.FOCE, samples=20, seed=12345,
bootstrapOptions=" -no-skip_minimization_terminated -threads=2",
subfolder="Bootstrap", plot=TRUE)
## -- Fri Dec 11 02:44:18 2015
## New
## Submitted
## Job 485cd642-5dad-45b6-80dc-f96be705ace9 progress:
## Running [ ...... ]
## Importing Results
## Copying the result data back to the local machine for job ID 485cd642-5dad-45b6-80dc-f96be705ace9...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd03b667669 to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase14/Bootstrap
## Done.
##
##
## The following elements were parsed successfully:
## RawResults
## TaskInformation
## Estimation:PopulationEstimates
## Estimation:PrecisionPopulationEstimates
## Estimation:IndividualEstimates
## Estimation:Likelihood
##
## The following MESSAGEs were raised during the job execution:
## minimization_successful: 1
## covariance_step_run: 0
## rounding_errors: 0
## hessian_reset: 0
## zero_gradients: 0
## final_zero_gradients: 0
## estimate_near_boundary: 0
## s_matrix_singular: 0
## significant_digits: 4.2
## nmoutput2so_version: This SOBlock was created with nmoutput2so version 4.5.11
##
## Completed
## -- Fri Dec 11 02:46:22 2015
## [[1]]
##
## [[2]]
## NULL
##
## [[3]]
##
## [[4]]
## NULL
##
## [[5]]
## NULL
Load results from a bootstrap previously performed bootstrapResults <- LoadSOObject(“Bootstrap/UseCase14_FOCE.SO.xml”) Export bootstrap histograms to a pdf
pdf(paste0(uc,"_Bootstrap.pdf"))
print(boot.hist(results.file = file.path("Bootstrap",paste0("raw_results_",uc,"_FOCE.csv")),
incl.ids.file = file.path("Bootstrap","included_individuals1.csv")))
## [[1]]
##
## [[2]]
## NULL
##
## [[3]]
##
## [[4]]
## NULL
##
## [[5]]
## NULL
dev.off()
## rj.GD
## 2
Extract parameter estimates and precision from bootstrap results.
print(getPopulationParameters(bootstrapResults, what="estimates"))
## $MLE
## POP_HBASE POP_BTATRT
## 9.8839000 0.0300856
##
## $Bootstrap
## Parameter Mean Median
## POP_HBASE POP_HBASE 9.96834000 9.87130500
## POP_BTATRT POP_BTATRT 0.04451292 0.00471935
Extract the information regarding the precision intervals
print(getPopulationParameters(bootstrapResults,what="intervals")$Bootstrap)
## Parameter Mean Median Perc_5 Perc_95
## 1 POP_BTATRT 0.04451292 0.00471935 0.004000 0.2523397
## 2 POP_HBASE 9.96834000 9.87130500 6.965948 12.9629100
Simulation of TTE models is not implemented in the current version
Simulx cannot be used for time to event models in the current version