Guided-demo: Exploring 'ddmore' R functionality with the warfarin model

UseCase1 : ODE model for warfarin population pharmacokinetics

This example script is intended to illustrate how to use the 'ddmore' R package to perform a M&S workflow using the DDMoRe Standalone Execution Environment (SEE). It uses UseCase1.mdl - ODE model to describe the warfarin population pharmacokinetics.

The following steps are implemented in this workflow:

To run a task, select wiht the coursor any code lines you wish to execute and press CTRL+R+R in your keyboard. An HTML file containing the commands in this file and associated output will be provided to allow the user to compare the results

Initialisation

Clear workspace and set working directory under 'UsesCasesDemo' project

rm(list=ls(all=F))
mydir <- file.path(Sys.getenv("MDLIDE_WORKSPACE_HOME"),"UseCasesDemo")
setwd(mydir)

Set name of .mdl file and dataset for future tasks

uc <- "UseCase1"
datafile <- "warfarin_conc.csv"
mdlfile <- paste0(uc,".mdl")

Create a new folder to be used as working directory and where results will be stored

wd <- file.path(mydir,uc)
dir.create(wd)

Copy the dataset and the .mdl file available under “models” into the working directory

file.copy(file.path(mydir,"models", datafile),wd)
## [1] TRUE
file.copy(file.path(mydir,"models",mdlfile),wd)
## [1] TRUE

Set the working directory.

setwd(file.path(mydir,uc))

The working directory needs to be set differently when knitr R package is used to spin the file

library(knitr)
opts_knit$set(root.dir = file.path(mydir,uc)) 

List files available in working directory

list.files()
## [1] "UseCase1.mdl"      "warfarin_conc.csv"

Introduction to 'ddmore' R package

View objects within the .mdl file

Use 'ddmore' function getMDLObjects() to retrieve model object(s) from an existing .mdl file. This function reads the MDL in an .mdl file and parses the MDL code for each MDL Object into an R list of objects of appropriate types with names corresponding to the MDL Object names given in the file.

myMDLObj <- getMDLObjects(mdlfile)
length(myMDLObj)
## [1] 4
names(myMDLObj)
## [1] "warfarin_PK_ODE_dat"  "warfarin_PK_ODE_par"  "warfarin_PK_ODE_mdl" 
## [4] "warfarin_PK_ODE_task"
str(myMDLObj)
## List of 4
##  $ warfarin_PK_ODE_dat :Formal class 'dataObj' [package "ddmore"] with 5 slots
##   .. ..@ SOURCE                :List of 1
##   .. .. ..$ srcfile:List of 2
##   .. .. .. ..$ file       : chr "warfarin_conc.csv"
##   .. .. .. ..$ inputFormat: chr "nonmemFormat"
##   .. ..@ DECLARED_VARIABLES    :List of 2
##   .. .. ..$ :List of 2
##   .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. ..$ name    : chr "GUT"
##   .. .. ..$ :List of 2
##   .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. ..$ name    : chr "Y"
##   .. ..@ DATA_INPUT_VARIABLES  :List of 8
##   .. .. ..$ ID    :List of 1
##   .. .. .. ..$ use: chr "id"
##   .. .. ..$ TIME  :List of 1
##   .. .. .. ..$ use: chr "idv"
##   .. .. ..$ WT    :List of 1
##   .. .. .. ..$ use: chr "covariate"
##   .. .. ..$ AMT   :List of 2
##   .. .. .. ..$ use     : chr "amt"
##   .. .. .. ..$ variable: chr "GUT"
##   .. .. ..$ DVID  :List of 1
##   .. .. .. ..$ use: chr "dvid"
##   .. .. ..$ DV    :List of 2
##   .. .. .. ..$ use     : chr "dv"
##   .. .. .. ..$ variable: chr "Y"
##   .. .. ..$ MDV   :List of 1
##   .. .. .. ..$ use: chr "mdv"
##   .. .. ..$ logtWT:List of 1
##   .. .. .. ..$ use: chr "covariate"
##   .. ..@ DATA_DERIVED_VARIABLES: list()
##   .. ..@ name                  : chr "warfarin_PK_ODE_dat"
##  $ warfarin_PK_ODE_par :Formal class 'parObj' [package "ddmore"] with 4 slots
##   .. ..@ DECLARED_VARIABLES:List of 2
##   .. .. ..$ :List of 2
##   .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. ..$ name    : chr "ETA_CL"
##   .. .. ..$ :List of 2
##   .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. ..$ name    : chr "ETA_V"
##   .. ..@ STRUCTURAL        :List of 8
##   .. .. ..$ POP_CL    :List of 2
##   .. .. .. ..$ value: chr "0.1"
##   .. .. .. ..$ lo   : chr "0.001"
##   .. .. ..$ POP_V     :List of 2
##   .. .. .. ..$ value: chr "8"
##   .. .. .. ..$ lo   : chr "0.001"
##   .. .. ..$ POP_KA    :List of 2
##   .. .. .. ..$ value: chr "0.362"
##   .. .. .. ..$ lo   : chr "0.001"
##   .. .. ..$ POP_TLAG  :List of 2
##   .. .. .. ..$ value: chr "1"
##   .. .. .. ..$ lo   : chr "0.001"
##   .. .. ..$ BETA_CL_WT:List of 2
##   .. .. .. ..$ value: chr "0.75"
##   .. .. .. ..$ fix  : chr "true"
##   .. .. ..$ BETA_V_WT :List of 2
##   .. .. .. ..$ value: chr "1"
##   .. .. .. ..$ fix  : chr "true"
##   .. .. ..$ RUV_PROP  :List of 2
##   .. .. .. ..$ value: chr "0.1"
##   .. .. .. ..$ lo   : chr "0.001"
##   .. .. ..$ RUV_ADD   :List of 2
##   .. .. .. ..$ value: chr "0.1"
##   .. .. .. ..$ lo   : chr "0.001"
##   .. ..@ VARIABILITY       :List of 5
##   .. .. ..$ PPV_CL  :List of 2
##   .. .. .. ..$ value: chr "0.1"
##   .. .. .. ..$ type : chr "sd"
##   .. .. ..$ PPV_V   :List of 2
##   .. .. .. ..$ value: chr "0.1"
##   .. .. .. ..$ type : chr "sd"
##   .. .. ..$ PPV_KA  :List of 2
##   .. .. .. ..$ value: chr "0.1"
##   .. .. .. ..$ type : chr "sd"
##   .. .. ..$ PPV_TLAG:List of 3
##   .. .. .. ..$ value: chr "0.1"
##   .. .. .. ..$ type : chr "sd"
##   .. .. .. ..$ fix  : chr "true"
##   .. .. ..$ OMEGA   :List of 3
##   .. .. .. ..$ parameter: chr "[ETA_CL,ETA_V]"
##   .. .. .. ..$ value    : chr "[0.01]"
##   .. .. .. ..$ type     : chr "corr"
##   .. ..@ name              : chr "warfarin_PK_ODE_par"
##  $ warfarin_PK_ODE_mdl :Formal class 'mdlObj' [package "ddmore"] with 11 slots
##   .. ..@ IDV                       : chr "T"
##   .. ..@ COVARIATES                :List of 1
##   .. .. ..$ :List of 2
##   .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. ..$ name    : chr "logtWT"
##   .. ..@ VARIABILITY_LEVELS        :List of 2
##   .. .. ..$ ID:List of 2
##   .. .. .. ..$ level: chr "2"
##   .. .. .. ..$ type : chr "parameter"
##   .. .. ..$ DV:List of 2
##   .. .. .. ..$ level: chr "1"
##   .. .. .. ..$ type : chr "observation"
##   .. ..@ STRUCTURAL_PARAMETERS     : chr [1:8] "POP_CL" "POP_V" "POP_KA" "POP_TLAG" ...
##   .. ..@ VARIABILITY_PARAMETERS    : chr [1:4] "PPV_CL" "PPV_V" "PPV_KA" "PPV_TLAG"
##   .. ..@ RANDOM_VARIABLE_DEFINITION:List of 5
##   .. .. ..$ :List of 5
##   .. .. .. ..$ .subtype: chr "RandVarDefn"
##   .. .. .. ..$ blkAttrs:List of 1
##   .. .. .. .. ..$ level: chr "DV"
##   .. .. .. ..$ args    :List of 2
##   .. .. .. .. ..$ mean: chr "0"
##   .. .. .. .. ..$ var : chr "1"
##   .. .. .. ..$ name    : chr "EPS_Y"
##   .. .. .. ..$ distType: chr "Normal"
##   .. .. ..$ :List of 5
##   .. .. .. ..$ .subtype: chr "RandVarDefn"
##   .. .. .. ..$ blkAttrs:List of 1
##   .. .. .. .. ..$ level: chr "ID"
##   .. .. .. ..$ args    :List of 2
##   .. .. .. .. ..$ mean: chr "0"
##   .. .. .. .. ..$ sd  : chr "PPV_CL"
##   .. .. .. ..$ name    : chr "ETA_CL"
##   .. .. .. ..$ distType: chr "Normal"
##   .. .. ..$ :List of 5
##   .. .. .. ..$ .subtype: chr "RandVarDefn"
##   .. .. .. ..$ blkAttrs:List of 1
##   .. .. .. .. ..$ level: chr "ID"
##   .. .. .. ..$ args    :List of 2
##   .. .. .. .. ..$ mean: chr "0"
##   .. .. .. .. ..$ sd  : chr "PPV_V"
##   .. .. .. ..$ name    : chr "ETA_V"
##   .. .. .. ..$ distType: chr "Normal"
##   .. .. ..$ :List of 5
##   .. .. .. ..$ .subtype: chr "RandVarDefn"
##   .. .. .. ..$ blkAttrs:List of 1
##   .. .. .. .. ..$ level: chr "ID"
##   .. .. .. ..$ args    :List of 2
##   .. .. .. .. ..$ mean: chr "0"
##   .. .. .. .. ..$ sd  : chr "PPV_KA"
##   .. .. .. ..$ name    : chr "ETA_KA"
##   .. .. .. ..$ distType: chr "Normal"
##   .. .. ..$ :List of 5
##   .. .. .. ..$ .subtype: chr "RandVarDefn"
##   .. .. .. ..$ blkAttrs:List of 1
##   .. .. .. .. ..$ level: chr "ID"
##   .. .. .. ..$ args    :List of 2
##   .. .. .. .. ..$ mean: chr "0"
##   .. .. .. .. ..$ sd  : chr "PPV_TLAG"
##   .. .. .. ..$ name    : chr "ETA_TLAG"
##   .. .. .. ..$ distType: chr "Normal"
##   .. ..@ INDIVIDUAL_VARIABLES      :List of 4
##   .. .. ..$ :List of 5
##   .. .. .. ..$ transform: chr "ln"
##   .. .. .. ..$ .subtype : chr "TransDefn"
##   .. .. .. ..$ args     :List of 4
##   .. .. .. .. ..$ trans : chr "ln"
##   .. .. .. .. ..$ pop   : chr "POP_CL"
##   .. .. .. .. ..$ fixEff: chr "[{coeff=BETA_CL_WT, cov=logtWT}]"
##   .. .. .. .. ..$ ranEff: chr "[ETA_CL]"
##   .. .. .. ..$ name     : chr "CL"
##   .. .. .. ..$ func     : chr "linear"
##   .. .. ..$ :List of 5
##   .. .. .. ..$ transform: chr "ln"
##   .. .. .. ..$ .subtype : chr "TransDefn"
##   .. .. .. ..$ args     :List of 4
##   .. .. .. .. ..$ trans : chr "ln"
##   .. .. .. .. ..$ pop   : chr "POP_V"
##   .. .. .. .. ..$ fixEff: chr "[{coeff=BETA_V_WT, cov=logtWT}]"
##   .. .. .. .. ..$ ranEff: chr "[ETA_V]"
##   .. .. .. ..$ name     : chr "V"
##   .. .. .. ..$ func     : chr "linear"
##   .. .. ..$ :List of 5
##   .. .. .. ..$ transform: chr "ln"
##   .. .. .. ..$ .subtype : chr "TransDefn"
##   .. .. .. ..$ args     :List of 3
##   .. .. .. .. ..$ trans : chr "ln"
##   .. .. .. .. ..$ pop   : chr "POP_KA"
##   .. .. .. .. ..$ ranEff: chr "[ETA_KA]"
##   .. .. .. ..$ name     : chr "KA"
##   .. .. .. ..$ func     : chr "linear"
##   .. .. ..$ :List of 5
##   .. .. .. ..$ transform: chr "ln"
##   .. .. .. ..$ .subtype : chr "TransDefn"
##   .. .. .. ..$ args     :List of 3
##   .. .. .. .. ..$ trans : chr "ln"
##   .. .. .. .. ..$ pop   : chr "POP_TLAG"
##   .. .. .. .. ..$ ranEff: chr "[ETA_TLAG]"
##   .. .. .. ..$ name     : chr "TLAG"
##   .. .. .. ..$ func     : chr "linear"
##   .. ..@ MODEL_PREDICTION          :List of 2
##   .. .. ..$ :List of 2
##   .. .. .. ..$ DEQ     :List of 3
##   .. .. .. .. ..$ :List of 3
##   .. .. .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. .. .. ..$ name    : chr "RATEIN"
##   .. .. .. .. .. ..$ expr    : chr "if (T>=TLAG) then GUT*KA else 0"
##   .. .. .. .. ..$ :List of 1
##   .. .. .. .. .. ..$ GUT:List of 3
##   .. .. .. .. .. .. ..$ deriv: chr "(-RATEIN)"
##   .. .. .. .. .. .. ..$ init : chr "0"
##   .. .. .. .. .. .. ..$ x0   : chr "0"
##   .. .. .. .. ..$ :List of 1
##   .. .. .. .. .. ..$ CENTRAL:List of 3
##   .. .. .. .. .. .. ..$ deriv: chr "(RATEIN-CL*CENTRAL/V)"
##   .. .. .. .. .. .. ..$ init : chr "0"
##   .. .. .. .. .. .. ..$ x0   : chr "0"
##   .. .. .. ..$ .subtype: chr "BlockStmt"
##   .. .. ..$ :List of 3
##   .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. ..$ name    : chr "CC"
##   .. .. .. ..$ expr    : chr "CENTRAL/V"
##   .. ..@ OBSERVATION               :List of 1
##   .. .. ..$ :List of 4
##   .. .. .. ..$ .subtype: chr "EquationDef"
##   .. .. .. ..$ funcArgs:List of 4
##   .. .. .. .. ..$ additive    : chr "RUV_ADD"
##   .. .. .. .. ..$ proportional: chr "RUV_PROP"
##   .. .. .. .. ..$ eps         : chr "EPS_Y"
##   .. .. .. .. ..$ prediction  : chr "CC"
##   .. .. .. ..$ funcName: chr "combinedError1"
##   .. .. .. ..$ name    : chr "Y"
##   .. ..@ GROUP_VARIABLES           : list()
##   .. ..@ name                      : chr "warfarin_PK_ODE_mdl"
##  $ warfarin_PK_ODE_task:Formal class 'taskObj' [package "ddmore"] with 3 slots
##   .. ..@ ESTIMATE:List of 1
##   .. .. ..$ algo: chr "saem"
##   .. ..@ SIMULATE: list()
##   .. ..@ name    : chr "warfarin_PK_ODE_task"

Use 'ddmore' function getDataObjects() to retrieve only data object(s) from an existing .mdl file. This function returns a list of Parameter Object(s) from which we select the first element Hover over the variable name to see its structure

myDataObj <- getDataObjects(mdlfile)[[1]]

Use 'ddmore' function getParameterObjects() to retrieve only parameter object(s) from an existing .mdl file

myParObj <- getParameterObjects(mdlfile)[[1]]

Use 'ddmore' function getModelObjects() to retrieve only model object(s) from an existing .mdl file.

myModObj <- getModelObjects(mdlfile)[[1]]

Use 'ddmore' function getTaskPropertiesObjects() to retrieve only task properties object(s) from an existing .mdl file

myTaskObj <- getTaskPropertiesObjects(mdlfile)[[1]]

Exploratory Data Analysis

Recall that getDataObjects only reads the MDL code from the .mdl file. Use 'ddmore' function readDataObj() to create an R object from the MDL data object.

myData <- readDataObj(myDataObj)

Let's look at the first 6 lines of the data set

head(myData)
##   ID TIME   WT AMT DVID  DV MDV      logtWT
## 1  1  0.0 66.7 100    0  NA   1 -0.04829029
## 2  1  0.5 66.7  NA    1 0.0   0 -0.04829029
## 3  1  1.0 66.7  NA    1 1.9   0 -0.04829029
## 4  1  2.0 66.7  NA    1 3.3   0 -0.04829029
## 5  1  3.0 66.7  NA    1 6.6   0 -0.04829029
## 6  1  6.0 66.7  NA    1 9.1   0 -0.04829029

Extract only observation records

myEDAData<-myData[myData$MDV==0,]

Open an R window to record and access all your plots

windows(record=TRUE) 

Plot the data using xyplot from the lattice library

plot1 <- xyplot(DV~TIME,groups=ID,data=myEDAData,type="b",ylab="Conc. (mg/L)",xlab="Time (h)")
print(plot1)

plot of chunk unnamed-chunk-14

plot2 <- xyplot(DV~TIME|ID,data=myEDAData,type="b",layout=c(3,4),ylab="Conc. (mg/L)",xlab="Time (h)",scales=list(relation="free"))
print(plot2)

plot of chunk unnamed-chunk-14 plot of chunk unnamed-chunk-14 plot of chunk unnamed-chunk-14

Export the results in a pdf file

pdf(paste0(uc,"_EGA.pdf"))
 print(plot1)
 print(plot2)
dev.off()
## rj.GD 
##     2

Model Development

ESTIMATE model parameters using Monolix

The ddmore “estimate” function translates the contents of the .mdl file to a target language and then estimates parameters using the target software. After estimation, the output is converted to a Standardised Output object which is saved in a .SO.xml file.

Translated files and Monolix output will be returned in the ./Monolix subfolder. The Standardised Output object (.SO.xml) is read and parsed into an R object called “mlx” of (S4) class “StandardOutputObject”.

mlx <- estimate(mdlfile, target="MONOLIX", subfolder="Monolix")
## -- Fri Dec 11 03:04:27 2015
## New
## Submitted
## Job c567eda1-e36e-4be5-b5d3-05dae7f7b066 progress:
## Running [ ....... ]
## Importing Results
## Copying the result data back to the local machine for job ID c567eda1-e36e-4be5-b5d3-05dae7f7b066...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd031075b6d to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase1/Monolix
## Done.
## 
## 
## The following elements were parsed successfully:
##       ToolSettings
##       RawResults
##       TaskInformation
##       Estimation:PopulationEstimates
##       Estimation:PrecisionPopulationEstimates
##       Estimation:IndividualEstimates
##       Estimation:Residuals
##       Estimation:Predictions
##       Estimation:Likelihood
## 
## Completed
## -- Fri Dec 11 03:06:51 2015
slotNames(mlx)
## [1] "ToolSettings"     "RawResults"       "TaskInformation" 
## [4] "Estimation"       "ModelDiagnostic"  "Simulation"      
## [7] "OptimalDesign"    ".pathToSourceXML"

The ddmore “LoadSOObj” reads and parsed existing Standardise Output Objects mlx <- LoadSOObject(“Monolix/UseCase1.SO.xml”) The ddmore function “getPopulationParameters” extracts the Population Parameter values from an R object of (S4) class “StandardOutputObject” and returns the estimates. See documentation for getPopulationParameters to see other arguments and settings for this function.

parameters_mlx <- getPopulationParameters(mlx, what="estimates")$MLE
parameters_mlx
##        POP_V    BETA_V_WT       POP_KA       POP_CL   BETA_CL_WT 
##      8.04978      1.00000      1.54310      0.13478      0.75000 
##     POP_TLAG        PPV_V       PPV_KA       PPV_CL     PPV_TLAG 
##      0.93563      0.13511      1.06495      0.26221      0.01000 
## ETA_V_ETA_CL      RUV_ADD     RUV_PROP 
##      0.23644      0.36791      0.05446
getPopulationParameters(mlx, what="precisions")
## $MLE
##       Parameter     MLE      SE   RSE
## 1    BETA_CL_WT 0.75000 0.00000  0.00
## 2     BETA_V_WT 1.00000 0.00000  0.00
## 3  ETA_V_ETA_CL 0.23644 0.20720 87.63
## 4        POP_CL 0.13478 0.00646  4.79
## 5        POP_KA 1.54310 0.48896 31.69
## 6      POP_TLAG 0.93563 0.00720  0.77
## 7         POP_V 8.04978 0.22780  2.83
## 8        PPV_CL 0.26221 0.03493 13.32
## 9        PPV_KA 1.06495 0.23580 22.14
## 10     PPV_TLAG 0.01000 0.00000  0.00
## 11        PPV_V 0.13511 0.02336 17.29
## 12      RUV_ADD 0.36791 0.05769 15.68
## 13     RUV_PROP 0.05446 0.01058 19.42
getEstimationInfo(mlx)
## $Likelihood
## $Likelihood$LogLikelihood
## [1] -350.115
## 
## $Likelihood$IndividualContribToLL
##    Subject ICtoLL
## 1        1 -24.99
## 2        2  -5.73
## 3        3 -13.49
## 4        4 -12.56
## 5        5 -11.56
## 6        6  -8.38
## 7        7 -17.03
## 8        8 -22.05
## 9        9 -29.85
## 10      10  -6.29
## 11      12 -19.45
## 12      13 -22.84
## 13      14 -20.96
## 14      15 -12.02
## 15      16 -15.29
## 16      17  -7.27
## 17      18  -5.76
## 18      19  -6.41
## 19      20  -5.76
## 20      21  -6.85
## 21      22  -6.18
## 22      23  -7.95
## 23      24  -5.43
## 24      25  -7.77
## 25      26  -7.45
## 26      27  -5.81
## 27      28  -7.11
## 28      29  -5.98
## 29      30  -5.29
## 30      31  -5.64
## 31      32  -5.29
## 32      33  -5.67
## 
## $Likelihood$InformationCriteria
## $Likelihood$InformationCriteria$AIC
## [1] 720.23
## 
## $Likelihood$InformationCriteria$BIC
## [1] 734.89
## 
## 
## 
## $Messages
## list()

Perform model diagnostics for the base model using Xpose functions

Use 'ddmore' function as.xpdb() to create an Xpose database object from the Standardised Output object, regardless of target software used for estimation.

mlx.xpdb<-as.xpdb(mlx,datafile)
## 
## Removed dose rows in rawData slot of SO to enable merge with Predictions data.

We can then call Xpose functions referencing this mlx.xpdb object as the input. Perform some basic goodness of fit

print(basic.gof(mlx.xpdb))

plot of chunk unnamed-chunk-17

print(ind.plots(mlx.xpdb))

plot of chunk unnamed-chunk-17 plot of chunk unnamed-chunk-17

Export graphs to PDF file

pdf("GOF_MLX.pdf")
 print(basic.gof(mlx.xpdb))
 print(ind.plots(mlx.xpdb))
dev.off()
## rj.GD 
##     2

SAEM Estimation with NONMEM

NM <- estimate(mdlfile, target="NONMEM", subfolder="NONMEM")
## -- Fri Dec 11 03:07:06 2015
## New
## Submitted
## Job 141d7b4b-be6c-4b59-a9d9-66ceda92604a progress:
## Running [ ........................... ]
## Importing Results
## Copying the result data back to the local machine for job ID 141d7b4b-be6c-4b59-a9d9-66ceda92604a...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd01425e55 to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase1/NONMEM
## Done.
## 
## 
## The following elements were parsed successfully:
##       RawResults
##       TaskInformation
##       Estimation:PopulationEstimates
##       Estimation:IndividualEstimates
##       Estimation:Residuals
##       Estimation:Predictions
##       Estimation:Likelihood
## 
## The following MESSAGEs were raised during the job execution:
##  minimization_successful: 1
##  covariance_step_run: 0
##  rounding_errors: 0
##  estimate_near_boundary: 0
##  s_matrix_singular: 0
##  nmoutput2so_version: This SOBlock was created with nmoutput2so version 4.5.11
## 
## Completed
## -- Fri Dec 11 03:16:11 2015

Load previous results NM <- LoadSOObject(“NONMEM/UseCase1.SO.xml”)

Results from NONMEM should be comparable with results from MONOLIX

parameters_nm <- getPopulationParameters(NM, what="estimates")
print(parameters_nm)
## $MLE
##       POP_CL        POP_V       POP_KA     POP_TLAG     RUV_PROP 
##   0.13300900   8.16777000   1.53003000   0.90651600   0.11039500 
##      RUV_ADD       PPV_CL ETA_CL_ETA_V        PPV_V       PPV_KA 
##   0.00100001   0.26241600   0.26605600   0.13633700   0.90796000 
##   BETA_CL_WT    BETA_V_WT     PPV_TLAG 
##   0.75000000   1.00000000   0.10000000
print(parameters_mlx)
##        POP_V    BETA_V_WT       POP_KA       POP_CL   BETA_CL_WT 
##      8.04978      1.00000      1.54310      0.13478      0.75000 
##     POP_TLAG        PPV_V       PPV_KA       PPV_CL     PPV_TLAG 
##      0.93563      0.13511      1.06495      0.26221      0.01000 
## ETA_V_ETA_CL      RUV_ADD     RUV_PROP 
##      0.23644      0.36791      0.05446

Covariance step cannot be requested in the current version of the Framework

#getPopulationParameters(NM, what="precisions") 

print(getEstimationInfo(NM))
## $Likelihood
## $Likelihood$Deviance
## [1] -291.5739
## 
## 
## $Messages
## $Messages$Info
## $Messages$Info$minimization_successful
## [1] "1"
## 
## $Messages$Info$covariance_step_run
## [1] "0"
## 
## $Messages$Info$rounding_errors
## [1] "0"
## 
## $Messages$Info$estimate_near_boundary
## [1] "0"
## 
## $Messages$Info$s_matrix_singular
## [1] "0"
## 
## $Messages$Info$nmoutput2so_version
## [1] "This SOBlock was created with nmoutput2so version 4.5.11"

Xpose diagnostics using NONMEM output

nm.xpdb<-as.xpdb(NM,datafile)
## 
## Removed dose rows in rawData+Predictions slot of SO to enable merge with Residuals data.
## 
## Residuals data does not currently contain dose rows in output from Nonmem executions.

Perform some basic goofness of fit

print(basic.gof(nm.xpdb))

plot of chunk unnamed-chunk-21

print(ind.plots(nm.xpdb))

plot of chunk unnamed-chunk-21 plot of chunk unnamed-chunk-21

print(parm.hist(nm.xpdb))

plot of chunk unnamed-chunk-21

Export plots to a pdf file

pdf("GOF_NM.pdf")
 print(basic.gof(nm.xpdb))
 print(ind.plots(nm.xpdb))
 print(parm.hist(nm.xpdb))
dev.off()
## rj.GD 
##     2

Change estimation method to FOCEI (for speed)

MDL Objects can be manipulated from R to change for example the estimation algorithm

myTaskProperties <- getTaskPropertiesObjects(mdlfile)[[1]]
myNewTaskProperties <- myTaskProperties
myNewTaskProperties@ESTIMATE$algo <- "focei"

Assembling the new Modelling Object Group (MOG). Note that we reuse the data, parameters and model from the MOG.

myNewerMOG <- createMogObj(dataObj = getDataObjects(mdlfile)[[1]], 
        parObj = getParameterObjects(mdlfile)[[1]], 
        mdlObj = getModelObjects(mdlfile)[[1]], 
        taskObj = myNewTaskProperties)

We can then write the MOG back out to an .mdl file.

mdlfile.FOCEI <- paste0(uc,"_FOCEI.mdl")
writeMogObj(myNewerMOG,mdlfile.FOCEI)

Test estimation using this new MOG in NONMEM via PsN

NM.FOCEI <- estimate(mdlfile.FOCEI, target="PsN", subfolder="NONMEM_FOCEI")
## -- Fri Dec 11 03:16:47 2015
## New
## Submitted
## Job 662ff4c8-88c2-4ceb-96c0-79bf6393b6f2 progress:
## Running [ .......... ]
## Importing Results
## Copying the result data back to the local machine for job ID 662ff4c8-88c2-4ceb-96c0-79bf6393b6f2...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd0656d6bc8 to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase1/NONMEM_FOCEI
## Done.
## 
## 
## The following elements were parsed successfully:
##       RawResults
##       TaskInformation
##       Estimation:PopulationEstimates
##       Estimation:IndividualEstimates
##       Estimation:Residuals
##       Estimation:Predictions
##       Estimation:Likelihood
## 
## The following MESSAGEs were raised during the job execution:
##  minimization_successful: 1
##  covariance_step_run: 0
##  rounding_errors: 0
##  hessian_reset: 0
##  zero_gradients: 0
##  final_zero_gradients: 0
##  estimate_near_boundary: 0
##  s_matrix_singular: 0
##  significant_digits: 3.4
##  nmoutput2so_version: This SOBlock was created with nmoutput2so version 4.5.11
## 
## Completed
## -- Fri Dec 11 03:20:12 2015

Load previous results NM.FOCEI <- LoadSOObject(“NONMEM_FOCEI/UseCase1_FOCEI.SO.xml”) Results from NONMEM should be comparable to previous results

getPopulationParameters(NM.FOCEI,  what="estimates")
## $MLE
##       POP_CL        POP_V       POP_KA     POP_TLAG     RUV_PROP 
##    0.1346550    8.0464900    1.2941000    0.8633020    0.0626403 
##      RUV_ADD       PPV_CL ETA_CL_ETA_V        PPV_V       PPV_KA 
##    0.3155580    0.2636160    0.2136520    0.1366600    0.9183900 
##   BETA_CL_WT    BETA_V_WT     PPV_TLAG 
##    0.7500000    1.0000000    0.1000000
parameters_nm
## $MLE
##       POP_CL        POP_V       POP_KA     POP_TLAG     RUV_PROP 
##   0.13300900   8.16777000   1.53003000   0.90651600   0.11039500 
##      RUV_ADD       PPV_CL ETA_CL_ETA_V        PPV_V       PPV_KA 
##   0.00100001   0.26241600   0.26605600   0.13633700   0.90796000 
##   BETA_CL_WT    BETA_V_WT     PPV_TLAG 
##   0.75000000   1.00000000   0.10000000

Xpose diagnostics using NONMEM output

nmfocei.xpdb<-as.xpdb(NM.FOCEI,datafile)
## 
## Removed dose rows in rawData+Predictions slot of SO to enable merge with Residuals data.
## 
## Residuals data does not currently contain dose rows in output from Nonmem executions.

Basic diagnostics for NONMEM fit.

print(basic.gof(nmfocei.xpdb))

plot of chunk unnamed-chunk-26

Export graph to a pdf

pdf("GOF_NM_FOCEI.pdf")
 print(basic.gof(nmfocei.xpdb))
dev.off()
## rj.GD 
##     2

Run the Bootstrap using PsN

The ddmore “bootstrap.PsN” function is a wrap up function that calls Bootstrap PsN functionality using as input an MDL file that will be translated to NMTRAN as first step. Additional PsN arguments can be specified under the “bootstrapOptions” attribute. After task execution, the output from PsN is converted to a Standardised Output object which is saved in a .SO.xml file. Translated files and PsN output will be returned in the ./Bootstrap subfolder

bootstrapResults <- bootstrap.PsN(mdlfile.FOCEI, samples=20, seed=123456,
        bootstrapOptions=" -no-skip_minimization_terminated -threads=2",
        subfolder="Bootstrap", plot=TRUE)
## -- Fri Dec 11 03:20:23 2015
## New
## Submitted
## Job 1920f1b1-426d-4d6b-8bca-42aaad9b2699 progress:
## Running [ ................................................................................. ]
## Importing Results
## Copying the result data back to the local machine for job ID 1920f1b1-426d-4d6b-8bca-42aaad9b2699...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd06a2746a2 to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase1/Bootstrap
## Done.
## 
## 
## The following elements were parsed successfully:
##       RawResults
##       TaskInformation
##       Estimation:PopulationEstimates
##       Estimation:PrecisionPopulationEstimates
##       Estimation:IndividualEstimates
##       Estimation:Residuals
##       Estimation:Predictions
##       Estimation:Likelihood
## 
## The following WARNINGs were raised during the job execution:
##  bootstrap_parameter_scale: The parameters PPV_CL, ETA_CL_ETA_V, PPV_V, PPV_KA and PPV_TLAG were requested on the sd/corr scale but are given on the var/cov scale in all bootstrap results.
## 
## The following MESSAGEs were raised during the job execution:
##  minimization_successful: 1
##  covariance_step_run: 0
##  rounding_errors: 0
##  hessian_reset: 0
##  zero_gradients: 0
##  final_zero_gradients: 0
##  estimate_near_boundary: 0
##  s_matrix_singular: 0
##  significant_digits: 3.4
##  nmoutput2so_version: This SOBlock was created with nmoutput2so version 4.5.11
## 
## Completed
## -- Fri Dec 11 03:47:30 2015
## Warning: NAs introduced by coercion
## [[1]]

plot of chunk Bootstrap

## 
## [[2]]
## NULL
## 
## [[3]]

plot of chunk Bootstrap

## 
## [[4]]
## NULL
## 
## [[5]]
## NULL

Load results from a bootstrap previously performed bootstrapResults <- LoadSOObject(“Bootstrap/UseCase1_FOCEI.SO.xml”) Export bootstrap histograms to a pdf

pdf(paste0(uc,"_Bootstrap.pdf"))
 print(boot.hist(results.file = file.path("Bootstrap",paste0("raw_results_",uc,"_FOCEI.csv")),
                incl.ids.file = file.path("Bootstrap","included_individuals1.csv")))
## [[1]]
## 
## [[2]]
## NULL
## 
## [[3]]
## 
## [[4]]
## NULL
## 
## [[5]]
## NULL
dev.off()
## rj.GD 
##     2

Extract parameter estimates and precision from bootstrap results.

print(getPopulationParameters(bootstrapResults, what="estimates"))
## $MLE
##       POP_CL        POP_V       POP_KA     POP_TLAG     RUV_PROP 
##    0.1346550    8.0464900    1.2941000    0.8633020    0.0626403 
##      RUV_ADD       PPV_CL ETA_CL_ETA_V        PPV_V       PPV_KA 
##    0.3155580    0.2636160    0.2136520    0.1366600    0.9183900 
##   BETA_CL_WT    BETA_V_WT     PPV_TLAG 
##    0.7500000    1.0000000    0.1000000 
## 
## $Bootstrap
##                 Parameter        Mean      Median
## POP_CL             POP_CL 0.135671700 0.136903500
## POP_V               POP_V 8.048746000 7.971910000
## POP_KA             POP_KA 1.348676000 1.245350000
## POP_TLAG         POP_TLAG 0.844600300 0.862184500
## RUV_PROP         RUV_PROP 0.093457160 0.085709550
## RUV_ADD           RUV_ADD 0.189159800 0.202150500
## BETA_CL_WT     BETA_CL_WT 0.750000000 0.750000000
## BETA_V_WT       BETA_V_WT 1.000000000 1.000000000
## PPV_CL             PPV_CL 0.070646390 0.070932100
## ETA_CL_ETA_V ETA_CL_ETA_V 0.006351653 0.007229185
## PPV_V               PPV_V 0.014731450 0.015570800
## PPV_KA             PPV_KA 0.710713800 0.655655500
## PPV_TLAG         PPV_TLAG 0.010000000 0.010000000

Extract the information regarding the precision intervals

print(getPopulationParameters(bootstrapResults, what="intervals")$Bootstrap)
##       Parameter        Mean      Median       Perc_5    Perc_95
## 1    BETA_CL_WT 0.750000000 0.750000000  0.750000000 0.75000000
## 2     BETA_V_WT 1.000000000 1.000000000  1.000000000 1.00000000
## 3  ETA_CL_ETA_V 0.006351653 0.007229185 -0.006357748 0.01609685
## 4        POP_CL 0.135671700 0.136903500  0.123908600 0.14836850
## 5        POP_KA 1.348676000 1.245350000  0.702492100 2.39337800
## 6      POP_TLAG 0.844600300 0.862184500  0.550804400 0.96143250
## 7         POP_V 8.048746000 7.971910000  7.704302000 8.55141000
## 8        PPV_CL 0.070646390 0.070932100  0.027702590 0.12466510
## 9        PPV_KA 0.710713800 0.655655500  0.185926400 1.38019500
## 10     PPV_TLAG 0.010000000 0.010000000  0.010000000 0.01000000
## 11        PPV_V 0.014731450 0.015570800  0.002586434 0.02593207
## 12      RUV_ADD 0.189159800 0.202150500  0.001990000 0.42164610
## 13     RUV_PROP 0.093457160 0.085709550  0.037336740 0.15733780

VPC of model

Before running the VPC with PsN we must update the (initial) values in the MDL Parameter Object MLE estimates from previous step can be used

structuralPar <- getPopulationParameters(NM.FOCEI, what="estimates",block='structural')$MLE
variabilityPar <- getPopulationParameters(NM.FOCEI, what="estimates",block='variability')$MLE

In the current version of the SO standard, we need to manually update parameter names for correlation and covariance parameters to match the SO with the MDL. This will not be needed in future releases. The SO object returned from NONMEM has parameter ETA_CL_ETA_V. This needs to be renamed to conform to model Correlation name OMEGA

variabilityNames <- names(myParObj@VARIABILITY)
names(variabilityPar)[names(variabilityPar)=="ETA_CL_ETA_V"] <- grep("OMEGA",variabilityNames,value=T)
names(variabilityPar)
## [1] "PPV_CL"   "OMEGA"    "PPV_V"    "PPV_KA"   "PPV_TLAG"

Update the parameter object using the ddmore “updateParObj” function. This function updates an R object of (S4) class “parObj”. The user chooses which block to update, what items within that block, and what to replace those items with. NOTE: that updateParObj can only update attributes which ALREADY EXIST in the MDL Parameter Object for that item. This ensures that valid MDL is preserved.

myParObj <- getParameterObjects(mdlfile)[[1]]
myParObjUpdated <- updateParObj(myParObj,block="STRUCTURAL",
        item=names(structuralPar),
        with=list(value=structuralPar))
myParObjUpdated <- updateParObj(myParObjUpdated,block="VARIABILITY",
        item=names(variabilityPar),
        with=list(value=variabilityPar))

Check that the appropriate initial values have been updated to the MLE values from the previous fit.

print(myParObjUpdated@STRUCTURAL)
## $POP_CL
## $POP_CL$value
## [1] "0.134655"
## 
## $POP_CL$lo
## [1] "0.001"
## 
## 
## $POP_V
## $POP_V$value
## [1] "8.04649"
## 
## $POP_V$lo
## [1] "0.001"
## 
## 
## $POP_KA
## $POP_KA$value
## [1] "1.2941"
## 
## $POP_KA$lo
## [1] "0.001"
## 
## 
## $POP_TLAG
## $POP_TLAG$value
## [1] "0.863302"
## 
## $POP_TLAG$lo
## [1] "0.001"
## 
## 
## $BETA_CL_WT
## $BETA_CL_WT$value
## [1] "0.75"
## 
## $BETA_CL_WT$fix
## [1] "true"
## 
## 
## $BETA_V_WT
## $BETA_V_WT$value
## [1] "1"
## 
## $BETA_V_WT$fix
## [1] "true"
## 
## 
## $RUV_PROP
## $RUV_PROP$value
## [1] "0.0626403"
## 
## $RUV_PROP$lo
## [1] "0.001"
## 
## 
## $RUV_ADD
## $RUV_ADD$value
## [1] "0.315558"
## 
## $RUV_ADD$lo
## [1] "0.001"
print(myParObjUpdated@VARIABILITY)
## $PPV_CL
## $PPV_CL$value
## [1] "0.263616"
## 
## $PPV_CL$type
## [1] "sd"
## 
## 
## $PPV_V
## $PPV_V$value
## [1] "0.13666"
## 
## $PPV_V$type
## [1] "sd"
## 
## 
## $PPV_KA
## $PPV_KA$value
## [1] "0.91839"
## 
## $PPV_KA$type
## [1] "sd"
## 
## 
## $PPV_TLAG
## $PPV_TLAG$value
## [1] "0.1"
## 
## $PPV_TLAG$type
## [1] "sd"
## 
## $PPV_TLAG$fix
## [1] "true"
## 
## 
## $OMEGA
## $OMEGA$parameter
## [1] "[ETA_CL,ETA_V]"
## 
## $OMEGA$value
## [1] "0.213652"
## 
## $OMEGA$type
## [1] "corr"

A bug in the writeMogObj function means that for now, we must manually add the square bracket around the OMEGA value to signify that this is a vector (of length 1).

myParObjUpdated@VARIABILITY$OMEGA$value<-paste0("[",myParObjUpdated@VARIABILITY$OMEGA$value,"]")

Assembling the new MOG. Note that we reuse the data, model and tasks from the previous run.

myVPCMOG <- createMogObj(dataObj = getDataObjects(mdlfile)[[1]], 
        parObj = myParObjUpdated, 
        mdlObj = getModelObjects(mdlfile)[[1]], 
        taskObj = getTaskPropertiesObjects(mdlfile)[[1]])

We can then write the MOG back out to an .mdl file.

mdlfile.VPC <- paste0(uc,"_VPC.mdl")
writeMogObj(myVPCMOG,mdlfile.VPC)

Similarly as above, ddmore “VPC.PsN” function can be used to run a VPC using PsN as target tool

vpcFiles <- VPC.PsN(mdlfile.VPC,samples=20, seed=12345,
        vpcOptions ="-n_simulation=10 -auto_bin=10",
        subfolder="VPC", plot=TRUE) 
## -- Fri Dec 11 03:47:50 2015
## New
## Submitted
## Job b3aade9d-41c0-4acb-b06a-fa8223d8f039 progress:
## Running [ .... ]
## Importing Results
## Copying the result data back to the local machine for job ID b3aade9d-41c0-4acb-b06a-fa8223d8f039...
## From C:\Users\zparra\AppData\Local\Temp\Rtmpkfi8Jd\DDMORE.jobfd055a5631f to D:/SEE-1.2-SNAPSHOT-10DEC2015/MDL_IDE/workspace/UseCasesDemo/UseCase1/VPC
## Done.
## 
## 
## The following elements were parsed successfully:
##       RawResults
##       TaskInformation
##       Simulation
## 
## The following MESSAGEs were raised during the job execution:
##  nmoutput2so_version: This SOBlock was created with nmoutput2so version 4.5.11
## 
## Completed
## -- Fri Dec 11 03:49:15 2015

plot of chunk VPC

To replay the visualisation using information from the VPC SO file

pdf(paste0(uc,"_VPC.pdf"))
 plot(xpose.VPC(vpc.info= file.path("./VPC",vpcFiles@RawResults@DataFiles$PsN_VPC_results$path),
        vpctab= file.path("./VPC",vpcFiles@RawResults@DataFiles$PsN_VPC_vpctab$path),
        main="VPC warfarin"))
dev.off()
## rj.GD 
##     2

Simulation using simulx

The mlxR package has been developed to visualize and explore models that are encoded in MLXTRAN or PharmML The ddmore function as.PharmML translates an MDL file (extension .mdl) to its PharmML representation. The output file (extension .xml) is saved in the working directory.

myPharmML <- as.PharmML(mdlfile)

Use parameter values from the FOCEI estimation

parValues <- getPopulationParameters(NM.FOCEI, what="estimates")$MLE

Simulate for the typical weight of 70. Recall that logtWT = log(WT/70).

p <- c(parValues,logtWT=0)

Parameter values used in simulation

print(p)
##       POP_CL        POP_V       POP_KA     POP_TLAG     RUV_PROP 
##    0.1346550    8.0464900    1.2941000    0.8633020    0.0626403 
##      RUV_ADD       PPV_CL ETA_CL_ETA_V        PPV_V       PPV_KA 
##    0.3155580    0.2636160    0.2136520    0.1366600    0.9183900 
##   BETA_CL_WT    BETA_V_WT     PPV_TLAG       logtWT 
##    0.7500000    1.0000000    0.1000000    0.0000000

Simulate PK parameters for individuals

ind <- list(name = c('TLAG','KA','CL','V'))

Simulate predicted (CC) and observed concentration values (Y)

f   <- list( name = c('CC'), time = seq(0,to=50,by=1))
y   <- list( name = c('Y'), time = c(0, 0.5, 1, 2, 3, 4, 6, 8, 12, 24, 36, 48))

Simulate for a dose of 100mg given at time 0 into the GUT (oral administration)

adm <- list(target='GUT', time = 0, amount = 100)

Simulate 12 subjects

g <- list( size = 12, level = 'individual', treatment=adm)

Call simulx

res  <- simulx(model =myPharmML,
               parameter = p,
               group = g,
               output = list(ind,y,f))

Simulated parameter values for each individual

print(res$parameter)
##    id      TLAG        KA        CL         V
## 1   1 0.8375986 0.9920872 0.1509609  8.537704
## 2   2 0.8860958 0.8962518 0.2178778 10.326384
## 3   3 0.8776870 0.3838913 0.1717796  9.129087
## 4   4 0.8543526 0.5390183 0.1201255  7.584033
## 5   5 0.8205742 1.7842693 0.1445231  8.346977
## 6   6 0.8131703 1.2031132 0.1839108  9.457811
## 7   7 0.8966237 2.0364958 0.1927721  9.691372
## 8   8 0.7396332 2.5387691 0.1373349  8.129114
## 9   9 0.8739534 0.5823834 0.0884723  6.472044
## 10 10 0.8561372 0.7268690 0.1263188  7.784280
## 11 11 1.1477758 2.1630780 0.1369385  8.116943
## 12 12 0.8629955 0.8036848 0.1314386  7.946272

Plot simulated results

plot(ggplot() + 
                geom_line(data=res$CC, aes(x=time, y=CC, colour=id)) +
                geom_point(data=res$Y, aes(x=time, y=Y, colour=id)) +
                xlab("time (h)") + ylab("concentration") )

plot of chunk unnamed-chunk-47

Simulate 1000 subjects - with simulx this is a QUICK process!

g <- list( size = 1000, level = 'individual',  treatment = adm)

Call simulx

res.1000  <- simulx(model =myPharmML,
                    parameter = p,
                    group = g,
                    output = list(ind,y))

Plot of observed concentrations (with residual error) band defines the percentile bands displayed level = range of values to examine (in %) 100 = full range of values number = number of bins within the level range.

plot(prctilemlx(res.1000$Y,band=list(number=9, level=90)))

plot of chunk unnamed-chunk-49

Table of the same information

print(prctilemlx(res.1000$Y,band=list(number=10, level=100), plot=F)$y)
##    time         0%        10%        20%        30%         40%
## 1   0.0 -0.7367892 -0.4188878 -0.3131498 -0.1905105 -0.14161468
## 2   0.5 -0.6027510 -0.4082833 -0.2200972 -0.1167445 -0.04065004
## 3   1.0 -0.5371799  0.2743341  0.7101220  1.0009734  1.34157408
## 4   2.0  2.2714452  5.5964348  6.4583131  7.7882759  8.64207723
## 5   3.0  2.6283857  7.5135828  8.6382275  9.5501936  9.99517181
## 6   4.0  3.9258288  8.4683969  9.1395611  9.8584007 10.52896350
## 7   6.0  5.3036081  8.5987116  9.6612880 10.2294718 10.54512092
## 8   8.0  5.6303819  8.1929343  9.3549571  9.8825912 10.17776574
## 9  12.0  4.6324950  8.2650898  8.7782961  9.1960534  9.62448369
## 10 24.0  5.4868471  5.9656036  6.7701568  7.3227634  7.70263888
## 11 36.0  2.8693406  5.3954686  5.8518125  6.2640316  6.55615224
## 12 48.0  1.5950740  3.8098463  4.5281105  4.8755757  5.29940734
##             50%          50%         60%        70%        80%        90%
## 1  -0.044014491 -0.044014491  0.01783634  0.1749709  0.2520452  0.4209893
## 2   0.009139546  0.009139546  0.09762520  0.1540492  0.2034509  0.3397408
## 3   1.690379409  1.690379409  2.21018696  3.2811919  4.6226969  6.1370359
## 4   9.201113363  9.201113363  9.86412981 10.9580638 11.4927256 12.2441507
## 5  10.721295557 10.721295557 11.24165402 12.2172259 12.9322771 13.7117881
## 6  10.912309692 10.912309692 11.79997481 12.4598525 12.8987635 13.7841396
## 7  11.095046220 11.095046220 11.43133806 12.0591680 12.7244747 13.8639621
## 8  10.923284027 10.923284027 11.26192243 11.8657339 12.3734302 13.8150349
## 9  10.079842889 10.079842889 10.96807274 11.4822120 11.9794693 12.6325038
## 10  8.275969551  8.275969551  8.77827571  9.2169920  9.7304951 10.6619173
## 11  6.921711877  6.921711877  7.17136143  7.7298722  8.3841908  9.0208923
## 12  5.626831839  5.626831839  5.98476612  6.3530532  6.9256588  7.6198832
##          100%
## 1   0.7340669
## 2   0.5878415
## 3  11.5787461
## 4  14.7992211
## 5  17.9727340
## 6  15.2917303
## 7  17.1127002
## 8  15.9425439
## 9  15.4389169
## 10 15.3372079
## 11 12.4121838
## 12 10.1885257