Visualising model dependencies

1. Problem statement

When checking an actuarial model, common approaches are to do a detailed check of code and inputs (which should, in theory, pick up any errors in the model, but with the disadvantage that you can miss seeing the wood for the trees) and to test the reasonableness of outputs (with the advantage that this focusses on the implications of the model results for the real-world situation, but which might miss flaws in the details that would become critical under different parameterisations or uses).

We propose a way in which model parameters (and, in particular, the dependencies between them) can be visualised in order to provide a higher-level review of a model’s structure, to complement the above checks.

The example used here is a pension scheme valuation model, and in particular reviewing the programming of a pension scheme’s benefit structure.

2. Suggested approach

We suggest the use of a network map analysis of a pension scheme’s programmed benefit structure.

This was programmed as an interactive map so that the reviewer can vary the level of detail (for example, to zoom in on particular areas) given the large number of variables. The example output used in this blog post takes the form of screenshots from the interactive version.

Figure 1 below shows an example, focussing in on the calculation of accrued pension at normal pension age (which is 60 in this case) for one section of a large pension scheme.

This shows that, for example, one of the factors affecting accrued pension (the large purple dot, or Accrued NPA60 Pension) is the Final Pensionable Earnings (which is defined in the scheme rules) at the member’s future retirement. In turn, this will depend on the member’s full-time equivalent (FTE) salary at retirement, their date of leaving (DOL), and so on.

A pension scheme valuation model needs to reflect the wide range of benefits that a pension scheme might provide (for example, retirement pensions on early retirement, normal retirement or late retirement; any lump sum benefits paid on retirement or death; any ill-health retirement pension that might become payable; any deferred pensions payable where a member has changed jobs or otherwise left the scheme; any benefits paid to a spouse or other partner, etc). In each case, the scheme rules will specify the calculation of the member’s benefit, which will depend on a number of different elements. The network map allows all such parameters and dependencies to be visualised.

3. Rationale and commentary

Network maps show the connections (links) between different items (or nodes). Network maps can be either undirected (where the links show connections between the items with no direction specified) or directed (where one-way or two-way arrows indicate the directions of the connections). We have used a directed network map in this case.

The rationale for using a network map for this purpose includes:

• Visualising the parameters in this way can check for consistency between the benefit programming of different schemes – do the network maps for similar schemes have a similar structure?
  
• An initial review of the network map can highlight any unexpected features, for example a member’s retirement pension that depends on any assumed characteristics of their spouse or partner would usually be an error.
• The network map easily highlights areas of complexity and simplicity, which might suggest further investigation is required.  For example, if a particular variable has an unusually high number of dependencies, this warrants a closer look.
• Such a review of the model complements other checks, and might highlight features which might have been missed otherwise.
• The network map allows for an easier view of the interdependencies of benefit scheme structures and could lead to a streamlining of the setup.

Here are a few examples of how the network map can be used:

Figure 2 shows the calculation of accrued pension for a scheme where some of the benefits have accrued with a Normal Pension Age (NPA) of 60. It sets out the items which it relies on – i.e. service and salary. The network graph clearly shows that the accrued NPA60 pension is correctly calculated using NPA60 service. However, in a previous iteration the graph helped to identify that the pension was incorrectly being calculated using some NPA65 service. 

Figures 3 and 4 below show an example with an unusually complex benefit structure. Checking the setup using the traditional format (which involves numerous lines of data interacting with each other) would take significant time and could still result in errors being overlooked. The interactive map allows for easier and more accurate identification of interactions between items. 

Figure 3 shows the large number of items which are linked to a particular benefit item, and Figure 4 shows the same item with a number of the more distance nodes removed. This can be enabled via a distance selector in the tool. This feature helps the user to focus on particular parts of the structure. 

Figure 5 below highlights an additional feature of the tool which allows the network map to be shown in various formats. Figure 5 shows the same benefit item as shown in figure 3 but the colour coding highlights the different states at which the calculations are carried out, in particular distinguishing between the calculations that are carried out at the point at which the member joins the scheme, to withdrawal, to retirement and finally death of the pensioner. This feature allows the user to concentrate on specific areas which might be causing an issue in the model.

4. Applicability and alternatives

In theory such an approach can be used with any model. Its applicability will depend on the complexity of the model and its interaction with the suite of other model checks.

5. Implementation

The example illustrated here was developed using R studio, also using Shiny (an R package which makes it easy to build interactive web apps).

The R code reads a Microsoft Excel file which contains the detailed benefit structure including the interdependencies between data and benefit items. The code processes the data in the Excel file to create tables of the nodes and the node connections by assigning a unique identifier to each node.

The tables are then processed to produce the necessary nodes and connections to display in the interactive network map.

6. Context

Pension schemes provide a wide range of benefits to members and their dependants. The benefits payable will depend on factors including:

 

1. What happens to the member (and their dependant in the future), for example whether they are unable to work due to ill-health, whether they choose to take their retirement pension after their normal retirement date and how long they live for; 
2. The member’s work and salary history (including assumed future experience where the member is still in service); 
3. The precise benefit specification (which is set out in the scheme rules – sometimes also affected by legislation); and
4. External variables (eg the rate of consumer price inflation)

 

Many of these features interact.  In order to value the liabilities of a pension scheme (which is usually carried out in order to assess the sufficiency of the assets held in the scheme), a pension valuation model combines information on scheme members with detail of scheme benefits and a number of assumptions about unknown future experience (such as longevity and inflation assumptions).

The approach described above complements other checks on the model’s programming and parameterisation in order to reduce the risk of model and parameter error.