Model Risk

What is Model Risk? 

Model risk arises when a financial model measures any company's quantitative information, such as its market risk or Value of transactions. As a result, the model fails or performs poorly, resulting in adverse effects for the company.

A model is a system, quantitative method, or approach based on

• Economic
• Statistical
• Mathematical
• Financial techniques and theoretical assumptions

Any model-derived process data inputs into a structured output. Financial institutions and investors usually use models to determine the theoretical Value of stock prices and identify trading opportunities. 

While models can be valuable tools in investment analysis, they are also subject to various risks that can arise from incorrect data use, programming, technical, and errors resulting in misinterpretation of model results. 

Model risk is considered a subset of operational risk because it primarily affects the business that creates and uses the model. 

Traders or other investors using a given model may not fully understand its assumptions and limitations, which limits the usefulness and application of the model itself.

In financial firms, pattern risk can affect the outcome of a prudential valuation, but it is also a factor in other industries. For example, a model may not accurately predict the likelihood that an airline passenger is a terrorist or the possibility of a fraudulent credit card transaction. 

It can be due to incorrect assumptions, programming or technical errors, and other factors that increase the risk of a bad outcome.

Sources of Model Risk

It can originate from different sources. A primary source is the incorrect model specification, which can manifest as missing risk factors, misidentification of the model's underlying stochastic processes, or a lack of knowledge regarding the crucial variables.

Incorrect model applications frequently result in this risk. It can be because you either use the wrong model for the problem or employ a model that is no longer the recommended practice. Another type of risk metric is the implementation risk.

Let's discuss the significant sources briefly:

1. Data

The data used in a model may be inaccurate, incomplete, or distorted. Nevertheless, it is essential for developing an effective model; inaccurate data can affect the whole model. 

2. Model Execution 

Incorrect or incomplete model implementation can lead to erroneous results, negatively affecting model outcomes and the organization's decision-making.

3. Inconsistency over time 

Bruschi and Corelli formalize the concept of "time inconsistency" for non-arbitrage models that allow for a perfect fit to the term structure of interest rates. 

In these models, the current yield curve is the input so that new observations on the yield curve can be used to update the model at regular intervals. In addition, they explore the question of time-appropriate and self-funding strategies in this model. 

Risk modeling affects the three main phases of risk management: specification, estimation, and implementation. 

4. Uncertain correlation

Trading options that depend on the performance of a basket, known as a rainbow option, are more susceptible to model uncertainty than index options, according to Cont and Digest. 

Uncertainty in correlation parameters is another vital source of model risk for multi-asset derivatives.

Gennheimer argues that investors must be very confident of the dependency structure governing the basket of assets. 

Default products should calculate prices according to alternative copula specifications and check their simulation error estimates for the least amount of model risk they incur.

5. Parameters and assumptions: 

Bruschi and Corelli formalize the concept of "time inconsistency" for non-arbitrage models that allow for a perfect fit to the term structure of interest rates. 

6. Complication

The complexity of a financial model or contract can be a source of risk metrics, leading to poor identification of its risk factors. For example, this factor was considered a significant source of risk for mortgage-backed securities portfolios during the 2007 crisis.

7. Liquidity and risk metrics

Risk metrics do not exist only for complex financial contracts. Frey (2000) presents a study on how illiquidity is marketed as a source of model risk. 

Convertible, mortgage-backed, and high-yield bonds can often be illiquid and difficult to Value. 

Types of Model Risk

We define model risk as the risk that a model will be misdefined, improperly implemented, or used inappropriately.

Consider the swap assignment. 

1. A financial engineer can use economic theory to develop a model. 
2. A programmer can execute the model as a computer program. 
3. A trader can use this implementation to price a swap. 

In this example, there is a risk that the financial engineer specifies the wrong model, that the programmer may implement the model incorrectly, or that the trader uses it in a way that is not intended, possibly by pricing a non-standard swap for which the model does not fit. 

Here we have three types of risk metrics: 

• Type A: model specification risk, 
• Type B: model implementation risks  
• Type C: model application risk. 

Each quantitative model has three components:

• Inputs 
• Analysis 
• Outputs

To evaluate such risk, we need to assess the probability for each type of model risk in each component of the model.

Type A: Model Specification Risk

Model risk is the risk that the model is not correctly specified. The question is not whether the model is "correct" but whether it is "useful."

All meaningful models make predictions. A well-specified model makes generally useful predictions when properly implemented and used. It is true regardless of whether the model is used for earthquake forecasting, weather forecasting, or market risk assessment.

The risk of model specifications arises primarily when the model is designed. For example, if the inputs are operationally defined, mathematical calculations and outputs are specified, and everything is preferably done in a formal design document. 

Inadvertent mistakes, such as formula typing errors, are always a problem. Still, the more common problem is that certain combinations of inputs and calculations can produce misleading or useless results. 

Experienced modelers can review design documentation and provide estimates of model performance, but the final test is to implement the model and evaluate its performance empirically.

Model specification errors also occur after the model is designed. Finally, the model itself is subject to change from time to time, perhaps to address new products being traded, to address issues that are perceived in its performance, or to catch up with industry practices. 

It can also occur without apparent changes to the Value of at-risk measurements. For example, historical values ​​for a particularly crucial factor can be obtained from a particular time series maintained by the data provider. 

How data providers calculate this time-series value or simple quality control can adversely affect the performance of value-at-risk measurements.

Type B: Model Implementation Risks

Model implementation risk is when the model deviates from what is specified in the design document during model implementation. For example, inputs may vary when historical data for essential elements comes from sources other than those specified in the design document. 

The formula may be different. For example, it may be due to a simple typing error. Alternatively, the programmer may implement the procedure in a way that accidentally modifies it. 

Results can be misrepresented. The two numbers may appear side by side in the risk report, or the results may be confusing or misleading. 

Implementation errors can cause an excellent model to be implemented as a wrong model, but that's only one part of the problem. 

If the implemented model deviates from its design, whether it works or not, it is not what the designer or user believes. They think there is one model when there is another, due to which the results may be unpredictable. 

Implementation errors are usually due to human error. Coding and logic errors are almost inevitable in large software projects. Implementation risks also arise if a rogue programmer decides on a trade-off. 

Unfortunately, such errors do occur, but they are not common in large-scale value-at-risk implementations, which are subject to rigorous testing and validation.

Type C: Model Application Risk

Model application risk is the risk that the model will be misinterpreted or misused. Misunderstandings can lead to false assurance and adverse actions.

Modelers tend to have different backgrounds than traders and senior management. Model designers understand the meaning of the output at a technical level, but users may find it less accurate and more intuitive. 

Modelers generally understand the reliability of the model's results better than users who may have unrealistic expectations. If the user doesn't like what the model is saying, it is more likely that user emotions will cloud the evaluation of the model's Value. 

A common misconception about Value at risk is that it is the most significant loss. However, in the scheme of things, the loss quantile is more straightforward to grasp than other PMMRs, such as B—the standard deviation of expected tail loss or return. 

A common form of model application risk is to use a value-at-risk indicator that is otherwise inadequate for the portfolio. 

It can happen as an organization's trading activities evolve, but its risk value measurements have not been updated to reflect new products or trading strategies. 

Alternatively, the model's environment may evolve, making its predictions useless. The model may be specified appropriately for the old domain but is irrelevant when applied to the new environment. 

For example, value-at-risk indicators must consider critical factors associated with liquid and actively traded commodities. But the market is evolving, and liquidity can be exhausted.

Quantitative Approaches in Model Risk 

They are techniques that help quantify the modeling risk inherent in each source, using a methodology based on the sensitivity of the model output to potential variations that characterize the uncertainty of the head.

Risk mitigation models are identified and quantified by applying appropriate measures, depending on the nature of each source.

Even when modeling risk cannot be eliminated, an approach combining a rigorous risk management structure with the sensitive and detailed quantification described can be an effective strategy to reduce risk.

Let's discuss some fundamental quantitative approaches to mitigating these risks:

The model mean and worst-case approach 

Rantala (2006) mentions, "When faced with model risk, instead of basing decisions on a single selected 'best' model, modelers can base their inferences on a complete set of models using the mean of the models." This approach avoids the "default of the mean." 

Another approach to model risk is the worst-case approach, or maximum approach, advocated by Gilboa and Schmeidler in decision theory. This approach considers a series of models, minimizing the worst-case loss. 

Jokhadze and Schmidt (2018) propose risk measurement models using the Bayesian method. They introduce layered risk measures that combine risk metrics and enable consistent model and market risk management. 

In addition, they provide the hypothesis of these risk measures and identify some real-world examples of overlapping risk metric measures in the context of financial risk management and potential loss pricing.

Quantify the model's level of risk 

Measure the risk posed by a model. It must be compared with an alternative model or a set of alternative reference models. The question is how to choose these reference models. 

In the context of derivatives valuation, Cont (2006) offers a quantitative approach to measuring the model risk of derivative portfolios, where a set of reference models are specified and adjusted to the market price of the liquid product model.

Risk measurements arise from the difference between the current portfolio and worst-case valuation within the benchmark model framework. In addition, such indicators can determine the risk metrics reserved for derivative portfolios. 

Position limits and valuation reserves 

Jokhadze and Schmidt (2018) introduce financial market risk measures that cover the loss of risk metrics. 

Their methodology harmonizes market and risk management, allowing them to define risk position limits and capital requirements. Kato and Yoshiba discuss qualitative and quantitative methods for managing risk metrics.

They write: "From a quantitative point of view, you can create a reserve in the price model to explain the difference in estimation from the alternative model. Cont (2006) advocates using risk metrics to calculate such accounts.

The risk measurement model provides scenario analysis for various volatility patterns. You can do this. You can set risk factors or location limits based on the information obtained from the scenario analysis.

Model Risk Management Framework 

Model-based risk management aims to identify and reduce the potential negative consequences of decisions based on incorrect or misused models. This risk management uses techniques and practices to identify, measure, and reduce risk metrics, i.e., the possibility of model failure or misuse.  

In finance, it is the risk of loss resulting from using models that are not accurate enough for decision-making. 

It is becoming more common in financial services sectors, which include consumer credit scoring, predicting real-time probabilities of fraudulent credit card transactions, and money laundering.

Financial institutions rely heavily on credit, market, and behavioral models because modeling risk has become central to risk management and performance. 

These institutions make money by taking risks - they leverage models to assess risk, understand client behavior, evaluate capital adequacy, make investment decisions, and manage data risk analysis. 

Implementing a practical model-based risk management framework is a prerequisite for organizations that rely heavily on quantitative models for operations and decision-making.

Processes in the Lifecycle of MRM Framework

A good Model Risk Management (MRM) framework should be designed around industry best practices and in compliance with regulatory guidelines. 

The authority for comparing MRM frameworks is the US Federal Reserve's Supervision Guidelines for Model Risk Management (SR 11-07). 

An organization must create a framework for managing risk metrics to model risk effectively. The MRM Framework shows how to manage modeling risk. 

Regardless of the size and structure of the organization, regulators require that business model risk management frameworks cover all essential aspects of the MRM lifecycle with roles and responsibilities assigned. 

Like other generic risk management frameworks, modeled risk management provides a comprehensive explanation of the four pillars below:

1. Risk modeling standards

Minimum standards must have come for model development and must be followed and adhered to. Internal standards should be at the same level or higher than regulatory standards, such as the Model Risk Management Monitoring Guide (SR 11-07).

These standards should include data quality, model modification, model usage, expert judgment, model methodology, model validation, documentation, model data external, and model reporting, among other standards.

Risk appetite pattern: After establishing the risk policy, it is prudent that the board's statement of the MR appetite is clearly articulated to manage it effectively. Risk appetite is the amount of risk an organization is prepared for and can accept to achieve its desired objectives.

The model's risk preference for risk will depend on the objective to which the model is applied. The most express the model's risk appetite in terms of risk tolerance and related metrics such as overall quantitative risk level, number of high-risk models, etc.

2. Risk Metrics Determination

It is necessary to identify the specific risks affecting the organization. An inventory of existing models should be conducted to determine fundamental model changes. Model inventory should categorize features as follows (among others):

• Form name
• Describe the purpose of the model
• How to use the model
• How often to use it
• Assumptions or inputs of the model

3. Model risk assessment and measurement

It should perform quantitative and qualitative risk assessments to assess each model's risk. Both approaches will lead to an enterprise-wide risk assessment framework.

The risk quantifiers use different modeling risk measures, or they may use operational risk modeling approaches. There are three main techniques for quantifying risk, namely:

• Sensitivity Analysis - Changes in the model's assumptions and parameters and monitoring the evolution of the results
• Backtesting - Test the model using historical data and compare outputs with past results
• Challenge Model - Compare the results of one model with the results of another alternative model using the same data

The quantitative assessment will measure and aggregate each quantitative risk assessment separately through appropriate correlation factors. Qualitative risk assessment involves looking at the model's fit to the objective.

The results will indicate how robust the model is, impacting the model's risk rating.

The qualitative assessment uses qualitative measures to measure risk in a model - including model fit to standards, cumulative model error, degree of risk assessment, model risk, and other qualitative factors.

4. Risk Mitigation Model

Possible risk reduction strategies may include the following:

• Changes during model development.
• Conduct additional validation of the model, taking into account changes in the nature and structure of existing risks and the emergence of new threats the organization faces.
• Use independent expert judgment to interpret model results due to model uncertainty. 
• The Compliance models and applicability to new risk regulations.
• Measures to improve model efficiency and applicability to reduce risk, such as additional capital, can help mitigate risk.

5. Risk monitoring and reporting model

The model's risk monitoring and reporting functionality aim to identify the following issues:

• Monitor whether the policy follows the risk metrics policy and risk appetite. The process will recommend whether management intervention is needed in the event of a discrepancy.
• A material model inventory should be performed on each model to measure whether the MRM policy framework is using the model.
• The model's risk assessment and validation results should be analyzed, and corrective actions should be taken for any identified weaknesses.
• An overview of new trends in modeling risk management and other related topics.

Examples of Model Risks

Models can fail due to technical errors, such as a lack of mathematical rigor, data problems, and implementation errors. These types of model-risk failures can be considered part of operational risk.

However, the models produce inaccurate results simply because of the inherent uncertainty associated with the mathematical model. Of course, this is a feature of modeling, but failure to properly recognize this level of uncertainty is a major element of risk metrics.

Failure of a model can result in financial loss, regulatory or legal penalties, or damage to an organization's reputation due to the use of a model that contains data or assumptions that do not fit that model.

Some examples of model risk failures witnessed over the past years are as follows:

1. Long-term capital management

Long-term capital management (LTCM) failure in 1998 was attributed to risk metrics. In this case, a small error in the company's computer models was magnified by some large orders due to the high-leverage trading strategy used by LTCM.

At its peak, the hedge fund managed more than $100 billion in assets and posted annual returns of over 40%.

LTCM had two economics Nobel laureates as major shareholders, but the company collapsed due to its financial model failing in this market environment.

2. JPMorgan Chase

JPMorgan Chase (JPM) suffered massive trading losses from its Value at Risk (VaR) model that contained formulas and operational errors. Risk managers use VaR models to estimate a portfolio's future losses.

In 2012, CEO Jamie Dimon declared a "storm in the teapot" that turned out to be a $6.2 billion loss from erroneous transactions in his General Credit Portfolio (SCP). As a result, a trader established prominent derivative locations indicated by the VaR pattern.

The bank's chief investment officer responded by adjusting the VaR model. However, owing to a spreadsheet flaw in the program, trading losses might accrue without notice from the model.

It is not the first time that VaR models have failed. For example, in 2007 and 2008, the VaR model was criticized for failing to predict many banks' significant losses during the global financial crisis.

Researched and authored by Kavya Sharma | Linkedin

Reviewed and edited by Priyansh Singal 

Free Resources 

• Discount Factor
• 3 Statement Model
• LBO Modeling 
• Payback Period
• Scenario Analysis