Interview Questions for Advanced Score Study and Analysis

Both logistic regression and decision trees are powerful tools for building scorecards, but they differ significantly in their approach. Logistic regression is a linear model that estimates the probability of an event (e.g., default) based on a weighted sum of input variables. It provides a continuous score, representing the likelihood of the event. Think of it as a smooth, continuous function that maps inputs to probabilities. Decision trees, on the other hand, are non-parametric models that partition the input space into regions, each associated with a predicted outcome. They create a tree-like structure of decisions based on feature thresholds, resulting in a piecewise constant score. Imagine it as a series of ‘if-then-else’ statements, creating distinct score ranges based on various combinations of features.

In scorecard development, logistic regression offers better interpretability due to its linear nature – you can easily see the contribution of each variable. However, decision trees can capture non-linear relationships more effectively and are less prone to overfitting with careful regularization. The choice depends on the specific data, business needs (interpretability vs. predictive accuracy), and the complexity of the relationships between variables.

For example, if interpretability is paramount (e.g., regulatory requirements), logistic regression might be preferred. If the underlying relationships are highly complex and non-linear, a decision tree (or ensemble methods like Random Forests or Gradient Boosting Machines) might provide better predictive power.

Feature engineering is crucial for building high-performing scorecards. It involves transforming raw data into features that are more informative and predictive. My experience includes a wide range of techniques, such as:

• Creating interaction terms: For example, combining age and income to create a ‘wealth’ indicator, capturing the combined effect of these two variables.
• Binning continuous variables: Transforming continuous variables like credit score into categorical bins (e.g., ‘low’, ‘medium’, ‘high’) to handle non-linear relationships and improve model stability. The choice of binning method (e.g., equal width, equal frequency, quantile) is important and depends on the data distribution.
• Creating dummy variables: Converting categorical variables into numerical representations (one-hot encoding) for use in models like logistic regression.
• Deriving new features from existing ones: For instance, calculating the ratio of debt to income or the percentage of late payments from a history of transactions. This process requires domain expertise to create truly meaningful features.
• Using date/time features: Extracting features like time since last transaction or the number of transactions in a given time period.

In one project, I improved a credit risk scorecard’s performance by creating a feature representing the frequency of account inquiries in the past six months. This proved far more predictive than simply using the total number of inquiries.

Imbalanced datasets, where one class (e.g., defaults) is significantly under-represented compared to the other (non-defaults), are a common challenge in scorecard development. This can lead to models that are highly accurate on the majority class but poor at identifying the minority class (which is often the class of primary interest). Several techniques can be applied:

• Resampling techniques: Oversampling the minority class (creating synthetic samples) or undersampling the majority class can balance the dataset. However, oversampling needs to be done carefully to avoid overfitting. Techniques like SMOTE (Synthetic Minority Over-sampling Technique) are effective.
• Cost-sensitive learning: Assigning different misclassification costs to the classes, penalizing misclassifications of the minority class more heavily. This approach modifies the model’s objective function to prioritize the minority class.
• Anomaly detection techniques: If the minority class is extremely rare, it might be more appropriate to frame the problem as an anomaly detection task, using techniques like isolation forests or one-class SVMs.
• Ensemble methods: Combining multiple models trained on different subsets or using different resampling strategies can improve performance on imbalanced data.

The best approach depends on the severity of the imbalance and the characteristics of the data. Careful evaluation and comparison are crucial to choose the most effective strategy.

‘Lift’ is a measure of how much better a scorecard is at identifying the target group (e.g., defaulters) compared to random selection. It’s calculated by comparing the percentage of the target group captured by a given scorecard percentile to the overall percentage of the target group in the entire population. For example, if the top 10% of scores contain 30% of the defaulters, the lift is 3 (30%/10%).

Lift is crucial because it quantifies the efficiency of the scorecard in identifying the most risky individuals. A high lift indicates that the scorecard is concentrating the target population in the top scoring percentiles. This is important for resource allocation, as it allows focusing efforts on the most likely candidates, such as targeted interventions or enhanced monitoring.

Imagine a marketing campaign: a scorecard with high lift allows focusing marketing efforts on the high-probability customers who are more likely to respond positively, maximizing the ROI.

Scorecard calibration aims to ensure that the predicted probabilities from the model accurately reflect the observed event rates. Several methods are used:

• Platt scaling: Fits a logistic regression model to the model’s output scores and the observed event rates to adjust the probabilities.
• Isotonic regression: Fits a non-decreasing piecewise linear function to the model’s output scores and the observed event rates. This is particularly useful when the relationship between scores and probabilities is non-linear.
• Binning and recalibration: Grouping the scores into bins and recalibrating the probabilities within each bin based on the observed event rates in that bin.

The choice of method depends on the model and the nature of the probability calibration required. Platt scaling is commonly used and relatively simple to implement, while isotonic regression offers more flexibility to handle non-linear relationships.

Robust validation is essential to ensure a scorecard generalizes well to unseen data. My experience includes various techniques:

• Holdout sample validation: Splitting the data into training and holdout sets. The model is trained on the training set and evaluated on the holdout set to assess its performance on unseen data. This is a fundamental technique.
• Out-of-time validation: Testing the model on data from a time period later than the data used for training. This assesses the model’s stability and ability to handle changes in the underlying data-generating process over time. It’s critical for detecting concept drift.
• K-fold cross-validation: Repeatedly splitting the data into training and validation sets, averaging the performance across multiple folds. This provides a more robust estimate of model performance than a single holdout split.

In practice, I often use a combination of these techniques. For instance, I might train a model using a large training set, evaluate its performance on a holdout sample, and then conduct out-of-time validation to ensure long-term stability.

Assessing scorecard stability over time is critical to ensure its continued reliability. This involves monitoring several aspects:

• Performance monitoring: Tracking key performance indicators (KPIs) like AUC, KS statistics, and lift over time. A significant drop in these metrics can indicate a problem.
• Concept drift detection: Regularly checking for shifts in the relationship between predictor variables and the target variable. Statistical tests can be used to detect significant changes.
• Out-of-time validation: Regularly evaluating the scorecard on data from progressively later time periods to detect performance degradation. This is the most direct method for detecting instability.
• Regular model retraining: Periodically retraining the model with updated data to account for changes in the underlying population and maintain its accuracy.

A decline in performance or significant concept drift necessitates investigation into the causes. This might involve updating features, retraining the model, or even re-engineering the entire scorecard.

The Kolmogorov-Smirnov (KS) statistic is a crucial metric in scorecard evaluation that measures the discriminatory power of a scoring model. It quantifies the degree of separation between the cumulative distribution functions (CDFs) of good and bad customers (or events). A higher KS statistic indicates better model performance, signifying that the model effectively differentiates between the two groups.

Imagine you’re sorting apples and oranges. A perfect model would have all the apples on one side and all the oranges on the other. The KS statistic essentially measures how well your model manages this sorting, with a higher KS value indicating a cleaner separation.

In practice, we plot the cumulative percentage of good and bad customers against the score percentiles. The maximum vertical distance between these two curves is the KS statistic. A KS value of 0.5 or higher typically signifies a very good model; lower values indicate weaker predictive ability.

Handling missing values is critical in scorecard development, as they can significantly impact model accuracy and performance. The approach depends on the nature and extent of missing data. Common strategies include:

• Deletion: Removing observations with missing values is a straightforward approach but can lead to information loss, especially with a large percentage of missing data. This method is suitable only when the missing data is minimal and does not show any systematic bias.
• Imputation: Replacing missing values with estimated values based on available data. Methods include mean/median imputation, k-nearest neighbors imputation, or more sophisticated techniques like multiple imputation. The choice depends on the data characteristics and the potential impact on the model.
• Indicator Variable: Creating a new binary variable indicating the presence or absence of a missing value. This captures the information about missingness, allowing the model to learn its potential relationship with the outcome.

The best approach often involves a combination of these strategies, tailored to the specific dataset and modeling goals. For instance, if missingness is correlated with the outcome variable, simple imputation may be misleading, and employing an indicator variable might be more appropriate.

Real-world scorecard implementations face numerous challenges:

• Data quality issues: Inconsistent data, missing values, and outliers can significantly affect model accuracy and performance. Robust data preprocessing techniques are crucial.
• Dynamic environments: Customer behavior and market conditions change constantly. Regular model monitoring and retraining are essential to maintain performance over time. A model built on past data may not be reflective of the future.
• Regulatory compliance: Scorecards, particularly those used in lending or credit risk assessment, must comply with strict regulations, such as Fair Lending laws, to ensure fairness and prevent discrimination.
• Explainability and interpretability: Understanding *why* a scorecard assigns a particular score is often crucial for business decisions and regulatory compliance. Complex models may sacrifice interpretability for higher accuracy.
• Data drift: Changes in data distribution over time can render the model obsolete, highlighting the need for continuous monitoring and updates.

Addressing these challenges requires a robust development process, careful model selection, ongoing monitoring, and a strong understanding of the business context.

Measuring the business impact of a scorecard involves quantifying its contribution to key business objectives. This typically involves:

• Improved profitability: For example, a credit scoring model might reduce loan defaults, leading to increased profits.
• Reduced risk: Better risk assessment can help minimize losses from bad loans or fraudulent activities.
• Enhanced efficiency: Streamlined processes and automated decision-making can reduce operational costs.
• Increased customer satisfaction: Tailored offers and improved customer experience can boost loyalty and retention.

These impacts can be measured using metrics such as return on investment (ROI), reduction in default rates, improvement in customer acquisition costs, or changes in customer churn rates. A comprehensive analysis, comparing performance before and after scorecard implementation, is crucial to demonstrate its value to the business.

I have extensive experience with both point-in-time and behavioral scoring methods.

Point-in-time scoring uses a snapshot of data at a specific moment to assess risk or creditworthiness. Think of a traditional credit score based on a single credit report. It’s straightforward and easy to implement, but it may not capture the dynamic nature of customer behavior.

Behavioral scoring considers past customer behavior over time to predict future actions. This approach is particularly valuable for applications such as customer churn prediction or fraud detection. For example, a telecommunications company might use a customer’s call frequency, data usage, and payment history to predict their likelihood of churning. It’s more complex to implement but offers a more nuanced and predictive assessment.

The choice between these methods depends on the specific application and the availability of data. In some cases, a hybrid approach combining both point-in-time and behavioral data can provide the most accurate and comprehensive assessment.

Model explainability is the ability to understand and interpret how a model arrives at its predictions. It’s crucial in scorecard development for several reasons:

• Regulatory compliance: Many industries have regulations requiring transparency in decision-making processes. Explainable models help ensure compliance and avoid legal challenges.
• Business understanding: Understanding *why* a model makes a particular prediction is essential for building trust and making informed business decisions. This allows for the identification and fixing of biases.
• Model debugging and improvement: Explainability helps identify flaws in the model and refine its design. If you can see why a model is failing, you can rectify it.
• Fairness and bias detection: Explainable models make it easier to detect and address potential biases that may disproportionately affect certain groups.

Techniques for improving explainability include using simpler models (e.g., logistic regression), employing feature importance analysis (e.g., SHAP values), and creating visual representations of the model’s logic. The choice of technique depends on the complexity of the model and the specific needs of the application.

Variable selection for a scorecard is a critical step that significantly impacts performance and interpretability. The process typically involves:

• Business understanding: Identifying variables that are relevant to the business problem and have predictive power. This requires collaboration with domain experts.
• Data exploration: Analyzing the data to understand the distribution, relationships between variables, and potential issues such as missing values and outliers.
• Feature engineering: Creating new variables from existing ones to improve model performance. For example, creating interaction terms or transforming variables to improve their predictive ability.
• Statistical methods: Employing techniques like correlation analysis, chi-squared tests, information value (IV), and weight of evidence (WOE) to assess the predictive power of individual variables.
• Model building and evaluation: Iteratively building and evaluating models with different combinations of variables using metrics like KS statistic, AUC, and Gini coefficient. This step identifies the optimal subset of variables that balance predictive power and model complexity.
• Regularization techniques: Using techniques like L1 or L2 regularization to penalize model complexity and prevent overfitting. This helps to improve the model’s generalizability.

The final selection considers factors like statistical significance, business relevance, and the trade-off between model accuracy and interpretability.

Collinearity, the presence of high correlation between predictor variables in a scorecard, is a significant concern as it inflates variance and can lead to unstable and unreliable models. Identifying and addressing it is crucial for building robust scorecards.

• Correlation Matrix and Visualizations: A simple yet effective first step involves calculating the correlation matrix of predictor variables. High correlations (typically above 0.7 or 0.8, depending on the context and the model’s robustness) suggest collinearity. Visualizations like heatmaps can clearly show these relationships.

• Variance Inflation Factor (VIF): VIF measures how much the variance of an estimated regression coefficient is increased due to collinearity. A VIF greater than 5 or 10 (again, context-dependent) generally indicates a problem. This provides a quantitative measure of the severity of collinearity for each predictor.

• Feature Selection Techniques: Several techniques can help mitigate collinearity. These include:

  • Stepwise Regression: This iterative process adds or removes variables based on their contribution to the model, effectively minimizing redundancy.

  • Principal Component Analysis (PCA): PCA transforms correlated variables into a set of uncorrelated components, reducing dimensionality and resolving collinearity. It’s particularly useful when dealing with a large number of highly correlated variables.

  • Regularization Techniques (L1 and L2): These methods, like Lasso and Ridge regression, shrink the coefficients of highly correlated variables, reducing their influence and stabilizing the model.

• Domain Expertise: Sometimes, the best approach involves leveraging domain knowledge. If two variables are highly correlated and represent essentially the same underlying phenomenon, one might be eliminated based on understanding of the business problem. For instance, in a credit scoring model, ‘annual income’ and ‘total assets’ might be highly correlated, and a business decision might be made to keep just one based on its predictive power and ease of data collection.

While often used interchangeably, a scoring model and a scoring system are distinct concepts. A scoring model is the statistical algorithm or equation that predicts the target variable (e.g., credit risk, fraud probability). It’s the underlying mathematical engine. A scoring system, on the other hand, encompasses the entire process, including the data collection, model development, score calculation, score interpretation, and decision-making rules based on the score. Think of it like this: the scoring model is the heart of the system, but the system includes the circulatory and nervous systems that allow the heart to function within the body.

For example, a logistic regression model predicting default probability is a scoring model. The complete system, including data preprocessing, model training, score generation, and the thresholds used for making loan approval decisions, constitutes the scoring system.

Determining the optimal cutoff score involves balancing the trade-off between sensitivity and specificity (or, in business terms, between acceptance and rejection rates). There’s no single best method, but several approaches can guide the decision. The choice often depends on the business objective and risk tolerance.

• Kolmogorov-Smirnov (KS) Statistic: The KS statistic measures the separation between the cumulative distributions of good and bad cases. A higher KS indicates better model discrimination, helping to find a cutoff point maximizing this separation.

• Gini Coefficient: Similar to KS, the Gini coefficient assesses model discrimination power, offering another metric for optimizing the cutoff.

• Lift Chart Analysis: A lift chart plots the cumulative percentage of events (e.g., defaults) against the percentage of the population selected. The ideal cutoff point yields the highest lift, signifying superior targeting.

• Cost-Benefit Analysis: A more business-oriented approach involves assigning costs (e.g., cost of a bad loan) and benefits (e.g., profit from a good loan) to each classification. The optimal cutoff point minimizes the total cost or maximizes profit.

• ROC Curve and AUC: The Receiver Operating Characteristic (ROC) curve plots the true positive rate against the false positive rate for various cutoff scores. The Area Under the Curve (AUC) summarizes the overall performance; higher AUC indicates better discrimination. An optimal point on the ROC curve can be selected based on the desired balance between true positives and false positives.

Ultimately, the best cutoff score is determined through a combination of statistical measures and business judgment. It’s often an iterative process involving testing different cutoffs and evaluating their impact on key performance indicators (KPIs).

Monitoring and maintenance are essential for ensuring the ongoing performance and accuracy of a deployed scorecard. Over time, the relationships between predictor variables and the target variable can change due to various factors (e.g., economic shifts, changes in customer behavior, regulatory updates). This necessitates regular review and potential updates.

• Performance Monitoring: Key metrics like AUC, KS statistic, and predictive power should be tracked regularly to detect any performance degradation. This often involves comparing the model’s performance on recent data against its historical performance.

• Data Drift Detection: Regularly analyze the input data to identify any significant changes in the distribution of predictor variables. Data drift can significantly impact the model’s accuracy and requires recalibration or retraining.

• Regulatory Compliance: Ensure the scorecard continues to comply with relevant regulations and guidelines. This may involve updating the model or documentation to reflect any changes in legislation or best practices.

• Model Retraining: Periodically retrain the model using updated data to maintain its accuracy and predictive power. The frequency of retraining depends on the stability of the underlying relationships and the rate of data drift.

• Documentation and Audit Trails: Maintain thorough documentation of the scorecard development, deployment, and maintenance processes. This is crucial for auditing, compliance, and future model updates.

A well-defined monitoring and maintenance plan is vital for sustaining the scorecard’s effectiveness and avoiding potential negative impacts on business decisions.

Handling regulatory requirements is paramount in scorecard development and deployment. Regulations vary significantly across jurisdictions and industries, but common themes include fairness, transparency, and accountability.

• Fair Lending Laws: Scorecards must avoid discriminatory practices. This often involves rigorous testing for disparate impact across protected characteristics (e.g., race, gender, age).

• Data Privacy Regulations (e.g., GDPR, CCPA): Ensure compliance with data privacy laws by properly handling sensitive customer information and obtaining necessary consents.

• Explainability and Transparency: Many regulations require scorecards to be explainable, allowing stakeholders to understand how scores are derived. Techniques like SHAP values or LIME can be used to provide insights into model predictions.

• Model Validation and Auditing: Thorough validation and auditing processes are often mandated to verify the model’s accuracy, stability, and compliance with regulations.

• Documentation: Maintaining comprehensive documentation of the scorecard development process, including data sources, model specifications, and validation results, is essential for demonstrating compliance.

Staying abreast of evolving regulatory landscapes and adapting scorecard development processes accordingly is an ongoing responsibility.

My experience spans various scorecard applications, including:

• Credit Risk Scorecards: I’ve worked extensively on developing and deploying scorecards to assess the creditworthiness of individuals and businesses, utilizing various techniques like logistic regression, tree-based models, and neural networks. This includes handling both consumer and commercial loan applications.

• Fraud Detection Scorecards: I’ve built scorecards to identify fraudulent transactions, employing techniques suitable for handling imbalanced datasets (e.g., SMOTE, cost-sensitive learning). The focus here is on maximizing the detection rate while minimizing false positives, which can have significant cost implications.

• Marketing Scorecards: I’ve developed scorecards to segment customers and predict their likelihood of responding to marketing campaigns. These scorecards often incorporate customer demographics, purchase history, and web behavior data to optimize marketing spend and ROI. I have specific experience with identifying high-value customers, and developing models for customer churn prediction.

In each case, the specific techniques and model choices are tailored to the unique characteristics of the data and the business objectives.

I possess significant experience handling large datasets in scorecard development. Working with big data presents unique challenges, requiring efficient data processing techniques and scalable modeling approaches. My experience includes:

• Distributed Computing Frameworks (e.g., Spark, Hadoop): I’ve leveraged these frameworks to process and analyze massive datasets that wouldn’t fit into the memory of a single machine. This is crucial for training complex models on large volumes of data.

• Data Sampling and Feature Engineering: Efficient data sampling techniques are crucial for managing computational resources. Careful feature engineering is essential to extract relevant information from large datasets while avoiding the curse of dimensionality.

• Scalable Machine Learning Algorithms: I’m proficient in using scalable machine learning algorithms that can handle large datasets efficiently. This often involves using techniques like gradient boosting machines (GBM) or ensemble methods adapted for distributed computing.

• Data Governance and Quality Control: Handling large datasets necessitates robust data governance and quality control procedures to ensure data accuracy, consistency, and compliance with regulations.

My experience ensures that I can effectively manage the computational and data-related challenges associated with large-scale scorecard development projects.

Communicating complex scorecard results to a non-technical audience requires translating technical jargon into plain language and focusing on the key takeaways. Instead of presenting raw numbers and statistical models, I prioritize visualizations and storytelling.

For example, if a scorecard assesses credit risk, I wouldn’t present a ROC curve or Gini coefficient. Instead, I’d use a simple bar chart showing the percentage of good and bad loans predicted by the scorecard, comparing it to the historical performance. I’d explain the implications in terms of potential profit and loss, using clear, concise language avoiding technical terms like ‘sensitivity’ or ‘specificity’. I might say something like: “This scorecard helps us identify 80% of the risky loans, reducing our potential losses by 20%”. I also incorporate analogies – for instance, comparing the scorecard’s accuracy to a weather forecast’s reliability.

I believe in focusing on the business impact. What does this scorecard mean for the company’s bottom line? How will it improve decision-making? By concentrating on these high-level implications, I ensure that even non-technical stakeholders understand the value and purpose of the scorecard.

My proficiency in programming languages and statistical software is crucial for effective scorecard development. I am highly proficient in Python, utilizing libraries like scikit-learn for machine learning algorithms, pandas for data manipulation, and matplotlib and seaborn for visualization. I also have extensive experience with R, leveraging packages such as caret for model training and ggplot2 for creating insightful visualizations. For more robust statistical analysis, I use SAS and SPSS. In addition to these, I’m comfortable using SQL for database management and data extraction, a crucial part of the data preparation pipeline.

Data visualization is paramount in scorecard analysis. My experience encompasses a wide range of techniques, each suited for different aspects of analysis and audience comprehension. For instance, I use histograms to show the distribution of scores, ensuring we understand the range and concentration of risk. Scatter plots help examine the relationship between predictor variables and the target variable (e.g., credit score and default rate). Box plots are excellent for comparing score distributions across different groups (e.g., demographics) to identify potential bias. For demonstrating model performance, I use ROC curves and lift charts. Finally, dashboards, combining multiple visualizations, provide a holistic overview of the scorecard’s performance and key metrics, making them ideal for presentations to stakeholders.

Recently, I explored interactive dashboards using tools like Tableau and Power BI, enabling deeper exploration of scorecard results and improved collaboration among team members. The choice of visualization depends heavily on the audience and the specific insight we aim to convey. The goal is always clarity and effective communication.

Fairness and ethical considerations are paramount in scorecard development. I follow a rigorous process to mitigate bias and ensure responsible use. This begins with careful data selection and preprocessing – scrutinizing the data for potential biases related to race, gender, or other protected characteristics. I utilize techniques like fairness-aware algorithms and statistical methods to detect and address disparities.

For example, I might employ techniques like re-weighting samples to balance class distributions or use regularization methods to reduce the impact of high-risk variables potentially correlated with protected characteristics. Furthermore, I conduct thorough impact assessments, analyzing the scorecard’s outcomes across different demographic groups to identify any disproportionate effects. Transparency is key; I document all steps of the process, including the data used, algorithms implemented, and fairness-related considerations. This enables scrutiny and ensures accountability.

AutoML (Automated Machine Learning) significantly streamlines scorecard development. I’ve leveraged AutoML platforms like Google Cloud AutoML and Azure Automated Machine Learning to automate tasks such as feature engineering, model selection, and hyperparameter tuning. This speeds up the development process and allows for rapid exploration of various modeling techniques. However, I maintain a cautious approach. AutoML is a powerful tool, but it shouldn’t replace human judgment entirely. I use it to explore a broader range of options efficiently, but I carefully review the generated models, paying close attention to their performance, interpretability, and fairness implications before deployment. It is crucial to validate and refine the models produced by AutoML, ensuring they meet the specific requirements and ethical considerations of the project.

In a previous project, an existing credit scorecard exhibited declining performance due to shifts in customer behavior. My approach involved a systematic evaluation of the model. I started by analyzing the changes in predictor variables over time, identifying features that had lost their predictive power. This was followed by exploring new potential predictor variables that better captured the evolving customer behavior. Using feature importance analysis, I assessed the contribution of existing and new variables. I then retrained the model using a combination of existing and new features, incorporating techniques like boosting and ensemble methods to enhance predictive accuracy. The result was a significant improvement in the model’s performance, with a noticeable reduction in default rates and improved profitability.

This experience reinforced the importance of continuous monitoring and retraining of scorecards to adapt to evolving data patterns and maintain performance. Regular model audits and updates are essential to ensure the ongoing validity and effectiveness of any scoring system.

Several emerging trends are shaping advanced score study and analysis. One is the increasing use of explainable AI (XAI) to enhance the transparency and interpretability of complex models. This is crucial for building trust and meeting regulatory requirements. Another significant trend is the application of advanced techniques like deep learning and natural language processing (NLP) to scorecards. Deep learning can capture complex non-linear relationships, while NLP enables the incorporation of unstructured data such as text comments and social media activity for improved risk assessment. Finally, the emphasis on responsible AI and ethical considerations is rapidly growing, driving the development of fairness-aware algorithms and techniques to detect and mitigate bias in scoring models.