Monitoring the performance of AMC's

Last week I posted an article that discussed how lenders and brokers using Appraisal Management Companies (AMC’s) need to have an effective method for monitoring the performance of their vendors. As stated then, it’s my feeling that at a minimum there are four key factors to examine when doing so. Revision rate (what percentage of the total number of reports require a correction due to an error by the appraiser), service level met rate (percentage of time that the vendor delivers the report within the agreed upon timeframe), the volume rate (the percentage of volume that each vendor is providing to your business) and finally the average score from the GSE’s collateral review engines (typically from FNMA’s CU). During the remainder of this article, I will provide my insights on how I used each of these components to evaluate the performance of the AMC’s that my companies used.

Revision Rates

Of the four components, revisions are likely the most difficult to accurately monitor because they are not as black and white as they may seem. On the surface, a company could simply sum up any appraisals that had a revision. However, once you look deeper into the data you find that to do so wrongly impacts the AMC. Specifically, minor revisions such as sales price updates or an underwriter requesting an additional comparable should not be reflected as an error on the report. These should be categorized as items used to update/strengthen the report rather than an indication of fault on the original appraiser or a weakness in the AMC’s quality control. Remember, this component is intended to evaluate the quality of the reports at the time of delivery. Items that are out of the control of the appraiser at the time he/she prepares the report should not be included here. 

Revisions of a higher magnitude should be categorized and used to calculate into the Revision Rate. Examples such as the appraiser failing to report that the property is located proximate to commercial or external influences, condotels, additional units, etc, etc. This list goes on and on. These are the types of revisions that need to be monitored and summarized when developing and fairly calculating a revision rate for your AMC’s. Once you identify these high impact revisions, simply total them and divide by the total appraisals delivered to arrive at an accurate revision rate. Applying this method to each of your vendor relationships will give you a good basis to start a successful monitoring program.

Service Level Rates

When selecting an AMC to partner with, the turn time expectations should be set within a Service Level Agreement (SLA). They should be a realistic guideline that reflects the strategic goals of the lender in terms of timeliness to close their loans. Obviously, the appraisal process is a big factor in the lenders ability to close during a certain period. Therefore, it is imperative that the lender partners with an AMC that can consistently deliver within the expectations agreed upon in the SLA. Failure to do so creates a whole host of headaches to not only the lenders, but most importantly the consumer. 

The AMC and the Lender should agree on the expectation by discussing the process and the markets that the properties will be located. Realizing that every property is different and various factors could impact the AMC’s ability to deliver the report by the SLA is critical. Just as with the revision rates, the most simplistic approach of counting days on the calendar may not be the fairest way to approach this. First, the lender and the AMC must agree whether to use calendar days or business days. Business days is the most commonly used. Next, a formula needs to be developed that will consider ON HOLD times that are out of the AMC/Appraisers control that could negatively impact the turn time. Depending on the method or system used to track the events of the appraisal order, these dates may or may not be readily available. This will determine how sophisticated your SLA rate calculation can be. The important thing to remember when comparing AMC’s against each other is to realize that the data available may or may not be consistent across each vendor. Each lender must decide how to fairly apply this component.

Volume rates

This is probably the easiest component to monitor and calculate for a lender. Some lenders choose to completely control the allocation that is given to an AMC by a percentage. Other lenders use a “round robin” method, where each AMC receives the same number of orders. Finally, some lenders have decided that the placement may be dependent on a set of factors about the loan, property or geography to determine the AMC that receives each order.

Some lenders even allow their customers/brokers to select the AMC from an approved list of vendors. In this case, the allocation is completely dependent upon the reputation and relationship the AMC has with the customers of the lender. When this method is used, the lender yields all control of where the orders are placed other than which AMC’s are on the list. The risk with this approach is that statistically, the poorest performing AMC’s could still receive a high volume (percentage) of the orders for your company. However, its presumed that over time this corrects itself as the customers/brokers will start to place orders with different AMC’s if the rate of revisions is high. One way or another, a poorly performing AMC could impact your business with this approach.

Average CU Scores

The last component is to calculate an average of the scores the appraisal receives from the GSE’s collateral review engine. For the purposes of this article, Fannie Mae’s Collateral Underwriter (CU) engine will be used. 

The CU engine renders scores on a range from 1.0 to 5.0. Scores of 1.0 are deemed to be the lowest risk and scores of 5.0 the highest risk. Therefore, by simply gathering the scores of each appraisal and matching the AMC name, one can calculate an average score for each. Historically, I would expect to see the averages for each AMC to fall within the 2.0 to 3.0 range. However, depending on the markets that each lender is originating these scores can vary due to the availability of data for the CU engine. Metropolitan and urban areas are going to be much more accurate since there are generally a good supply of comparable closings. Conversely, more rural markets or areas with uniquely complex properties the supply of comparable closings is less. These factors must be considered when attempting to fairly compare AMC’s in this component. An AMC that is being assigned orders in a metropolitan market will generally score better than an AMC that is adversely assigned orders in rural markets. Each lender must determine how this component is impacted by their own assignment method.

Summary

In summary, a solid vendor monitoring approach is necessary to evaluating the performance of the AMC’s your company is partnering with. Data availability from each AMC is critical to applying fair approaches to the methods used. If your company uses a centralized ordering platform, the data should be consistent between each vendor. However, if not, it can pose some challenges in making the data fair and consistent. It is up to each company to determine what is possible. Once monitoring procedures like those described are in place, you can begin to determine if you are partnering with the right AMC’s. In the many years that I created similar programs, it was easy to see those AMC’s that consistently performed well and those that did not. At a minimum on a quarterly basis, your organization should also be deciding whether the bottom performing AMC’s are worth maintaining a relationship.