Metrc API Bulletin 089
Metrc API Bulletin
- Bulletin Number: 089
- Distribution Date: 02/26/2021
- Effective Date: 03/5/2021
- Contact Point: api-info@metrc.com
Subject:
Update on California Production Environment and Implementation of Phase 3 Rate Limits in California Reason: To provide third party integrators with an update regarding system performance in California, and notice of implementation of phase 3 Rate Limiting in the California instance.
Greetings Metrc® Integration Providers,
The Metrc team continues to field multiple reports on the performance and responsiveness of Metrc in the California production system. As you are aware, these issues have been occurring intermittently since February 17. This notice will provide you with what steps have already been taken or are in progress to address these issues, and what additional steps are planned.
The additional steps include implementation of phase 3 rate limiting. The volume and pace of calls to Metrc’s API has been a significant contributing factor to server load and applying the rate limits in California that are already in place in other Metrc states will help alleviate that load. Once those limits are in place in California, we will continue to evaluate these limits and determine if modifications are necessary.
Most importantly, we want to acknowledge your patience, and that of your customers, as we work to address system performance in California, and to apologize for integration disruptions. While it is no doubt frustrating to have to report error messages or timeouts to the API support team, we appreciate your continuing to do so in a timely manner, as such reports are a critical aid to us in understanding and responding to system performance concerns.
Timeline
Initial performance concerns surfaced on Wednesday, February 17, and continued intermittently to that Friday. Our mitigation measures (described below) appeared to be successful by Friday afternoon, though scattered reports of performance concerns came in Saturday as well. On Monday, February 22, performance again degraded, and these issues have recurred intermittently this week.
Mitigation Measures
• On February 17, Metrc’s Technology team identified and corrected an error in a background job which, on initial investigation, was determined to be the potential cause to performance issues.
• On February 18, Metrc significantly increased resources on our servers for California while ongoing investigation to the issues ensued.
• Throughout this process, we have reached out to integrators who were generating high volumes of API traffic to discuss their business needs and how they might be met without impacting overall system performance. The integrators we have reached out to have been uniformly responsive and constructive both in sharing information and in modulating their traffic to Metrc, and we thank all involved in that process.
Ongoing Efforts
Metrc continues to take steps to improve this situation:
• Further server upgrades are in the works. We are anticipating those changes will require a brief production outage; if so, we will schedule that outage so as to minimize the impact on licensees and integrators, and we will provide notice to Metrc online users via a banner message within the application.
• We are investigating other potential architectural changes and will provide you with notice of such changes if they are not expected to be transparent to you.
• We also will be implementing Rate Limiting Phase 3 as another measure to improve the performance situation.
How You Can Help
In addition to the work being done on the Metrc end, we wanted to also outline some items that would be helpful for the integrators to consider.
1. Apply the best practices within the Metrc API Documentation for maintaining inventory in sync and requesting multiple days of data.
2. Consider any large jobs that require pulling data out of Metrc to be run in off business hours (e.g., “between 9pm and 7am”)
3. Apply a “wait time” between API requests, especially if you notice slower response times.
Rate Limiting Update
As we noted in multiple previous bulletins, starting with bulletin 70 issued on July 14, 2020, we are phasing in rate limits for third-party integrators. Our goal is to improve system performance for all users by limiting the number of API calls a third-party integrator can make in a given amount of time to ensure no single user or company can overwhelm the system.
Rate limiting is a best practice used by software platforms worldwide, such as Twitter, LinkedIn, Slack, Quick Books, and Google. Metrc’s rate limiting approach is intentionally designed to gradually phase in so integrators have ample time to adjust their systems as necessary.
We would like to announce that for the California instance of Metrc, the third rate limit will be implemented, on March 5th, 2021. This date was selected to strike a balance between providing advance notice to integrators and alleviating the issues noted in this bulletin. In addition to this notice, Metrc will reach out to those integrators we expect to be most impacted by the phase 3 rate limits.
Phase 3 Rate Limits
• Go-Live Date: January 21, 2021 (excluding California)/ March 5, 2021 (California)
• Rate Limits (all implemented simultaneously): o 50 GET calls per second per facility,
o 150 GET calls per second per integrator key,
o 10 concurrent GET calls per facility, and
o 30 concurrent GET calls per integrator
• Total Allowable Daily Calls per Integrator: o 12.96 million per day per integrator key
• Non-Affected Calls: Rate limits do not apply to PUT, POST, or DELETE calls
See our API Resources page for further information on Metrc rate limiting, including technical information: https://www.metrc.com/api-resources
We appreciate your understanding and support. Please contact api-info@metrc.com with any questions or concerns.
