Customer Engagement & Dynamics CRM Forum

Expand all | Collapse all

Very Slow Plugin Execution After Update 9

  • 1.  Very Slow Plugin Execution After Update 9

    Posted May 08, 2018 10:37 AM
    Edited by Field Chari May 09, 2018 01:05 PM
    Morning all,

    So I have been banging my head over performance issues we are having after the update 9 patch was installed on our Sandbox Environment. Plugins have slowed down dramatically, and i'm unsure what else to do other than turning to the CRMUG community. Here is an example of a quotedetail on Update plugin that fires onupdate to perform custom math operations:


    //Get all quote detail line items on this quote
    private EntityCollection getLineItemCollection(IOrganizationService service, Guid quoteID)
    {
    try
    {
    EntityCollection results = new EntityCollection();
    QueryExpression query = new QueryExpression
    {
    EntityName = "quotedetail",
    ColumnSet = new ColumnSet("cti_dorollup", "cti_recalculate", "cti_taxrate", "cti_gridtype", "cti_costcode", "cti_extendeditemcost", "baseamount", "tax", "extendedamount", "quantity", "priceperunit", "cti_itemcost", "cti_actualmargin")
    };
    query.Criteria.AddCondition("quoteid", ConditionOperator.Equal, quoteID);
    query.Criteria.AddCondition("cti_gridtype", ConditionOperator.NotNull);
    results = service.RetrieveMultiple(query);
    return results;
    }
    catch (Exception ex)
    {
    throw new InvalidPluginExecutionException("[GetLineItems]: " + ex.Message);
    }

    Before Update: 856ms
    After Update: 3482ms

    ---------------------------------------------------------------------------------------------------------------

    Then there is also this, after all math is applied, save the changes back to the line items on said quote:

    //Bulk Save All LineItems
    if (transactionRequest.Requests.Count > 0)
    {
    //Execute Bult Update
    list = ((OrganizationRequestCollection)transactionRequest.Requests);
    foreach (UpdateRequest item in list)
    {
    item.Target.Attributes.Remove("cti_gridtype");
    item.Target.Attributes.Remove("quotedetailid");
    item.Target.Attributes.Remove("transactioncurrencyid");
    item.Target.Attributes.Remove("quantity");
    }
    service.Execute(transactionRequest);
    }


    Number of items in ExecuteMultipleTransaction: 12
    Before Update: 1455ms
    After Update: 12482ms

    There is barely any significant data being manipulated here, but the performance is painfully slow, and not sure how else to optimize above code. I've removed some attributes hoping that improves performance, but barely any speed is gained.

    Is anything sticking out for you that i could try? The plugins themselves aren't that complicated, but the speed is really bad. Start to finish can take up to 80 seconds, just to update 12-24 quotedetail items. And there are no workflows associated to the quotedetail entity :(

    Any help would be appreciated! Thanks

    ------------------------------
    Field Chari
    Conference Technologies, Inc.
    Maryland Heights MO
    ------------------------------
    CRMUG Summit - Post


  • 2.  RE: Very Slow Plugin Execution After Update 9

    TOP CONTRIBUTOR
    Posted May 08, 2018 11:08 AM
    Edited by Peter Gulka May 08, 2018 11:10 AM
    Oh. OH!!! Us too!

    @Malcolm McAuley and I noticed SEVERELY degraded workflow performance, and overall slowness in the UI.

    I opened a case with MS, and an analyst and I did a screenshare to trace the problem. We identified that there was a 6-8 minute gap between when a workflow was being called to execute, and when it actually started to run. It just sat in the queue waiting for resources, or was cancelled with an error message that no resources were available. This with no data after a clean reset of the sandbox, and only 2 people logged in.


    Interestingly, we can't seem to find a "Try Again" button anywhere, so that was fun.

    Once the workflow was actually launched, it appeared to run normally - it was getting it out of the queue that was the problem. ​This was on Friday, and I have not heard back from them yet.

    EDIT: Also, importing a handful of base records we needed took several minutes when it should have taken seconds. They sat in "Submitted" for a really long time. We are talking 10-20 records in CSV file taking 15-20 minutes to complete.

    ------------------------------
    Peter Gulka
    National Manager, Systems Enhancement and Training
    Big Brothers Big Sisters of Canada
    Burlington ON
    ------------------------------

    CRMUG Summit - Post


  • 3.  RE: Very Slow Plugin Execution After Update 9

    Posted May 08, 2018 11:47 AM
    @Peter Gulka @Malcolm McAuley Ouch! Thats a huge perfomance slowdown from a fresh install!

    I did also notice that once in a while, the process goes from start to finish within 5 seconds (which is normal), but then goes back to very slow processing for no reason (sometimes hitting the 2 minute timeout), leading me to believe my code might not be the culprit, and it might be something else that's slowing down the back end processing.

    Thankfully this is isolated in our Development Environment as we postponed updating our production environment until the slow downs have been resolved.

    So have you heard back from Microsoft regarding this issue? @Margaret Mundt​ and I are considering opening a ticket as well. Kind of a relief knowing we aren't the only ones facing this :)​​

    ------------------------------
    Field Chari
    Conference Technologies, Inc.
    Maryland Heights MO
    ------------------------------

    CRMUG Summit - Post


  • 4.  RE: Very Slow Plugin Execution After Update 9

    TOP CONTRIBUTOR
    Posted May 08, 2018 11:50 AM
    Nothing back from MS yet. Our Production update is scheduled for July 27, so we have some time to figure this out. I'm leaning toward this being a sandbox server issue, not a v9 issue, but I could be wrong. That's just a guess.

    I'm a fan of always opening a ticket, particularly with new version issues. If they don't know what we are experiencing, they can't help. The more people who report something, the more info they have to troubleshoot, and see if it is systemic vs isolated to one client.

    ------------------------------
    Peter Gulka
    National Manager, Systems Enhancement and Training
    Big Brothers Big Sisters of Canada
    Burlington ON
    ------------------------------

    CRMUG Summit - Post


  • 5.  RE: Very Slow Plugin Execution After Update 9

    GOLD CONTRIBUTOR
    Posted May 09, 2018 11:02 AM
    ​Have you checked for a database mismatch?   There was a service update applied to v9 a little over week ago that caused multiple issues with a few of our instances.  Once the databases were fixed by Microsoft, those issues went away.


    ------------------------------
    Ben Edgren
    CRM Applications Manager
    Hufcor, Inc
    Janesville WI
    ------------------------------

    CRMUG Summit - Post


  • 6.  RE: Very Slow Plugin Execution After Update 9

    Posted May 09, 2018 01:01 PM
    Edited by Field Chari May 09, 2018 01:03 PM
    @Ben Edgren Interesting find, had not checked on that. I do notice that our Test Instance and Production Instance have the same 8.2.2.1559 version, but the problematic DEV Instance has a slight minor version mismatch. Will make sure to inform Microsoft Support on the differences!



    ------------------------------
    Field Chari
    Conference Technologies, Inc.
    Maryland Heights MO
    ------------------------------

    CRMUG Summit - Post


  • 7.  RE: Very Slow Plugin Execution After Update 9

    Posted Jun 20, 2018 08:56 PM
    In general, you shouldn't worry just because App and DB versions don't match. For more information read this blog.

    ------------------------------
    Vinay Singh
    Microsoft
    ------------------------------

    CRMUG Summit - Post


  • 8.  RE: Very Slow Plugin Execution After Update 9

    GOLD CONTRIBUTOR
    Posted May 09, 2018 07:30 AM
    I've had some 9.0 orgs that had some Async Workflow slowness issues,   Tickets have been open and apparently some sort of fix was applied.  Don't know if this was directly related to your issue, but anytime you experience a large slow down like what you've documented, I'd always call MS.

    ------------------------------
    Daryl LaBar
    Capability Manager, MVP
    PowerObjects - An HCL Technologies Company
    Fishers IN
    ------------------------------

    CRMUG Summit - Post


  • 9.  RE: Very Slow Plugin Execution After Update 9

    GOLD CONTRIBUTOR
    Posted May 09, 2018 07:12 AM
    That code is about as straightforward as you can get, so not sure how to optimize it.  With a query, you usually hear of performance returning all columns, but you are filtering cols.

    I am assuming this is a single call and not a loop, you don't have retrieve multiple plugins, and this is not firing multiple instances concurrently?

    Because this is post upgrade, one possibility is that this is an indexing issue. Upgrade means schema updates and that could affect indexing.

    Unfortunately, since this is Online, you have no control over this directly as you would on premise. I found a post from ages ago that indicates that Microsoft Support can apply indexes for you. I have never tried this but it might be worth opening a ticket.

    Tip #39: Custom indexes in CRM Online
    Dynamics CRM Tip Of The Day remove preview
    Tip #39: Custom indexes in CRM Online
    A little known secret for CRM Online users and a good reminder from Daniel Cai.
    View this on Dynamics CRM Tip Of The Day >

    #CustomerEngagement


    ------------------------------
    James Novak
    Technical Architect
    The FutureZ Consulting, LLC
    Springfield VA
    ------------------------------

    CRMUG Summit - Post


  • 10.  RE: Very Slow Plugin Execution After Update 9

    Posted May 09, 2018 01:02 PM
    You are correct, the service.execute is outside of a loop. Made sure that all loop operations were done before performing the execute function, so its only being called once. Also, the retrievemultiple is also being called once, so no bad coding there. Will also inform microsoft of potentially indexing our databases to improve performance!

    ------------------------------
    Field Chari
    Conference Technologies, Inc.
    Maryland Heights MO
    ------------------------------

    CRMUG Summit - Post


  • 11.  RE: Very Slow Plugin Execution After Update 9

    TOP CONTRIBUTOR
    Posted May 09, 2018 08:35 AM

    Field

    Any chance duplicate detection is on? A vendor "accidentally" activated this on a log table, yes I said log table. This resulted in our plugin hitting the 2 minute timeout whenever we create/update records.

    Cheers!



    ------------------------------
    Rex Tan
    Kent OH
    *Always be CURRENT with JavaScript and C#, NEVER be obsolete.

    DISCLAIMER: All views expressed on this site are my own and DO NOT represent the opinions of ANY entity whatsoever with which I have been, am now, or will be affiliated.
    ------------------------------

    CRMUG Summit - Post


  • 12.  RE: Very Slow Plugin Execution After Update 9

    Posted May 09, 2018 01:03 PM
    I actually turned off duplication detection to see if that would help, but there were zero performance gains​​​​ unfortunately. Duplication Detection is enabled on the other environments, and those are functioning at excellent speeds, so didn't think it was the cause of the slow downs either though

    ------------------------------
    Field Chari
    Conference Technologies, Inc.
    Maryland Heights MO
    ------------------------------

    CRMUG Summit - Post


  • 13.  RE: Very Slow Plugin Execution After Update 9

    TOP CONTRIBUTOR
    Posted May 10, 2018 08:31 AM
    Field

    Did you index the quoteid and the id field in the transaction request?

    ------------------------------
    Rex Tan
    Kent OH
    *Always be CURRENT with JavaScript and C#, NEVER be obsolete.

    DISCLAIMER: All views expressed on this site are my own and DO NOT represent the opinions of ANY entity whatsoever with which I have been, am now, or will be affiliated.
    ------------------------------

    CRMUG Summit - Post


  • 14.  RE: Very Slow Plugin Execution After Update 9

    Posted May 10, 2018 02:03 PM

    Did you index the quoteid and the id field in the transaction request?
    Rex Tan,  05-10-2018 08:31 AM
    Not sure i follow :/ How do you index a field in a transaction request?

    ------------------------------
    Field Chari
    Conference Technologies, Inc.
    Maryland Heights MO
    ------------------------------

    CRMUG Summit - Post


  • 15.  RE: Very Slow Plugin Execution After Update 9

    Posted May 30, 2018 10:07 AM
    It's not just updated environments.  We have deployed new V9 instances  for clients and are experiencing the same performance and reliability issues with workflows and plugins.  We have someone on the phone with Microsoft almost every day

    ------------------------------
    Doug McLachlan
    VP of Consulting Services
    ITK Consulting Inc.
    Vancouver
    ------------------------------

    CRMUG Summit - Post


  • 16.  RE: Very Slow Plugin Execution After Update 9

    Posted May 31, 2018 10:04 PM
    Hey Field,

    Go ahead and log a support ticket. Please let me know (michu@microsoft.com) the ticket number and I'll see if I can help them out. Are there any plugins registered for any operations happening within the context of the plugin, such as RetrieveMultiple or on Update. One thing to note is that it appears that you are using ExecuteMultipleTransaction within the context of a plugin. There is not a need to do this since you are in the context of a transaction already and it adds overhead and is subject to throttling. Take a look at Avoid usage of batch request types in plug-ins and workflow activities
    Microsoft remove preview
    Avoid usage of batch request types in plug-ins and workflow activities
    In this article Applies to: Dynamics 365 (online), version 9.0 Microsoft Dynamics 365 Dynamics CRM 2016 Online and Dynamics CRM 2016 on-premises Dynamics CRM 2015 Dynamics CRM 2013 Dynamics CRM 2011 Category: Usage, Reliability, Performance Impact potential: Medium The following are possible effects when using ExecuteMultipleRequest or ExecuteTransactionRequest message request classes within the context of a plug-in or workflow activity: Due to their long-running nature, batch request messages expose sandbox-isolated plug-in types to the two-minute (2000-ms) channel timeout exception and can degrade the user experience for synchronous registrations.
    View this on Microsoft >


    ------------------------------
    Mike Huguet
    PowerApps/Dynamics 365 - Senior Software Engineering Manager
    Microsoft
    City of Central LA
    ------------------------------

    CRMUG Summit - Post


  • 17.  RE: Very Slow Plugin Execution After Update 9

    Posted Jun 01, 2018 09:41 AM
    Good day @Mike Huguet,

    Thanks for chiming in. I have indeed got in touch with Microsoft Support regarding this issue, and it seems it was caused by one of two things:

    1. There is a mismatch in our CRM Version and Database Versions as shown here:

    CRM Database Mismatch

    2. Bug numbers 991997 and 991914 are bugs that have affected our instance of Dynamics and are still to be resolved. These bugs might also be the culprit:

    Dynamics Bug 991997 and 991914

    3. As for ExecuteMultipleTransaction, I have attempted to perform a foreach loop with a service.update, aswel as a ExecuteMultipleRequest. In terms of performance, ExecuteMultipleTransaction was the fastest of the three options by factors of seconds as compared to the other options, and that is why it was chosen

    I will for sure reach out by email and maybe you could shed further light on the matter.

    Thanks again

    ------------------------------
    Field Chari
    Conference Technologies, Inc.
    Maryland Heights MO
    ------------------------------

    CRMUG Summit - Post


  • 18.  RE: Very Slow Plugin Execution After Update 9

    GOLD CONTRIBUTOR
    Posted Jun 04, 2018 08:35 AM
    Hey Mike

    Thanks for the link to the article!  We've run into this issue several times recently but I did not know about this post.

    Jim

    ------------------------------
    James Novak
    Technical Architect
    The FutureZ Consulting, LLC
    Springfield VA
    ------------------------------

    CRMUG Summit - Post


  • 19.  RE: Very Slow Plugin Execution After Update 9

    TOP CONTRIBUTOR
    Posted Jun 21, 2018 08:42 AM

    I found this best practices from MS. It basically states NOT to use batch requests unless it is for outside services.

    Avoid usage of batch request types in plug-ins and workflow activities
    https://docs.microsoft.com/en-us/dynamics365/customer-engagement/guidance/server/avoid-batch-requests-plugin



    ------------------------------
    Rex Tan
    Kent OH
    *Always be CURRENT with JavaScript and C#, NEVER be obsolete. Soon to be MCSD (2 to go).

    DISCLAIMER: All views expressed on this site are my own and DO NOT represent the opinions of ANY entity whatsoever with which I have been, am now, or will be affiliated.
    ------------------------------

    CRMUG Summit - Post


  • 20.  RE: Very Slow Plugin Execution After Update 9

    SILVER CONTRIBUTOR
    Posted Jan 10, 2019 07:44 AM
    Has anyone had any progress with this?  Our V9 plugin performance is absolutely atrocious, as has been the support from Microsoft over the last two months we've had a ticket open.  They have now confirmed that they have made fundamental changes to the underlying platform and that they now expect performance to be worse than 8.2, but not as bad as we are experiencing.  Our users are far from impressed.

    ------------------------------
    Luke Sartain
    Dynamics 365 (CE) Developer
    ICS Cool Energy
    Southampton, UK
    ------------------------------

    CRMUG Summit - Post


  • 21.  RE: Very Slow Plugin Execution After Update 9

    GOLD CONTRIBUTOR
    Posted Jan 12, 2019 01:10 AM
    Hello Everyone,

    We upgraded our sandbox instance to version 9 and we have also started to notice severe performance issues with our plugins . The plugins have code written to calculate income share for an associate  based on the estimated and the actual revenue fields . This same piece of code works at great speeds in the version 8.2 . Its just the upgrade that has caused the issue.

    Please let us know if we should create a separate ticket with Microsoft to have a look into the issue.

    Thank you


    ------------------------------
    Bhartendu Pandey
    Willis Towers Watson
    London
    ------------------------------

    CRMUG Summit - Post


  • 22.  RE: Very Slow Plugin Execution After Update 9

    SILVER CONTRIBUTOR
    Posted Jan 14, 2019 03:59 AM
    Bhartendu,

    Definitely open a ticket.  After two months of the ticket being open they finally traced the plugins on their servers last Friday and found it was indeed something their end hogging all the resource (BPO.UpdateWithPipeline?) causing the performance issues.  Glad they've finally identified the cause but no awaiting an update on a solution.

    Luke

    ------------------------------
    Luke Sartain
    Dynamics 365 (CE) Developer
    ICS Cool Energy
    Southampton, UK
    ------------------------------

    CRMUG Summit - Post


  • 23.  RE: Very Slow Plugin Execution After Update 9

    GOLD CONTRIBUTOR
    Posted Jan 14, 2019 07:15 AM
    ​Hello Luke,

    Thank you for your reply.
    I have created a ticket with MSFT will keep everyone updated with the results.

    ------------------------------
    Bhartendu Pandey
    Willis Towers Watson
    London
    ------------------------------

    CRMUG Summit - Post


  • 24.  RE: Very Slow Plugin Execution After Update 9

    SILVER CONTRIBUTOR
    Posted Jan 23, 2019 06:37 AM
    Hi all

    To keep you updated on how we're getting on.  Microsoft have confirmed that system pricing calculation is kicking in each time ANY attribute is updated on quote product as opposed to just when pricing fields are updated.  In the short term they have suggested disabling sysem pricing calculation in system settings and then writing a custom plugin to do the calculations manually.  We've found it does speed things up but still not quite to the speed of 8.2.



    ------------------------------
    Luke Sartain
    Dynamics 365 (CE) Developer
    ICS Cool Energy
    Southampton, UK
    ------------------------------

    CRMUG Summit - Post


If you've found this thread useful, dive deeper into User Group community content by role