Customer Engagement & Dynamics CRM Forum

Expand all | Collapse all

How to prepare for 3 million new records?

  • 1.  How to prepare for 3 million new records?

    Posted 24 days ago
    Hello,

    I'm trying to prepare for an import of over 3 million records into our online instance of Dynamics CRM.  Specifically, 3 million new Contacts will be brought over from from a legacy system.  The business has been operating in Dynamics CRM with only 30,000 Contact records.  I'm wondering if anyone would be willing to share or suggest some things I should consider before the import occurs.

    Thank you,

    Bob


    ------------------------------
    Robert Chang
    CRM Application Developer
    Cystic Fibrosis Foundation
    Bethesda MD
    ------------------------------


  • 2.  RE: How to prepare for 3 million new records?

    TOP CONTRIBUTOR
    Posted 24 days ago
    Robert,
       Does your system need to be available 24x7 while you are doing the import? For example, are you running the CRM Portal on your system?
    Are you using a tool like Scribe or are you going to try to load it directly into SQL?
    Assuming that you are using D365 in the cloud, and your users can be off-line for a weekend, then you should turn off all:
    • plugins
    • workflows
    • auditing
    • data duplication rules
    • bulk delete operations
    • auto numbering (handle this in your data conversion)
    • email synchronization
    • marketing integration (like ClickDimensions or KnowWho)
    • custom Azure integrations


    ------------------------------
    Nelson Johnson
    Solution Architect
    BroadPoint Technologies, LLC
    Bethesda MD
    ------------------------------



  • 3.  RE: How to prepare for 3 million new records?

    Posted 13 days ago
    Nelson,

    Thank you for providing that list of things to consider.  Our CRM users will experience a some downtime while the import takes place.  I have some experience with Scribe but I've been using SSIS with Kingswaysoft Dynamics CRM tools for smaller integrations and ​plan to use it for this import.  My plan is to stage the CRM data (30k Contacts) and the 3 million records from the legacy system on a SQL database.  Using SSIS/Kingswaysoft, I'll run through multiple levels of matching and deduping.  With a final set, I'll use SSIS/Kingswaysoft to import the data in CRM.  Right now, I'm estimating this is going to take about 3-4 days to complete.

    On the configuration side of CRM, should forms, views and reports be modified at all?  I'm imagining views with dozens of columns could be condensed or forms and reports could also be optimized.

    Thanks again,

    Bob


    ------------------------------
    Robert Chang
    CRM Application Developer
    Cystic Fibrosis Foundation
    Bethesda MD
    ------------------------------



  • 4.  RE: How to prepare for 3 million new records?

    TOP CONTRIBUTOR
    Posted 13 days ago
    The system performance and the design of your views and forms should not be an issue. However, if you have any SSRS reports that use FetchXML, you may have problems, because FetchXML can only retrieve 5000 records. For most use-cases, it is not a big deal, but I have a client that has 100k subscribers to their magazine and they wanted to create a report that would generate mailing labels, which would not work for them - it simply gets to 5000 contacts and stops. We set them up with a dynamic spreadsheet and a Word merge instead.

    ------------------------------
    Nelson Johnson
    Solution Architect
    BroadPoint Technologies, LLC
    Bethesda MD
    Link with me! https://www.linkedin.com/in/nelsonjohnson/
    ------------------------------



  • 5.  RE: How to prepare for 3 million new records?

    Posted 11 days ago
    Excellent point!

    Thank you,

    Bob


    ------------------------------
    Robert Chang
    CRM Application Developer
    Cystic Fibrosis Foundation
    Bethesda MD
    ------------------------------



  • 6.  RE: How to prepare for 3 million new records?

    Posted 13 days ago
    Just a second thought.
    You can also use C# and.NET CRM SDK, if you know C# development. You can push about 1000 (or 999) contacts in one call. If everything goes well, your job may finish it in little over 2 days or 48 hours. Please let me know if you need more information on how to plan this.

    ------------------------------
    Jay Gade
    Gallup, Inc.
    Omaha NE
    ------------------------------