Making Moves — NetSuite File Storage Migration Patterns
This article is relevant if you’re looking to migrate files from local storage, the NetSuite File Cabinet, or from one cloud storage repository to another. We expect the requirements to retain associated joins with all associated NetSuite records.
File migration between repositories or from local storage to cloud storage has been one of the recurring themes we’ve seen in the last year. Many of our clients have been wanting to move files out of the file cabinet, local network drives, or from one cloud storage to another. The reason for these moves ranged from coming live on NetSuite from an on-premise solution to simply moving to a more cost effective and versatile object storage environment.
Below are some of the file migration projects we performed for clients in conjunction with our eXtendFiles solution.
Multi-Repository Migration Use Cases
OneDrive to S3: Migration from directory based to non-object storage
Our client required migration from OneDrive to AWS S3. They were already running eXtendFiles for about two (2) years. The client wanted to switch to S3 to increase file availability and gain more control on accessibility. To do this, the client had to authenticate using OAuth and provide S3 IAM user access on the required S3 bucket where files needed to land. We had to ensure the eXtendFiles records in NetSuite were updated with the S3 links upon migration. We moved all the files over to S3, replaced all links and then deprecated the OneDrive storage.
Box.com to S3: 10.8 million files / 16.3 TB of data = Annual Savings of over $110,000 per year!
While the Box.com integration for NetSuite has been around for a long time, user costs can escalate quickly. For clients leveraging existing Box.com for NetSuite integrations, we had to ensure files are migrated to S3 while retaining all the versions of the Box.com file. We also had ensure that each file was linked to the correct NetSuite record. One of our larger scale projects had more than 10.8 million files, totaling 16.3 Terabytes of data. This migration replaced the extensive history of linked files and took almost a month of migration processing time to reconnect all the records and rebuild. The result yielded an annual savings for our client exceeding $110,000 per year, so it certainly was a worthy endeavor.
Box.com to OneDrive: Directory based migration, consolidated operations in Office 365
This same client wanted to move their employees Box.com personal files to OneDrive while retaining the same folder structure as created in Box.com. Users were granted access using simple OAuth 2.0 screen (similar to the authentication process we have in eXtendFIles for adding a cloud storage repository). Once authenticated, we were able to migrate everyone’s existing structure they had originally built in Box into the OneDrive environment. The client is now running S3 for most of their high velocity, externally facing files, and OneDrive for employee collaboration.
GDrive to S3: Image repository / CDN requirement — migrate and upgrade
In this endeavor, the client was running GDrive which was also acting as a CDN (content delivery network) for their item images. Unfortunately, Gdrive isn’t a great foundation for CDN like functionality and the links had expired over time. The client wanted to switch their storage repository to AWS S3. So, we performed a GDrive to S3 file migration while upholding all the related record information so we could “swap” the links upon landing in their new destination. The client is now running S3 and no longer has to worry about link expiration issues.
During the migration process, we were very cautious to ensure no files were broken or omitted. Several saved searches allowed us to reconcile and confirm all files had successfully been replaced.
The examples shown are media images on the item record from Google Drive to S3. We provided client with side by side comparison of their GDrive and S3 images links for review after the migration.
We even added a slack notification to notify us of the migration progress on a slack channel. This enabled us to receive periodic status updates about the file migration process with how many eXtendFiles had been updated back into NetSuite with original links.
File Cabinet to Cloud Repository
With some of the new NetSuite tier thresholds recently introduced as well as the common file cabinet challenges, we’ve had several clients reach-out to us seeking to migrate their file cabinet usage to eXtendFiles. For each migration, it was imperative we retained all of the associated records and rebuild the connections upon the eXtendFiles creation. We needed to uphold the link to the file cabinet file as well to the newly created eXtendFiles record before deleting the file cabinet files. This dual retention prior to deletion allows for auditing and UAT to ensure client has not lost any data prior to purge. Once the client verified the parity of files, we were clear to purge the File Cabinet of the original files.
Private / Local Files Repository to S3 (Using Cloudberry)
When files are on network drive or a local Windows/Mac computer, we have used MSP360 (formally known as Cloudberry) to move those local files to the storage cloud. We also use Node.js scripts to index the content of directories prior to migration and put them into CSV files for auditing considerations.
A cool alternative to network transfer is the Amazon Snowball, so cool….the LED screen acts as the two-way Shipping Label which changes once it arrives at your facility and needs to be sent back!
File Cabinet to Cloud Repository (AWS S3/One Drive/SharePoint)
In this situation, the client wanted to use the native Expense Log available in the NetSuite Mobile App. The users would log their expense reports and upload receipt images. These postings would then wait until expense report is processed (fully paid). At that point, we would move the File Cabinet files into the repository, create eXtendFiles records, then link eXtendFiles to expense reports . This ensured all native aspects are sound. After which, we migrate and purge files contextually when it was appropriate. This feature can be configured using the bundle settings and support.
eXtendFiles lets you upload your files into cloud storage of your choice and lets you store all the files metadata onto a custom record. This enables our clients to do more with it using native saved searches, workflows and SuiteScripts.
We wrote scripts that would first index all the contents of source repository and then transmit it to another repository using multiple technologies involving Node.js, AWS EC2, shell scripts, RESTlets and native CSV uploads.
We wanted to ensure our clients were confident about file content not being corrupted over the network, so, we added hash integrity checks for both downloading the file and after uploading the file. The script also ensured all the multiple versions of files are downloaded and uploaded to the target repository with hash checks on each version of the file. For the post-mortem analysis, we also took a backup of file contents metadata from the source repository into the target repository to audit the contents. This helped us to ensure that everything was correctly and completely moved to the target repository.
After the contents were uploaded back into the target repository, we created CSV files for eXtendFiles uploads. This enabled much faster updates of links back into NetSuite records as CSV updates can achieve higher performance by avoiding the execution of SuiteScripts and workflows. Since NetSuite has limitations on the file size of a CSV import, we made the number of rows per file configurable. That way, it could be customized based on the specific needs and would not violate the NetSuite limitations.
Keeping Conscious of NetSuite Resource Utilization
We performed the CSV uploads back into NetSuite during non-business hours in an attempt to avoid performance issues. We even took it a step further for clients that have a SuiteCloud Plus license. This enabled us to take advantage of multiple CSV processing queues which gave us a higher throughput for updating eXtendFiles back into NetSuite.
We further added ability to configure the concurrent calls while downloading and uploading the files so, that we could fully exhaust the available concurrency limits of the source and target cloud repository considering the applicable rate limits. This is important from being able to meet the deadlines of the project. When performing migrations to AWS S3, we use EC2 instances within the same region to ensure data transfer cost is minimal and we get higher performance.
Monitoring and Auditing the migration
There are several approaches that can be taken to ensure your file migration can be properly audited. In some cases, it can be as basic as going into your cloud storage interface to review the count of files before and after migrations. Other times, you may want to have the ability to audit the exact content and perform comparisons file-by-file when needed.
Typically, when we migrate the files related to NetSuite records we also provide a saved search showing both the original and updated file links so clients have a simple side by side comparison of the migration results. We also usually provide the migrated content file name and paths in a CSV file which you can use as reference for future audit needs.
eXtendTech provides professional services and can aide you in your file migration needs. If you need to migrate files from one source to another or are looking for some best practices, we can help you with some smooth moves on the file migration dancefloor.
Contact Us if you would like to learn and discuss more.
This article originally appeared on the eXtendTech.net blog.