Salesforce Bulk API 2.0
In this article, we will explore the simplicity and beauty of the Salesforce Bulk API 2.0 with examples covering insert, update, query and delete operations.
How Bulk API 2.0 works?
Salesforce processes Bulk API 2.0 jobs you create and uses job state to describe the status of a queued job.
Salesforce queues a new job for processing once you’ve:
- created the job
2. uploaded data for the job, and
3. moved the job to the UploadComplete state.
- Once the job is de-queued, Salesforce starts processing the job operation and moves the job to the InProgress state.
- Once the job has been fully processed, Salesforce moves the job to the JobComplete state, or Failed state, depending on if the job was successfully processed or not.
How it is different from Bulk API 1.0?
- In Bulk API 2.0, Salesforce automatically chunks the data for your job into multiple internal batches to improve performance while processing the job. So managing batches and job status is moved from user space to system space!
- In Bulk API 2.0, Salesforce creates a separate batch for every 10,000 records in your job data, up to a daily maximum of 100 million records.
- In Bulk API 1.0 differently sized batches are counted equally against your org limits! as shown above.
- In Bulk API 2.0, limits are defined in terms of the number of records processed.
- Bulk API 2.0 REST lives in our native REST framework (/services/data/v47.0/jobs/)
- No more X-SFDC session headers needed in Bulk API 1.0
- Bulk API 2.0 supports all OAuth2 flows from Salesforce’s other REST APIs
- Bulk API 2.0 jobs can be queried through the API
What happens if the limits are exceeded?
- If the limit is exceeded while processing your job data, the remaining data isn’t processed. The job is marked as having failed. Create a new job to process the records that were not processed.
- For each internal batch, if Salesforce can’t process all the records in a batch (10k records) in 10 minutes or less, the batch fails.
- Salesforce will retry failed batches up to a maximum of 10 retries. If the batch still can’t be processed after 10 retries, the entire job is moved to the Failed state and remaining job data is not processed.
An ounce of practice is worth more than tons of preaching.
- Mohandas K Gandhi
Let us practice
Steps in Bulk API 2.0
Install sfbulk2
- Authenticate using OAuth2
- Create a Bulk API 2.0 job
- Upload all your data
- Make the job as UploadComplete to inform Salesforce to start the processing of data just uploaded.
- Check the status of the job
- If the job completes with no errors, we are done!. If the job completed with errors request the complete list of failed records with one API call and find out why those records are failed and submit a new job with corrections
1. Authenticate using OAuth2
2. Create a Bulk API 2.0 job
How to do Bulk API 2.0 Insert in SFDX:
3. Upload all your data
Check the status of the job
4. Make the job as UploadComplete to inform Salesforce to start the processing of data just uploaded
5. Check the status of the job
6. If the job completes with no errors, we are done! Check for any errors
QUERY operation
How to bulkapi 2.0 query in SFDX:
DELETE operation
Query Again! Let us check the delete operation results
How about Update Operation?
Complete Python example here:
Complete SFDX example here:
Bonus-1: Data Generation using Faker Util
Generates a CSV file and returns an array of tuples with fake records requested:
Bonus-2: VLookup using DataUtil
Create data files
Let us do vlookup on the column farm
You can use this feature to map External-Id fields to the Salesforce Internal-Id field. This will be useful when you have only External-Id but you like to update using Salesforce Internal-Id.
